Training method and device of map element prediction model, equipment and storage medium
By introducing sample image description information into the map feature prediction model, using the prediction network for feature extraction and query vector training, the problems of slow convergence speed and low accuracy of existing models are solved, and more efficient map feature prediction is achieved.
Patent Information
- Application Number
- CN202410892794.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-04
- Publication Date
- 2024-11-22
AI Technical Summary
The existing online map generation model has shortcomings in the convergence speed and accuracy of the model, and has failed to effectively utilize real information.
By obtaining the truth value detection results and description information in the sample image, using the prediction network to be trained for feature extraction, and training the prediction results based on the first query vector and the second query vector, the description information is introduced to improve the accuracy of the model and Convergence speed.
It improves the accuracy and convergence speed of the map feature prediction model, enhances the utilization of real information, and improves the overall performance of the model.
Smart Images

Figure CN119005284A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of computer technology, in particular to data processing technology and autonomous driving technology, and specifically to a training method, device, equipment and storage medium for a map element prediction model. Background Art
[0002] With the rapid development and widespread application of autonomous driving technology, the theory of "focusing on perception and neglecting maps" has been increasingly recognized and practiced. Its core logic is not to rely too much on high-precision maps. In places where there are no high-precision maps, the system can provide vehicles with the road information required for autonomous driving by building local environment maps (LocalMap) in real time, etc., to reduce dependence on high-precision maps.
[0003] Online map generation means that the vehicle perceives the surrounding environment in real time through its onboard sensors (such as lidar, camera, millimeter wave radar, etc.), and builds local road and traffic information online. The current online map generation technology is achieved by building a deep neural network to identify map elements in the vehicle's surrounding environment. The model converges slowly and the model accuracy is not high. Summary of the invention
[0004] The present invention provides a training method, device, equipment and storage medium for a map element prediction model to solve the problems that the existing online map generation model lacks effective use of real information and has a slow model convergence speed.
[0005] According to one aspect of the present invention, a method for training a map element prediction model is provided, comprising:
[0006] Obtaining a sample image, a true value detection result of a sample map element in the sample image, and description information of the sample map element in the sample image;
[0007] Extracting features of the sample image by using the prediction network to be trained to obtain sample image features;
[0008] Predicting the sample map element according to a first query vector to be learned and the sample image feature to obtain a first prediction result of the sample map element;
[0009] Determine a second query vector of the sample map element according to the description information of the sample map element in the sample image, and perform prediction according to the second query vector and the image feature of the sample image to obtain a second prediction result of the sample map element;
[0010] The prediction network to be trained is trained according to the first prediction result, the second prediction result and the true value detection result to obtain a map element prediction model.
[0011] According to another aspect of the present invention, there is provided a training device for a map element prediction model, comprising:
[0012] A data acquisition module, used to acquire a sample image, a true value detection result of a sample map element in the sample image, and description information of the sample map element in the sample image;
[0013] An image feature extraction module, used to extract features of the sample image through a prediction network to be trained to obtain sample image features;
[0014] A first prediction result acquisition module, used for predicting the first prediction result of the sample map element according to the first query vector to be learned of the sample map element and the sample image feature;
[0015] A second prediction result acquisition module, used to determine a second query vector of the sample map element according to the description information of the sample map element in the sample image, and to perform prediction according to the second query vector and the image features of the sample image to obtain a second prediction result of the sample map element;
[0016] The model training module is used to train the prediction network to be trained according to the first prediction result, the second prediction result and the true value detection result to obtain a map element prediction model.
[0017] According to another aspect of the present invention, a computer program product is provided, comprising a computer program, wherein when the computer program is executed by a processor, the computer program implements the method for training a map feature prediction model according to any embodiment of the present invention.
[0018] According to another aspect of the present invention, an electronic device is provided, the electronic device comprising:
[0019] at least one processor; and
[0020] a memory communicatively connected to the at least one processor; wherein,
[0021] The memory stores a computer program that can be executed by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the training method of the map element prediction model described in any embodiment of the present invention.
[0022] According to another aspect of the present invention, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to implement the training method of the map element prediction model described in any embodiment of the present invention when executed by a processor.
[0023] The embodiment of the present invention obtains a sample image, a true value detection result of a sample map element in the sample image, and description information of the sample map element in the sample image; extracts features of the sample image through a prediction network to be trained; predicts a first prediction result of the sample map element based on a first query vector to be learned of the sample map element and the sample image feature; determines a second query vector of the sample map element based on the description information of the sample map element in the sample image, and predicts a second prediction result of the sample map element based on the second query vector and the image feature of the sample image; trains the prediction network to be trained based on the first prediction result, the second prediction result, and the true value detection result to obtain a map element prediction model. Through the above technical solution, in the training stage of the map element prediction model, the description information of the sample map element in the sample image is introduced, and the second query vector of the sample map element is determined based on the description information of the sample map element, which is used to guide the first query vector to learn the description information of the sample map element, so that the first query vector can contain the description information of the map element, thereby improving the accuracy of the first query vector. By introducing the description information of the sample map element, the convergence of the model is accelerated and the accuracy of the model is improved.
[0024] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present invention, nor are they intended to limit the scope of the present invention. Other features of the present invention will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS
[0025] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.
[0026] Figure 1a It is a first flow chart of a method for training a map element prediction model provided by an embodiment of the present invention;
[0027] Figure 1b is a schematic diagram of the structure of a map element prediction model provided by an embodiment of the present invention;
[0028] Figure 2 is a second flow chart of the method for training a map element prediction model provided by an embodiment of the present invention;
[0029] Figure 3 is a third flow chart of the method for training a map element prediction model provided by an embodiment of the present invention;
[0030] Figure 4 It is a structural schematic diagram of a training device for a map element prediction model provided by an embodiment of the present invention;
[0031] Figure 5 It is a schematic diagram of the structure of an electronic device implementing an embodiment of the present invention. DETAILED DESCRIPTION
[0032] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.
[0033] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.
[0034] Figure 1a This is a first flow chart of a method for training a map element prediction model provided by an embodiment of the present invention. This embodiment is applicable to the case of training a map element prediction model. The method can be executed by a training device for a map element prediction model. The device can be implemented in the form of hardware and / or software. The device can be configured in an electronic device with corresponding data processing capabilities. Figure 1a As shown, the method includes:
[0035] S110, obtaining a sample image, a true value detection result of a sample map element in the sample image, and description information of the sample map element in the sample image.
[0036] During the training process of the map element prediction model, the sample map elements may include road boundaries, lane lines, sidewalks, stop lines, traffic lights, etc. The true value detection result may include the true value detection box of the sample map element and the element category to which it belongs. The description information of the sample map element is used to describe the scene in which the sample map element is located, the position relationship of the sample map element in the scene, etc.
[0037] Optionally, the vehicle senses the surrounding environment in real time through its onboard sensors (such as laser radar, camera, millimeter wave radar, etc.), collects environmental information around the vehicle, and obtains multi-view sample images. Obtain the true value detection results of the map elements marked in the sample images. Obtain the description information of the sample map elements in the sample images.
[0038] S120: extracting features of the sample image through the prediction network to be trained to obtain sample image features.
[0039] Figure 1b Schematic diagram of the structure of the map element prediction model provided by the embodiment of the present invention. Figure 1b As shown, the prediction network to be trained may include an image feature encoder, a language feature encoder, a decoder and a prediction head. Specifically, the image feature encoder in the prediction network to be trained extracts features from the multi-view image and converts it into a bird's-eye view (Birds-eye-view, BEV) to obtain sample image features. Optionally, the image feature encoder can adopt the feature encoder in MapTR (MapTR: Structured Modeling and Learning for Online Vectorized HD Map Construction, an end-to-end structural framework for efficient online vectorized high-precision map construction).
[0040] S130 , performing prediction based on a first query vector to be learned for the sample map element and the sample image feature to obtain a first prediction result for the sample map element.
[0041] Based on the attention mechanism, the first query vector is used as the query vector of the decoder; the image features of the sample image are used as the key vector and value vector of the decoder to decode and obtain the first decoding result. Through the prediction head, the first decoding result is predicted to obtain the first prediction result of the sample map element, which describes the shape and position of the map element, as well as the category of the map element and its confidence.
[0042] S140. Determine a second query vector of the sample map element according to the description information of the sample map element in the sample image, and perform prediction according to the second query vector and the image features of the sample image to obtain a second prediction result of the sample map element.
[0043] Specifically, the description information of the sample map element is input into the language feature encoder for feature encoding to obtain the description features of the sample map element as the second query vector. The second query vector is constructed based on the description information of the sample map element, and is a real value query vector obtained from real information. In the embodiment of the present disclosure, the first query vector can be randomly initialized, and the first query vector is also synchronously updated and learned during the process of updating the model parameters of the map element prediction model through gradient back propagation. The second query vector is determined based on the description information of the map element. During the training process, the second query vector does not need to be updated and remains unchanged.
[0044] In the process of the map element prediction model, by introducing a second query vector determined according to the description information of the sample map element, and decoding, predicting and other processing are performed on the second query vector to obtain a second prediction result, the first query vector is guided to learn the description information of the sample map element, so that the learned first query vector can contain the description information of the sample map element, thereby improving the learning accuracy of the first query vector; moreover, in the subsequent map element prediction process, the map element prediction model can predict the map element based on the first query vector containing the description information of the map element, thereby improving the accuracy of the prediction result.
[0045] In the disclosed embodiment, the first query vector input to the decoder may include not only sample image features but also point query vectors (point query, PQ). The sample image features as instance query vectors (instance query, IQ) correspond to the entirety of the map element and contain information at the instance level of the map element. The point query vector corresponds to a point in the overall point set of the sample map element and contains point-level information of the map element. The second query vector input to the decoder may include not only coded features obtained by encoding the description information of the sample map element but also point query vectors. That is, the instance query vector in the first query vector is equivalent to the coded features obtained by encoding the description information of the sample map element in the second query vector, and the coded features may be called ground truth query vectors (ground truth query, GTQ). By introducing point query vectors in the first query vector and the second query vector, the data accuracy of the sample map element is improved, and the quality of the model is further improved.
[0046] S150. According to the first prediction result, the second prediction result and the true value detection result, the prediction network to be trained is trained to obtain a map element prediction model.
[0047] In the disclosed embodiment, the parameters of the prediction network to be trained can be adjusted according to the difference between the first prediction result and the true value detection result and the difference between the second prediction result and the true value detection result to obtain a map element prediction model.
[0048] The embodiment of the present invention obtains a sample image, a true value detection result of a sample map element in the sample image, and description information of the sample map element in the sample image; extracts features of the sample image through a prediction network to be trained; predicts a first prediction result of the sample map element based on a first query vector to be learned of the sample map element and the sample image feature; determines a second query vector of the sample map element based on the description information of the sample map element in the sample image, and predicts a second prediction result of the sample map element based on the second query vector and the image feature of the sample image; trains the prediction network to be trained based on the first prediction result, the second prediction result, and the true value detection result to obtain a map element prediction model. Through the above technical solution, in the training stage of the map element prediction model, the description information of the sample map element in the sample image is introduced, and the second query vector of the sample map element is determined based on the description information of the sample map element, which is used to guide the first query vector to learn the description information of the sample map element, so that the first query vector can contain the description information of the map element, thereby improving the accuracy of the first query vector. By introducing the description information of the sample map element, the convergence of the model is accelerated and the accuracy of the model is improved.
[0049] It should be noted that in the process of map feature processing, a trained map feature prediction model can be obtained, image features can be obtained through an image feature encoder, and based on the attention mechanism, according to the image features and the query vector of the sample map features learned during the map feature prediction model training process, decoding features are obtained through a decoder, and the decoded features are predicted through a prediction head to obtain the prediction results of the map elements. In the deployment phase, that is, the prediction phase of the map feature prediction model, there is no longer any description information of the map elements, and there is no second query vector. Only the first query vector learned in the training phase will participate in the decoding process in the decoder. The introduced language feature encoder only participates in the training phase of the map feature prediction model, plays a role in supervised training, and the time consumption in the inference phase is consistent with that of the ordinary map feature prediction model, which is conducive to the deployment and application of the model.
[0050] Figure 2 FIG. 2 is a second flow chart of a method for training a map element prediction model provided by an embodiment of the present invention. This embodiment is optimized based on the above embodiment. Figure 2 As shown, the method includes:
[0051] S210: Obtain a sample image, a true value detection result of a sample map element in the sample image, and description information of the sample map element in the sample image.
[0052] S220: extracting features of the sample image through the prediction network to be trained to obtain sample image features.
[0053] S230 , performing prediction based on the first query vector to be learned for the sample map element and the sample image feature to obtain a first prediction result of the sample map element.
[0054] S240, inputting description information of sample map elements in the sample image into a language feature encoder to obtain description features of the sample map elements; and using the description features of the sample map elements as a second query vector of the sample map elements.
[0055] Specifically, the description information of the sample map elements is input into the language feature encoder to obtain the description features of the sample map elements as the second query vector. The second query vector is constructed based on the description information of the sample map elements, and is a real-value query vector obtained from real information. The sample map element description information includes the scene information of the map element and the association information between the map element and the scene. Exemplarily, the description information of the lane may be "the scene is a one-way straight lane with five lanes, and this is the first lane on the left." The description information of the sidewalk may be "the scene is an intersection with six horizontal lanes and eight longitudinal lanes, and this is a sidewalk crossing the horizontal lane", etc.
[0056] By introducing the second query vector and the descriptive information of map elements in the model training process, the model not only learns the information of map elements in the image dimension, but also learns the information of map elements in the language dimension, and further learns the scene information where the map elements are located, thereby improving the model's effective use of global information and further improving the accuracy of the model.
[0057] The first query vector is randomly initialized. During the training process, the prediction result corresponding to the first query vector is paired with the true value information. The difference between the prediction result and the true value is defined as the loss function. During the model training process, the model weights are updated through gradient back propagation, and the first query vector is updated during the training process.
[0058] The second query vector is obtained by extracting the description information of the map element through the language feature encoder, and the language feature encoder can use an existing text feature extraction model, such as a text representation network, a large language model, etc. The second query vector is obtained from the true value, and there is no need to update the second query vector during the training process.
[0059] S250. Based on the attention mechanism, determine the query vector of the decoder in the prediction network to be trained according to the second query vector, use the image features of the sample image as the key vector and value vector of the decoder to decode to obtain a second decoding result, and predict the second decoding result to obtain a second prediction result of the sample map element.
[0060] Specifically, based on the attention mechanism, the second query vector obtained according to the description information of the sample map element is used as the query vector of the decoder in the prediction network to be trained, and the image features of the sample image are used as the key vector and value vector of the decoder to decode and obtain the second decoding result. The second decoding result is predicted through the prediction head to obtain the second prediction result of the sample map element.
[0061] After training the prediction network, the map element prediction model learns the query vector of the sample map element for use in the subsequent prediction process. In the training stage of the map element prediction model, the description information of the sample map element in the sample image is introduced to guide the training process of the first query vector, so that the first query vector can contain the description information of the map element, thereby improving the accuracy of the first query vector.
[0062] S260. According to the first prediction result, the second prediction result and the true value detection result, the prediction network to be trained is trained to obtain a map element prediction model.
[0063] In an optional implementation, the prediction network to be trained is trained according to the first prediction result, the second prediction result and the true value detection result to obtain a map element prediction model, including: generating a first loss according to the first prediction result and the true value detection result, and training the prediction network to be trained according to the first loss; and generating a second loss according to the second prediction result and the true value detection result, and training the prediction network to be trained according to the second loss.
[0064] Specifically, a first loss is generated according to the first prediction result and the true value detection result, and the parameters of the map feature prediction model are adjusted through gradient back propagation according to the first loss; a second loss is generated according to the second prediction result and the true value detection result, and the parameters of the map feature prediction model are adjusted through gradient back propagation according to the second loss. Since the second query vector is an effective use of the true value map information, it does not need to be adjusted during the training process, and can better supervise the learning of the decoder and prediction network.
[0065] The embodiment of the present invention obtains a sample image, a true value detection result of a sample map element in the sample image, and description information of the sample map element in the sample image; extracts features of the sample image through a prediction network to be trained; predicts a first prediction result of the sample map element based on a first query vector to be learned of the sample map element and the sample image feature; inputs the description information of the sample map element in the sample image into a language feature encoder to obtain a description feature of the sample map element; and uses the description feature of the sample map element as a second query vector of the sample map element. Based on the attention mechanism, the query vector of the decoder in the prediction network to be trained is determined according to the second query vector, the image feature of the sample image is used as the key vector and value vector of the decoder to decode to obtain a second decoding result, and the second decoding result is predicted to obtain a second prediction result of the sample map element. The prediction network to be trained is trained according to the first prediction result, the second prediction result, and the true value detection result to obtain a map element prediction model.
[0066] Through the technical solution described in this embodiment, in the training stage of the map element prediction model, the description information of the sample map elements in the sample image is introduced, and the second query vector of the sample map elements is determined based on the description information of the sample map elements, which is used to guide the first query vector to learn the description information of the sample map elements, so that the first query vector can contain the description information of the map elements, thereby improving the accuracy of the first query vector. By introducing the description information of the sample map elements, the convergence of the model is accelerated and the accuracy of the model is improved. Introducing the description information of the map elements during the model training process allows the model to learn not only the information of the map elements in the image dimension, but also the information of the map elements in the language dimension, and further learns the scene information where the map elements are located, thereby improving the model's effective use of global information and further improving the accuracy of the model.
[0067] Figure 3 It is a third flow chart of the training method of the map element prediction model provided by an embodiment of the present invention. This embodiment is optimized on the basis of the above-mentioned embodiment. In an optional implementation manner, the description information of the sample map element is determined by: performing scene recognition on the sample image to obtain a target scene; determining the target element type associated with the target scene based on the target scene and the association relationship between preset candidate scenes and candidate element types; and generating the description information of the sample map element based on the target scene and the target element type based on a preset map element description template. Figure 3 As shown, the method includes:
[0068] S310: Perform scene recognition on the sample image to obtain a target scene.
[0069] Specifically, the scene in the sample image is identified through the image recognition model. For example, the scene in the sample image may be a lane, an intersection, etc.
[0070] S320: Determine a target element type associated with the target scene according to the target scene and a preset association relationship between candidate scenes and candidate element types.
[0071] The association between the candidate scene and the candidate element type may be the map element type that needs to be determined in the current scene. For example, when the scene is an intersection, the map element may be a sidewalk, a safety island, and a waiting area.
[0072] S330: Based on a preset map element description template, generate description information of a sample map element according to the target scene and the target element type.
[0073] Specifically, the target scene and the target element associated with the target scene are input into the preset map element description template to generate the description information of the sample map element. The corresponding map element associated with the lane is the lane number in the lane, and the map element associated with the intersection can be the sidewalk on which road, etc. Exemplarily, the description template for the lane can be "the scene is a straight lane in a certain direction of a certain lane, and this is the number of a certain lane in a certain direction". The scene information of the target lane and the target lane are input into the description template of the lane information, and the description information of the target lane can be obtained as "the scene is a one-way straight lane with five lanes, and this is the first lane from the left". The description template for the intersection can be "the scene is an intersection of a certain lane in the horizontal direction and a certain lane in the longitudinal direction, and this is a sidewalk across the lane in the certain direction". The scene information of the target intersection and the target sidewalk are input into the description template of the intersection information, and the description information of the target intersection can be obtained as "the scene is an intersection of six lanes in the horizontal direction and eight lanes in the longitudinal direction, and this is a sidewalk across the horizontal lane". The specific content of the description template is designed according to the map element, and this embodiment does not limit this.
[0074] S340: Obtain a sample image, a true value detection result of a sample map element in the sample image, and description information of the sample map element in the sample image.
[0075] S350: extracting features of the sample image through the prediction network to be trained to obtain sample image features.
[0076] S360: Predicting the first prediction result of the sample map element according to the first query vector to be learned of the sample map element and the sample image feature.
[0077] S370. Determine a second query vector of the sample map element according to the description information of the sample map element in the sample image, and perform prediction according to the second query vector and the image features of the sample image to obtain a second prediction result of the sample map element.
[0078] S380. According to the first prediction result, the second prediction result and the true value detection result, the prediction network to be trained is trained to obtain a map element prediction model.
[0079] The embodiment of the present invention performs scene recognition on a sample image to obtain a target scene; determines a target element type associated with the target scene according to the target scene and a preset association relationship between candidate scenes and candidate element types; and generates description information of a sample map element according to the target scene and the target element type based on a preset map element description template.
[0080] Through the scheme described in this embodiment, scene recognition is performed on the image, and based on the preset map element description template, the description information of the sample map element can be quickly obtained, and the obtained description information is standardized and comprehensive, laying the foundation for the subsequent determination of the second query vector of the sample map element, ensuring that the second query vector can fully and truly reflect the information of the map element, thereby better guiding the learning process of the first query vector; the model not only learns the information of the map element in the image dimension, but also learns the information of the map element in the language dimension, and further learns the scene information of the map element, thereby improving the model's effective use of global information and further improving the accuracy of the model.
[0081] Figure 4 Schematic diagram of the structure of the training device of the map element prediction model provided by the embodiment of the present invention. Figure 4 As shown, the device comprises:
[0082] The data acquisition module 410 is used to acquire the sample image, the true value detection result of the sample map element in the sample image and the description information of the sample map element in the sample image;
[0083] An image feature extraction module 420 is used to extract features of the sample image by using a prediction network to be trained to obtain sample image features;
[0084] A first prediction result acquisition module 430, configured to obtain a first prediction result of the sample map element by predicting the first query vector to be learned of the sample map element and the sample image feature;
[0085] A second prediction result acquisition module 440 is used to determine a second query vector of the sample map element according to the description information of the sample map element in the sample image, and to perform prediction according to the second query vector and the image features of the sample image to obtain a second prediction result of the sample map element;
[0086] The model training module 450 is used to train the prediction network to be trained according to the first prediction result, the second prediction result and the true value detection result to obtain a map element prediction model.
[0087] The training device for a map element prediction model provided in an embodiment of the present invention can execute the training method for a map element prediction model provided in any embodiment of the present invention, and has functional modules and beneficial effects corresponding to the execution method.
[0088] Optionally, the device further comprises:
[0089] A description information acquisition module is used to perform scene recognition on the sample image to obtain a target scene;
[0090] Determining a target element type associated with the target scene according to the target scene and a preset association relationship between candidate scenes and candidate element types;
[0091] Based on a preset map element description template, description information of a sample map element is generated according to the target scene and the target element type.
[0092] Optionally, the second prediction result acquisition module is specifically used to:
[0093] Inputting description information of sample map elements in the sample image into a language feature encoder to obtain description features of the sample map elements;
[0094] The description feature of the sample map element is used as the second query vector of the sample map element.
[0095] The second prediction result acquisition module is further specifically used for:
[0096] Based on the attention mechanism, the query vector of the decoder in the prediction network to be trained is determined according to the second query vector, the image features of the sample image are used as the key vector and value vector of the decoder to decode to obtain a second decoding result, and the second decoding result is predicted to obtain a second prediction result of the sample map element.
[0097] Optionally, the model training module is specifically used for:
[0098] A first loss is generated according to the first prediction result and the true value detection result, and the prediction network to be trained is trained according to the first loss; and a second loss is generated according to the second prediction result and the true value detection result, and the prediction network to be trained is trained according to the second loss.
[0099] The map element prediction model training device further described can also execute the map element prediction model training method provided by any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.
[0100] Figure 5 A schematic diagram of an electronic device 50 that can be used to implement an embodiment of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices (such as helmets, glasses, watches, etc.) and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or required herein.
[0101] like Figure 5 As shown, the electronic device 50 includes at least one processor 51, and a memory connected to the at least one processor 51 in communication, such as a read-only memory (ROM) 52, a random access memory (RAM) 53, etc., wherein the memory stores a computer program that can be executed by at least one processor, and the processor 51 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 52 or the computer program loaded from the storage unit 58 to the random access memory (RAM) 53. In the RAM 53, various programs and data required for the operation of the electronic device 50 can also be stored. The processor 51, the ROM 52, and the RAM 53 are connected to each other via a bus 54. An input / output (I / O) interface 55 is also connected to the bus 54.
[0102] A number of components in the electronic device 50 are connected to the I / O interface 55, including: an input unit 56, such as a keyboard, a mouse, etc.; an output unit 57, such as various types of displays, speakers, etc.; a storage unit 58, such as a disk, an optical disk, etc.; and a communication unit 59, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 59 allows the electronic device 50 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.
[0103] The processor 51 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the processor 51 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The processor 51 executes the various methods and processes described above, such as the training method of the map element prediction model.
[0104] In some embodiments, the training method of the map element prediction model can be implemented as a computer program, which is tangibly contained in a computer-readable storage medium, such as a storage unit 58. In some embodiments, part or all of the computer program can be loaded and / or installed on the electronic device 50 via the ROM 52 and / or the communication unit 59. When the computer program is loaded into the RAM 53 and executed by the processor 51, one or more steps of the training method of the map element prediction model described above can be performed. Alternatively, in other embodiments, the processor 51 can be configured to execute the training method of the map element prediction model in any other suitable manner (for example, by means of firmware).
[0105] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.
[0106] Computer programs for implementing the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that when the computer program is executed by the processor, the functions / operations specified in the flow chart and / or block diagram are implemented. The computer program may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.
[0107] In the context of the present invention, a computer-readable storage medium may be a tangible medium that may contain or store a computer program for use by or in combination with an instruction execution system, device or equipment. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or equipment, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.
[0108] To provide interaction with a user, the systems and techniques described herein may be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or trackball) through which the user can provide input to the electronic device. Other types of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form (including acoustic input, voice input, or tactile input).
[0109] The systems and techniques described herein may be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.
[0110] A computing system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The client and server relationship is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services.
[0111] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in the present invention can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solution of the present invention can be achieved, and this document does not limit this.
[0112] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.
Claims
1. A method for training a map element prediction model, characterized in that: The method comprises: Obtaining a sample image, a true value detection result of a sample map element in the sample image, and description information of the sample map element in the sample image; Extracting features of the sample image by using the prediction network to be trained to obtain sample image features; Predicting the sample map element according to a first query vector to be learned and the sample image feature to obtain a first prediction result of the sample map element; Determine a second query vector of the sample map element according to the description information of the sample map element in the sample image, and perform prediction according to the second query vector and the image feature of the sample image to obtain a second prediction result of the sample map element; The prediction network to be trained is trained according to the first prediction result, the second prediction result and the true value detection result to obtain a map element prediction model.
2. The method according to claim 1, characterized in that Determining a second query vector of the sample map element according to the description information of the sample map element in the sample image includes: Inputting description information of sample map elements in the sample image into a language feature encoder to obtain description features of the sample map elements; The description feature of the sample map element is used as the second query vector of the sample map element.
3. The method according to claim 1, characterized in that Predicting according to the second query vector and the image feature of the sample image to obtain a second prediction result of the sample map element includes: Based on the attention mechanism, the query vector of the decoder in the prediction network to be trained is determined according to the second query vector, the image features of the sample image are used as the key vector and value vector of the decoder to decode to obtain a second decoding result, and the second decoding result is predicted to obtain a second prediction result of the sample map element.
4. The method according to claim 1, characterized in that According to the first prediction result, the second prediction result and the true value detection result, the prediction network to be trained is trained to obtain a map element prediction model, including: A first loss is generated according to the first prediction result and the true value detection result, and the prediction network to be trained is trained according to the first loss; and a second loss is generated according to the second prediction result and the true value detection result, and the prediction network to be trained is trained according to the second loss.
5. The method according to claim 1, characterized in that The description information of the sample map element is determined in the following manner: Performing scene recognition on the sample image to obtain a target scene; Determining a target element type associated with the target scene according to the target scene and a preset association relationship between candidate scenes and candidate element types; Based on a preset map element description template, description information of a sample map element is generated according to the target scene and the target element type.
6. A training device for a map element prediction model, characterized in that: The device comprises: A data acquisition module, used to acquire a sample image, a true value detection result of a sample map element in the sample image, and description information of the sample map element in the sample image; An image feature extraction module, used to extract features of the sample image through a prediction network to be trained to obtain sample image features; A first prediction result acquisition module, used for predicting the first prediction result of the sample map element according to the first query vector to be learned of the sample map element and the sample image feature; A second prediction result acquisition module, used to determine a second query vector of the sample map element according to the description information of the sample map element in the sample image, and to perform prediction according to the second query vector and the image features of the sample image to obtain a second prediction result of the sample map element; The model training module is used to train the prediction network to be trained according to the first prediction result, the second prediction result and the true value detection result to obtain a map element prediction model.
7. The device according to claim 6, characterized in that The device also includes: A description information acquisition module is used to perform scene recognition on the sample image to obtain a target scene; Determining a target element type associated with the target scene according to the target scene and a preset association relationship between candidate scenes and candidate element types; Based on a preset map element description template, description information of a sample map element is generated according to the target scene and the target element type.
8. The device according to claim 6, characterized in that The second prediction result acquisition module is specifically used for: Based on the attention mechanism, the query vector of the decoder in the prediction network to be trained is determined according to the second query vector, the image features of the sample image are used as the key vector and value vector of the decoder to decode to obtain a second decoding result, and the second decoding result is predicted to obtain a second prediction result of the sample map element.
9. An electronic device, characterized in that: The electronic device comprises: at least one processor; and a memory communicatively connected to the at least one processor; wherein, The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute a training method for a map element prediction model as described in any one of claims 1-5.
10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement a training method for a map element prediction model according to any one of claims 1 to 5 when executed.
Citation Information
Patent Citations
Evaluation method and device based on high-precision map data crowdsourcing updating system
CN112766670A
Intelligent driving target detection method based on vision-language large model fine tuning
CN118155182A
Composition aware image querying
US10503775B1
Image processing method, apparatus and device, and computer-readable storage medium
US20230316742A1
Cited By
Lane attribute creation system, creation method, and computer program product
CN120027805A
