Image annotation method and device based on three-dimensional reconstruction

Abstract

The embodiment of the present invention discloses a method and device for image annotation based on three-dimensional reconstruction, which can utilize the principle of three-dimensional reconstruction to construct a feature map based on the initial image collected by the image acquisition device in the autonomous driving vehicle. A single manual operation on the feature map, i.e., a single annotation, can be projected onto multiple two-dimensional images to obtain an annotated image, thereby greatly improving the efficiency of annotation. Moreover, by utilizing high-precision three-dimensional reconstruction, a single manual annotation can be accurately projected onto multiple two-dimensional images to achieve consistency in annotation, avoid the generation of erroneous annotations, and thereby improve the accuracy of image annotation. In addition, when using a trained target detection model for element perception, the perceived current target element can be projected onto a two-dimensional image based on the result of three-dimensional reconstruction, and the perception result of the target detection model and the projection result can be compared to determine the perception result of the target detection model, thereby evaluating the accuracy of the model.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to an image annotation method and device based on three-dimensional reconstruction. Background Art

[0002] Image recognition based on deep learning requires a large number of sample images to train the model. Before model training, each sample image needs to be labeled to obtain the labeling results.

[0003] Known sample annotation methods usually use a pre-trained model to assist, that is, the pre-trained model is used on the image to be annotated to generate an annotation box, which is then manually adjusted. However, the annotation box generated by the pre-trained model has poor stability and may have errors and omissions, which require manual verification one by one, resulting in high labor costs and low annotation efficiency in the sample annotation process when the number is huge. Therefore, in order to improve the efficiency of image annotation, an image annotation method is urgently needed. Summary of the invention

[0004] The present invention provides a method and device for image annotation based on three-dimensional reconstruction to improve the efficiency of image annotation. The specific technical solution is as follows.

[0005] In a first aspect, an embodiment of the present invention provides an image annotation method based on three-dimensional reconstruction, the method comprising:

[0006] Acquire an initial image captured by an image acquisition device in each autonomous driving vehicle;

[0007] For each frame of the initial image, performing known object detection on the initial image to obtain category information and coordinate information of each known object included in the initial image;

[0008] For each autonomous driving vehicle, three-dimensional reconstruction is performed based on the category information and coordinate information of each known object included in each partial image according to the partial images collected by the image acquisition device in the autonomous driving vehicle, so as to obtain a three-dimensional reconstruction result corresponding to the autonomous driving vehicle, wherein the three-dimensional reconstruction result includes: the observation relationship of each known object, the position and posture of each partial image, and the three-dimensional coordinates of the known object included in each partial image in the world system; the observation relationship of any known object includes: the correspondence between the known object and each partial image containing the known object;

[0009] The 3D reconstruction results corresponding to each autonomous driving vehicle are fused to obtain a feature map;

[0010] Displaying the feature map and receiving the labeling results for the target elements in the feature map; the labeling results of any target element include the three-dimensional coordinates and category information of the target element;

[0011] For any target element in the feature map, identify the target object within a preset distance range of the target element among the known objects in the feature map, search for candidate images containing the target object in each of the initial images based on the observation relationship of each of the target objects, calculate the coordinate information of the target element in each of the candidate images based on the three-dimensional coordinates of the target element and the pose of each of the candidate images, determine the target image containing the target element based on the corresponding coordinate information of the target element in each of the candidate images, and construct a labeled image containing the coordinate information and category information of each target element.

[0012] Optionally, the method further includes:

[0013] Constructing an initial network model; inputting the annotated image into the initial network model, training the initial network model, and obtaining a target detection model;

[0014] Acquire an image to be recognized and the position and posture of the image to be recognized; input the image to be recognized into the target detection model to obtain a recognition result of the image to be recognized, wherein the recognition result includes coordinate information and category information of a current target element included in the image to be recognized;

[0015] Calculating the three-dimensional coordinates of the current target element in the feature map according to the coordinate information of the current target element and the position and posture of the image to be identified;

[0016] According to the three-dimensional coordinates of the current target element in the feature map, identify the current target object within a preset distance range of the current target element among the known objects in the feature map, search for the current candidate image in which the current target object exists in each of the initial images according to the observation relationship of each of the current target objects, calculate the coordinate information corresponding to the current target element in each of the current candidate images according to the three-dimensional coordinates of the current target element and the pose of each of the current candidate images, and determine the current target image containing the current target element according to the coordinate information corresponding to the current target element in each of the current candidate images, and obtain a standard result of the image to be identified, wherein the standard result includes the coordinate information and category information of the current target element included in each of the current target images;

[0017] The recognition result of the image to be recognized is compared with the standard result to determine whether the category information in the recognition result is correct.

[0018] Optionally, the method further includes:

[0019] An image to be identified with a correct recognition result is obtained, and the target detection model is optimized according to the identified image to be identified.

[0020] Optionally, after performing known object detection on each frame of the initial image to obtain category information and coordinate information of each known object included in the initial image, the method further includes:

[0021] For each frame of initial image captured by the image acquisition device in each autonomous driving vehicle, determine whether the known object detection result of the initial image is different from the known object detection results of a preset number of frames of initial images before and after the initial image, and whether the known object detection results of the preset number of frames of initial images before and after the initial image are the same; if so, delete the initial image.

[0022] Optionally, the step of calculating the coordinate information of the target element in each candidate image according to the three-dimensional coordinates of the target element and the pose of each candidate image, and determining the target image containing the target element according to the coordinate information corresponding to the target element in each candidate image includes:

[0023] According to the three-dimensional coordinates of the target element and the position and posture of each of the candidate images, the target element is projected to obtain the coordinate information of the target element in each of the candidate images;

[0024] For any of the candidate images, it is determined whether the coordinate information of the target element in the candidate image is within the coordinate range of the candidate image. If so, the candidate image is determined as the target image containing the target element.

[0025] In a second aspect, an embodiment of the present invention provides an image annotation device based on three-dimensional reconstruction, the device comprising:

[0026] An initial image acquisition module, used to acquire an initial image acquired by an image acquisition device in each autonomous driving vehicle;

[0027] A known object detection module is used to perform known object detection on each frame of the initial image to obtain category information and coordinate information of each known object included in the initial image;

[0028] A three-dimensional reconstruction module is used to perform three-dimensional reconstruction for each autonomous driving vehicle based on partial images captured by an image acquisition device in the autonomous driving vehicle and category information and coordinate information of each known object included in each partial image, so as to obtain a three-dimensional reconstruction result corresponding to the autonomous driving vehicle, wherein the three-dimensional reconstruction result includes: an observation relationship of each known object, a position and posture of each partial image, and a three-dimensional coordinate of the known object included in each partial image in a world system; the observation relationship of any known object includes: a corresponding relationship between the known object and each partial image containing the known object;

[0029] A feature map determination module is used to fuse the 3D reconstruction results corresponding to each autonomous driving vehicle to obtain a feature map;

[0030] A target element annotation module, used to display the feature map and receive annotation results for target elements existing in the feature map; the annotation results of any target element include the three-dimensional coordinates and category information of the target element;

[0031] The annotated image determination module is used to identify, for any target element in the feature map, a target object within a preset distance range of the target element among the known objects in the feature map, search for a candidate image in which the target object exists in each of the initial images based on the observation relationship of each of the target objects, calculate the coordinate information of the target element in each of the candidate images based on the three-dimensional coordinates of the target element and the pose of each of the candidate images, determine the target image containing the target element based on the corresponding coordinate information of the target element in each of the candidate images, and construct an annotated image containing the coordinate information and category information of each target element.

[0032] Optionally, the device further comprises:

[0033] A network model building module is used to build an initial network model; and input the annotated image into the initial network model, train the initial network model, and obtain a target detection model;

[0034] An image recognition module, used to obtain an image to be recognized and the position and posture of the image to be recognized; input the image to be recognized into the target detection model to obtain a recognition result of the image to be recognized, wherein the recognition result includes coordinate information and category information of a current target element included in the image to be recognized;

[0035] A three-dimensional coordinate determination module, used to calculate the three-dimensional coordinates of the current target element in the feature map according to the coordinate information of the current target element and the position and posture of the image to be identified;

[0036] A target element projection module, used to identify the current target object within a preset distance range of the current target element among the known objects in the feature map according to the three-dimensional coordinates of the current target element in the feature map, search for the current candidate image in which the current target object exists in each of the initial images according to the observation relationship of each of the current target objects, calculate the coordinate information corresponding to the current target element in each of the current candidate images according to the three-dimensional coordinates of the current target element and the pose of each of the current candidate images, and determine the current target image containing the current target element according to the coordinate information corresponding to the current target element in each of the current candidate images, to obtain a standard result of the image to be identified, the standard result including the coordinate information and category information of the current target element included in each of the current target images;

[0037] The recognition result comparison module is used to compare the recognition result of the image to be recognized with the standard result to determine whether the category information in the recognition result is correct.

[0038] Optionally, the device further comprises:

[0039] The model optimization module is used to obtain an image to be identified with a correct recognition result, and optimize the target detection model according to the identified image to be identified.

[0040] Optionally, the device further comprises:

[0041] The initial image processing module is used to determine, for each frame of initial image captured by the image acquisition device in each autonomous driving vehicle, whether the known object detection result of the initial image is different from the known object detection results of a preset number of frames of initial images before and after the initial image, and whether the known object detection results of the preset number of frames of initial images before and after the initial image are the same; if so, delete the initial image.

[0042] Optionally, the annotated image determination module includes:

[0043] An element projection submodule, used to project the target element according to the three-dimensional coordinates of the target element and the position and posture of each candidate image, so as to obtain the coordinate information of the target element in each candidate image;

[0044] The image determination submodule is used to determine, for any of the candidate images, whether the coordinate information of the target element in the candidate image is within the coordinate range of the candidate image; if so, determine the candidate image as a target image containing the target element.

[0045] From the above content, it can be seen that the image annotation method and device based on three-dimensional reconstruction provided by the embodiment of the present invention can utilize the principle of three-dimensional reconstruction to construct a feature map based on the initial image collected by the image acquisition device in the autonomous driving vehicle. A single manual operation on the feature map, that is, a single annotation, can be projected onto multiple two-dimensional images to obtain an annotated image, thereby greatly improving the efficiency of annotation. Moreover, by utilizing high-precision three-dimensional reconstruction, a single manual annotation can be accurately projected onto multiple two-dimensional images to achieve consistency in annotation, avoid the generation of erroneous annotations, and thus improve the accuracy of image annotation. Of course, implementing any product or method of the present invention does not necessarily require achieving all of the advantages described above at the same time.

[0046] The innovative features of the embodiments of the present invention include:

[0047] 1. Using the principle of 3D reconstruction, a feature map is constructed based on the initial image collected by the image acquisition device in the autonomous driving vehicle. A single manual operation on the feature map, i.e., a single annotation, can be projected onto multiple 2D images to obtain an annotated image, thereby greatly improving the efficiency of annotation. Moreover, using high-precision 3D reconstruction, a single manual annotation can be accurately projected onto multiple 2D images, achieving annotation consistency and avoiding the generation of erroneous annotations, thereby improving the accuracy of image annotation.

[0048] 2. When using the trained target detection model for element perception, the current target element perceived can be projected on a large scale onto a two-dimensional image based on the results of three-dimensional reconstruction using high-precision camera posture, and the perception results of the target detection model can be compared with the projection results to determine whether the perception results of the target detection model are accurate, thereby evaluating the accuracy of the target detection model.

[0049] 3. Based on the comparison between the perception results and the projection results of the target detection model, the image to be identified with correct perception results can be selected, so that the target detection model can be optimized according to the selected image to be identified. Since the image to be identified may correspond to different weather conditions, ambient brightness, etc., the generalization ability of the target detection model can be improved after optimizing the target detection model based on a large number of images to be identified. When the optimized target detection model is used for element perception in the future, accurate target perception can be performed on images of different qualities, thereby improving the accuracy of the target detection model's perception.

[0050] 4. Based on the known object detection results of the initial image, the initial image with obvious detection errors is eliminated, so that three-dimensional reconstruction can be performed based on the initial image with correct detection results to obtain an accurate feature map, which can improve the accuracy of image annotation. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the prior art descriptions. Obviously, the drawings described below are only some embodiments of the present invention. For those of ordinary skill in the art, other drawings can also be obtained based on these drawings without creative work.

[0052] Figure 1 A schematic diagram of a flow chart of an image annotation method based on three-dimensional reconstruction provided by an embodiment of the present invention;

[0053] Figure 2a A schematic diagram of the result of the element annotation performed by the user in the displayed feature map;

[0054] Figure 2b , 2c They are Figure 2a Schematic diagram of the labeled image corresponding to the labeled result shown;

[0055] Figure 3 Another schematic diagram of a flow chart of an image annotation method based on three-dimensional reconstruction provided by an embodiment of the present invention;

[0056] Figure 4 A schematic diagram of the structure of an image annotation device based on three-dimensional reconstruction provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0057] The following will be combined with the accompanying drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0058] It should be noted that the terms "including" and "having" and any variations thereof in the embodiments of the present invention and the accompanying drawings are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device comprising a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products or devices.

[0059] The embodiment of the present invention discloses a method and device for image annotation based on three-dimensional reconstruction, which can improve the efficiency of image annotation. The embodiment of the present invention is described in detail below.

[0060] Figure 1A schematic flow chart of a method for annotating an image based on three-dimensional reconstruction provided by an embodiment of the present invention. The method can be applied to electronic devices. The method specifically includes the following steps.

[0061] S110: Acquire an initial image captured by an image acquisition device in each autonomous driving vehicle.

[0062] In an embodiment of the present invention, an image acquisition device, such as a camera, a driving recorder, etc., can be installed in each autonomous driving vehicle. The image acquisition device can capture images of the vehicle's driving environment according to a set period, which can be called initial images, and send them to the electronic device.

[0063] For example, a wireless connection can be established in advance between the image acquisition device and the electronic device of each autonomous driving vehicle through any wireless connection method, so that after the image acquisition device of each autonomous driving vehicle captures the initial image, each initial image can be sent to the electronic device.

[0064] In one implementation, in order to save network resources, the image acquisition device may send a certain number of initial images, such as 10 frames, 20 frames, 30 frames, etc., to the electronic device after acquiring these initial images; or, after receiving the image acquisition request sent by the electronic device, all initial images that have not yet been sent may be sent to the electronic device. This is reasonable and the embodiments of the present invention do not specifically limit this.

[0065] S120: For each frame of the initial image, perform known object detection on the initial image to obtain category information and coordinate information of each known object included in the initial image.

[0066] The above-mentioned known objects may be one or more, which is not limited in the embodiment of the present invention. Specifically, the above-mentioned known objects may include, for example, benchmarks, lane lines, landmarks, etc., which is not limited in the embodiment of the present invention.

[0067] When detecting known objects in the initial image, specifically, the electronic device can use a pre-trained neural network model to detect the known objects, or the electronic device can use any known image detection method to detect the known objects in the initial image, and the embodiments of the present invention are not limited to this.

[0068] It is understandable that whether the known object detection is performed through a neural network model or other detection algorithms, erroneous detection results may occur, and erroneous detection results will directly affect the accuracy of the 3D reconstruction results.

[0069] As an implementation method of an embodiment of the present invention, in order to improve the accuracy of three-dimensional reconstruction results and thereby improve the accuracy of image annotation, the electronic device can, after performing known object detection on each initial image, remove initial images with obvious detection errors based on the detection results.

[0070] For example, the electronic device can determine, for each initial image frame captured by the image acquisition device in each autonomous driving vehicle, whether the known object detection result of the initial image is different from the known object detection results of a preset number of frames of initial images before and after the initial image, and whether the known object detection results of the preset number of frames of initial images before and after the initial image are the same; if so, delete the initial image.

[0071] Since the initial images collected by the image acquisition devices in each autonomous driving vehicle are continuous frames, the same known object may exist in the continuous frames, so the known object detection results in the continuous frames may be the same. When the known object detection result of any frame is different from the detection results of the previous and next frames, it indicates that the known object detection result of the initial image of the frame is wrong. In this case, the initial image can be deleted.

[0072] According to the known object detection results of the initial image, the initial image with obvious detection errors is eliminated, so that three-dimensional reconstruction can be performed based on the initial image with correct detection results to obtain an accurate feature map, which can improve the accuracy of image annotation.

[0073] S130: For each autonomous driving vehicle, three-dimensional reconstruction is performed based on the category information and coordinate information of each known object included in each partial image according to the partial images captured by the image acquisition device in the autonomous driving vehicle, so as to obtain a three-dimensional reconstruction result corresponding to the autonomous driving vehicle, wherein the three-dimensional reconstruction result includes: the observation relationship of each known object, the position and posture of each partial image, and the three-dimensional coordinates of the known object included in each partial image in the world system; the observation relationship of any known object includes: the correspondence between the known object and the partial images containing the known object.

[0074] In the embodiment of the present invention, in order to improve the efficiency of image annotation, a feature map can be obtained from the initial image by means of three-dimensional reconstruction, and then annotation is performed in the feature map, and the annotation results are projected into multiple two-dimensional images to obtain multiple two-dimensional images containing the annotation results. In other words, multiple annotated images containing the annotation results can be obtained based on only one annotation, thereby improving the efficiency of image annotation, which is one of the inventive points of the present invention.

[0075] Specifically, after acquiring each initial image, the electronic device can perform three-dimensional reconstruction based on the initial image captured by the image acquisition device in each autonomous driving vehicle, which can be called a partial image, and the known object detection result in step S120 to obtain each three-dimensional reconstruction result.

[0076] For example, for the partial images collected by the image acquisition device of any autonomous driving vehicle, two frames of partial images can be initialized and selected for the first bundle adjustment (BA), and then new partial images are added cyclically for new BA optimization, until all partial images are added to the BA optimization, at which time the poses of all partial images and the visual point cloud map of the scene are obtained, that is, the three-dimensional coordinates of the known objects included in each partial image in the world system. In addition, in the three-dimensional reconstruction process, the observation relationship of any known object can also be obtained, that is, the corresponding relationship between the known object and the partial images containing the known object.

[0077] S140: Fusing the three-dimensional reconstruction results corresponding to each autonomous driving vehicle to obtain a feature map.

[0078] For example, the three-dimensional reconstruction results may be fused according to the three-dimensional coordinates of each known object included in each three-dimensional reconstruction result in the world system, so as to obtain a feature map containing each three-dimensional reconstruction result.

[0079] S150: Display the feature map, and receive the labeling results for the target elements in the feature map; the labeling result of any target element includes the three-dimensional coordinates and category information of the target element.

[0080] After obtaining the feature map, target elements can be annotated based on the feature map. The target elements are different from the known objects mentioned above and can be elements that users need to detect, such as turn signs, sidewalk signs, etc.

[0081] Specifically, the electronic device can display a feature map, so that the user can mark the location of the target element on the displayed feature map, such as drawing a frame in the area where the target element is located to obtain the three-dimensional coordinates of the target element. For example, the coordinates of the four corner points of the frame determined by the user can be used as the three-dimensional coordinates of the target element. In addition, the category information of the target element can also be marked, for example, it can include a turn sign, a sidewalk sign, etc.

[0082] S160: For any target element in the feature map, identify the target objects within the preset distance range of the target element among the known objects in the feature map. According to the observation relationships of the target objects, search for candidate images in which the target objects exist in each initial image. Calculate the coordinate information of the target element in each candidate image based on the three-dimensional coordinates of the target element and the poses of the candidate images, and determine the target images containing the target element according to the coordinate information corresponding to the target element in each candidate image. Construct an annotation image containing the coordinate information and category information of each target element.

[0083] After annotating each target element in the feature map, projection can be performed according to the annotation results of each target element to obtain multiple two-dimensional images containing the annotation results, which can be called annotation images.

[0084] Specifically, the electronic device can identify the target objects within the preset distance range of any target element in the feature map among the known objects in the feature map. The above preset distance range can be, for example, 3 meters, 5 meters, 8 meters, etc. Then, according to the observation relationships of the target objects, search for candidate images in which the target objects exist in each initial image, that is, search for the images to be projected. After finding the candidate images, calculate the coordinate information of the target element in each candidate image based on the three-dimensional coordinates of the target element and the poses of the candidate images, that is, calculate the image point coordinates corresponding to each target element in each candidate image. Further, determine the target images containing the target element according to the coordinate information corresponding to the target element in each candidate image, and construct an annotation image containing the coordinate information and category information of each target element.

[0085] For example, the electronic device can project the target element based on the three-dimensional coordinates of the target element and the poses of the candidate images to obtain the coordinate information of the target element in each candidate image; and for any candidate image, determine whether the coordinate information of the target element in the candidate image is within the coordinate range of the candidate image. If so, determine the candidate image as the target image containing the target element.

[0086] In a specific embodiment, as Figure 2a shown, the user can mark the character "yong" 210 of the road sign in the displayed feature map to obtain the coordinates of the 4 corner points of the marked border. Through projection, annotation images such as Figure 2b 、 2c can be obtained, and the corresponding annotation results are 220 and 230.

[0087] As can be seen from the above content, this embodiment can use the principle of three-dimensional reconstruction to construct a feature map based on the initial image collected by the image acquisition device in the autonomous driving vehicle. A single manual operation on the feature map, i.e., a single annotation, can be projected onto multiple two-dimensional images to obtain an annotated image, thereby greatly improving the efficiency of annotation. Moreover, by using high-precision three-dimensional reconstruction, a single manual annotation can be accurately projected onto multiple two-dimensional images to achieve consistency in annotation, avoid the generation of erroneous annotations, and thus improve the accuracy of image annotation.

[0088] As an implementation method of the embodiment of the present invention, after the electronic device obtains the annotated image, it can also perform model training according to the annotated image. Specifically, Figure 3 As shown, the following steps may be included:

[0089] S310: construct an initial network model; input the annotated image into the initial network model, train the initial network model, and obtain a target detection model.

[0090] The convolutional neural network model has the characteristics of fast operation speed and accurate calculation, etc. In the embodiment of the present invention, in order to improve the efficiency and accuracy of element detection, the target detection model can be obtained by pre-training according to the annotated image.

[0091] The above-mentioned target detection model can be a convolutional neural network model. The convolutional neural network is a feedforward neural network. The artificial neurons in the convolutional neural network can respond to surrounding units within a part of the coverage range, and have excellent performance in large-scale data processing. Convolutional neural networks are generally composed of network layers such as convolutional layers, pooling layers, nonlinear layers and fully connected layers. Of course, the target detection model in this embodiment can also be a full convolutional neural network model, that is, a convolutional neural network that does not include a fully connected layer. The embodiment of the present invention does not limit the type and / or structure of the target detection model.

[0092] For example, any known method can be used to train the preset initial network model, which is not limited in the embodiment of the present invention. The specific object detection model training process can be performed on an existing deep learning network platform or through a built program framework, which is not limited here.

[0093] S320: Acquire an image to be recognized and a position and posture of the image to be recognized; input the image to be recognized into a target detection model to obtain a recognition result of the image to be recognized, wherein the recognition result includes coordinate information and category information of a current target element included in the image to be recognized.

[0094] After the target detection model is trained, when performing element detection on the image to be identified, the image to be identified is input into the pre-trained target detection model, and the target detection model can output the coordinate information and category information of the current target element corresponding to the image to be identified.

[0095] In one implementation, the target detection model may output the weights corresponding to each category information of the current target element, and the electronic device may use the category information with the highest weight as the category information of the current target element.

[0096] S330: Calculate the three-dimensional coordinates of the current target element in the feature map according to the coordinate information of the current target element and the position and posture of the image to be identified.

[0097] That is to say, the current target element can be projected into the three-dimensional space and the corresponding three-dimensional coordinates can be determined. For example, when the target detection model outputs the coordinates of the four corner points of the frame where the current target element is located, the projection coordinates of the four corner point coordinates in the feature map can be calculated respectively as the three-dimensional coordinates of the current target element in the feature map.

[0098] S340: According to the three-dimensional coordinates of the current target element in the feature map, identify the current target object within a preset distance range of the current target element among the known objects in the feature map, search for the current candidate image in which the current target object exists in each initial image according to the observation relationship of each current target object, calculate the corresponding coordinate information of the current target element in each current candidate image according to the three-dimensional coordinates of the current target element and the pose of each current candidate image, and determine the current target image containing the current target element according to the corresponding coordinate information of the current target element in each current candidate image, and obtain a standard result of the image to be identified, the standard result including the coordinate information and category information of the current target element included in each current target image.

[0099] Specifically, the electronic device can identify the current target object within a preset distance range of the current target element among the known objects in the feature map. The preset distance range can be, for example, 3 meters, 5 meters, 8 meters, etc. Afterwards, the current candidate image in which the current target object exists can be searched in each initial image according to the observation relationship of each current target object, that is, the image to be projected can be searched. After finding the current candidate image, the coordinate information of the current target element in each current candidate image can be calculated according to the three-dimensional coordinates of the current target element and the position and posture of each current candidate image, that is, the coordinates of the image point corresponding to the current target element in each current candidate image can be calculated. Further, the current target image containing the current target element can be determined according to the coordinate information corresponding to the current target element in each current candidate image to obtain a standard result of the image to be identified.

[0100] For example, the electronic device can project the current target element according to the three-dimensional coordinates of the current target element and the posture of each current candidate image to obtain the coordinate information of the current target element in each current candidate image; and for any current candidate image, determine whether the coordinate information of the current target element in the current candidate image is within the coordinate range of the current candidate image; if so, determine the current candidate image as the current target image containing the current target element.

[0101] S350: Compare the recognition result of the image to be recognized with the standard result to determine whether the category information in the recognition result is correct.

[0102] For example, the electronic device may determine whether the category information of the recognition result is completely identical to the category information of the standard result, and if not, determine that the category information in the recognition result is incorrect.

[0103] In an embodiment of the present invention, when using a trained target detection model to perceive elements, the perceived current target elements can be projected on a large scale onto a two-dimensional image based on the results of three-dimensional reconstruction through high-precision camera posture, and the perception results of the target detection model and the projection results can be compared to determine whether the perception results of the target detection model are accurate, thereby evaluating the accuracy of the target detection model.

[0104] As an implementation method of the embodiment of the present invention, it is also possible to obtain an image to be identified with a correct recognition result, and optimize the target detection model according to the identified image to be identified.

[0105] That is, the target detection model can be retrained according to the identified object to be identified, and the trained target detection model can be used to update the previously stored target detection model. The specific training method can be similar to the training method of the target detection model described above, and the embodiments of the present invention will not be repeated here.

[0106] Based on the comparison between the perception results and the projection results of the target detection model, the image to be identified with correct perception results can be selected, so that the target detection model can be optimized according to the selected image to be identified. Since the image to be identified may correspond to different weather conditions, ambient brightness, etc., the generalization ability of the target detection model can be improved after optimizing the target detection model based on a large number of images to be identified. When the optimized target detection model is used for element perception in the future, accurate target perception can be performed on images of different qualities, thereby improving the accuracy of perception of the target detection model.

[0107] like Figure 4 As shown, it shows a schematic diagram of the structure of an image annotation device based on three-dimensional reconstruction provided by an embodiment of the present invention, the device comprising:

[0108] An initial image acquisition module 410 is used to acquire an initial image acquired by an image acquisition device in each autonomous driving vehicle;

[0109] A known object detection module 420 is used to perform known object detection on each frame of the initial image to obtain category information and coordinate information of each known object included in the initial image;

[0110] The three-dimensional reconstruction module 430 is used to perform three-dimensional reconstruction for each autonomous driving vehicle based on the partial images captured by the image acquisition device in the autonomous driving vehicle and the category information and coordinate information of each known object included in each of the partial images, so as to obtain a three-dimensional reconstruction result corresponding to the autonomous driving vehicle, wherein the three-dimensional reconstruction result includes: the observation relationship of each of the known objects, the position and posture of each of the partial images, and the three-dimensional coordinates of the known objects included in each of the partial images in the world system; the observation relationship of any of the known objects includes: the correspondence between the known object and each of the partial images containing the known object;

[0111] A feature map determination module 440 is used to fuse the three-dimensional reconstruction results corresponding to each autonomous driving vehicle to obtain a feature map;

[0112] The target element annotation module 450 is used to display the feature map and receive annotation results for the target elements in the feature map; the annotation results of any target element include the three-dimensional coordinates and category information of the target element;

[0113] The annotated image determination module 460 is used to identify, for any target element in the feature map, a target object within a preset distance range of the target element among the known objects in the feature map, search for a candidate image in which the target object exists in each of the initial images based on the observation relationship of each of the target objects, calculate the coordinate information of the target element in each of the candidate images based on the three-dimensional coordinates of the target element and the pose of each of the candidate images, determine the target image containing the target element based on the corresponding coordinate information of the target element in each of the candidate images, and construct an annotated image containing the coordinate information and category information of each target element.

[0114] Optionally, the device further comprises:

[0115] A network model building module is used to build an initial network model; and input the annotated image into the initial network model, train the initial network model, and obtain a target detection model;

[0116] An image recognition module, used to obtain an image to be recognized and the position and posture of the image to be recognized; input the image to be recognized into the target detection model to obtain a recognition result of the image to be recognized, wherein the recognition result includes coordinate information and category information of a current target element included in the image to be recognized;

[0117] A three-dimensional coordinate determination module, used to calculate the three-dimensional coordinates of the current target element in the feature map according to the coordinate information of the current target element and the position and posture of the image to be identified;

[0118] A target element projection module, used to identify the current target object within a preset distance range of the current target element among the known objects in the feature map according to the three-dimensional coordinates of the current target element in the feature map, search for the current candidate image in which the current target object exists in each of the initial images according to the observation relationship of each of the current target objects, calculate the coordinate information corresponding to the current target element in each of the current candidate images according to the three-dimensional coordinates of the current target element and the pose of each of the current candidate images, and determine the current target image containing the current target element according to the coordinate information corresponding to the current target element in each of the current candidate images, to obtain a standard result of the image to be identified, the standard result including the coordinate information and category information of the current target element included in each of the current target images;

[0119] The recognition result comparison module is used to compare the recognition result of the image to be recognized with the standard result to determine whether the category information in the recognition result is correct.

[0120] Optionally, the device further comprises:

[0121] The model optimization module is used to obtain an image to be identified with a correct recognition result, and optimize the target detection model according to the identified image to be identified.

[0122] Optionally, the device further comprises:

[0123] The initial image processing module is used to determine, for each frame of initial image captured by the image acquisition device in each autonomous driving vehicle, whether the known object detection result of the initial image is different from the known object detection results of a preset number of frames of initial images before and after the initial image, and whether the known object detection results of the preset number of frames of initial images before and after the initial image are the same; if so, delete the initial image.

[0124] Optionally, the annotated image determination module includes:

[0125] An element projection submodule, used to project the target element according to the three-dimensional coordinates of the target element and the position and posture of each candidate image, so as to obtain the coordinate information of the target element in each candidate image;

[0126] The image determination submodule is used to determine, for any of the candidate images, whether the coordinate information of the target element in the candidate image is within the coordinate range of the candidate image; if so, determine the candidate image as a target image containing the target element.

[0127] As can be seen from the above content, this embodiment can use the principle of three-dimensional reconstruction to construct a feature map based on the initial image collected by the image acquisition device in the autonomous driving vehicle. A single manual operation on the feature map, i.e., a single annotation, can be projected onto multiple two-dimensional images to obtain an annotated image, thereby greatly improving the efficiency of annotation. Moreover, by using high-precision three-dimensional reconstruction, a single manual annotation can be accurately projected onto multiple two-dimensional images to achieve consistency in annotation, avoid the generation of erroneous annotations, and thus improve the accuracy of image annotation.

[0128] The above device embodiment corresponds to the method embodiment and has the same technical effect as the method embodiment. For specific description, please refer to the method embodiment. The device embodiment is obtained based on the method embodiment. For specific description, please refer to the method embodiment part, which will not be repeated here.

[0129] Those skilled in the art can understand that the accompanying drawings are only schematic diagrams of an embodiment, and the modules or processes in the accompanying drawings are not necessarily required to implement the present invention.

[0130] Those skilled in the art can understand that the modules in the device in the embodiment can be distributed in the device in the embodiment according to the description of the embodiment, or can be changed accordingly and located in one or more devices different from the embodiment. The modules in the above embodiment can be combined into one module, or can be further divided into multiple sub-modules.

[0131] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for image annotation based on three-dimensional reconstruction, characterized in that: The method comprises: Acquire an initial image captured by an image acquisition device in each autonomous driving vehicle; For each frame of the initial image, performing known object detection on the initial image to obtain category information and coordinate information of each known object included in the initial image; For each autonomous driving vehicle, three-dimensional reconstruction is performed based on the category information and coordinate information of each known object included in each partial image according to the partial images collected by the image acquisition device in the autonomous driving vehicle, so as to obtain a three-dimensional reconstruction result corresponding to the autonomous driving vehicle, wherein the three-dimensional reconstruction result includes: the observation relationship of each known object, the position and posture of each partial image, and the three-dimensional coordinates of the known object included in each partial image in the world system; the observation relationship of any known object includes: the correspondence between the known object and each partial image containing the known object; The 3D reconstruction results corresponding to each autonomous driving vehicle are fused to obtain a feature map; Displaying the feature map and receiving the labeling results for the target elements in the feature map; the labeling results of any target element include the three-dimensional coordinates and category information of the target element; For any target element in the feature map, identify the target object within a preset distance range of the target element among the known objects in the feature map, search for candidate images containing the target object in each of the initial images based on the observation relationship of each of the target objects, calculate the coordinate information of the target element in each of the candidate images based on the three-dimensional coordinates of the target element and the pose of each of the candidate images, determine the target image containing the target element based on the corresponding coordinate information of the target element in each of the candidate images, and construct a labeled image containing the coordinate information and category information of each target element.

2. The method according to claim 1, characterized in that The method further comprises: Constructing an initial network model; inputting the annotated image into the initial network model, training the initial network model, and obtaining a target detection model; Acquire an image to be recognized and the position and posture of the image to be recognized; input the image to be recognized into the target detection model to obtain a recognition result of the image to be recognized, wherein the recognition result includes coordinate information and category information of a current target element included in the image to be recognized; Calculating the three-dimensional coordinates of the current target element in the feature map according to the coordinate information of the current target element and the position and posture of the image to be identified; According to the three-dimensional coordinates of the current target element in the feature map, identify the current target object within a preset distance range of the current target element among the known objects in the feature map, search for the current candidate image in which the current target object exists in each of the initial images according to the observation relationship of each of the current target objects, calculate the coordinate information corresponding to the current target element in each of the current candidate images according to the three-dimensional coordinates of the current target element and the pose of each of the current candidate images, and determine the current target image containing the current target element according to the coordinate information corresponding to the current target element in each of the current candidate images, and obtain a standard result of the image to be identified, wherein the standard result includes the coordinate information and category information of the current target element included in each of the current target images; The recognition result of the image to be recognized is compared with the standard result to determine whether the category information in the recognition result is correct.

3. The method according to claim 2, characterized in that The method further comprises: An image to be identified with a correct recognition result is obtained, and the target detection model is optimized according to the identified image to be identified.

4. The method according to any one of claims 1 to 3, characterized in that: After performing known object detection on each frame of the initial image to obtain category information and coordinate information of each known object included in the initial image, the method further includes: For each frame of initial image captured by the image acquisition device in each autonomous driving vehicle, determine whether the known object detection result of the initial image is different from the known object detection results of a preset number of frames of initial images before and after the initial image, and whether the known object detection results of the preset number of frames of initial images before and after the initial image are the same; if so, delete the initial image.

5. The method according to any one of claims 1 to 4, characterized in that: The step of calculating the coordinate information of the target element in each candidate image according to the three-dimensional coordinates of the target element and the position and posture of each candidate image, and determining the target image containing the target element according to the coordinate information corresponding to the target element in each candidate image comprises: According to the three-dimensional coordinates of the target element and the position and posture of each of the candidate images, the target element is projected to obtain the coordinate information of the target element in each of the candidate images; For any of the candidate images, it is determined whether the coordinate information of the target element in the candidate image is within the coordinate range of the candidate image. If so, the candidate image is determined as the target image containing the target element.

6. An image annotation device based on three-dimensional reconstruction, characterized in that: The device comprises: An initial image acquisition module, used to acquire an initial image acquired by an image acquisition device in each autonomous driving vehicle; A known object detection module is used to perform known object detection on each frame of the initial image to obtain category information and coordinate information of each known object included in the initial image; A three-dimensional reconstruction module is used to perform three-dimensional reconstruction for each autonomous driving vehicle based on partial images captured by an image acquisition device in the autonomous driving vehicle and category information and coordinate information of each known object included in each partial image, so as to obtain a three-dimensional reconstruction result corresponding to the autonomous driving vehicle, wherein the three-dimensional reconstruction result includes: an observation relationship of each known object, a position and posture of each partial image, and a three-dimensional coordinate of the known object included in each partial image in a world system; the observation relationship of any known object includes: a corresponding relationship between the known object and each partial image containing the known object; A feature map determination module is used to fuse the 3D reconstruction results corresponding to each autonomous driving vehicle to obtain a feature map; A target element annotation module, used to display the feature map and receive annotation results for target elements existing in the feature map; the annotation results of any target element include the three-dimensional coordinates and category information of the target element; The annotated image determination module is used to identify, for any target element in the feature map, a target object within a preset distance range of the target element among the known objects in the feature map, search for a candidate image in which the target object exists in each of the initial images based on the observation relationship of each of the target objects, calculate the coordinate information of the target element in each of the candidate images based on the three-dimensional coordinates of the target element and the pose of each of the candidate images, determine the target image containing the target element based on the corresponding coordinate information of the target element in each of the candidate images, and construct an annotated image containing the coordinate information and category information of each target element.

7. The device according to claim 6, characterized in that The device also includes: A network model building module is used to build an initial network model; and input the annotated image into the initial network model, train the initial network model, and obtain a target detection model; An image recognition module, used to obtain an image to be recognized and the position and posture of the image to be recognized; input the image to be recognized into the target detection model to obtain a recognition result of the image to be recognized, wherein the recognition result includes coordinate information and category information of a current target element included in the image to be recognized; A three-dimensional coordinate determination module, used to calculate the three-dimensional coordinates of the current target element in the feature map according to the coordinate information of the current target element and the position and posture of the image to be identified; A target element projection module, used to identify the current target object within a preset distance range of the current target element among the known objects in the feature map according to the three-dimensional coordinates of the current target element in the feature map, search for the current candidate image in which the current target object exists in each of the initial images according to the observation relationship of each of the current target objects, calculate the coordinate information corresponding to the current target element in each of the current candidate images according to the three-dimensional coordinates of the current target element and the pose of each of the current candidate images, and determine the current target image containing the current target element according to the coordinate information corresponding to the current target element in each of the current candidate images, to obtain a standard result of the image to be identified, the standard result including the coordinate information and category information of the current target element included in each of the current target images; The recognition result comparison module is used to compare the recognition result of the image to be recognized with the standard result to determine whether the category information in the recognition result is correct.

8. The device according to claim 7, characterized in that The device also includes: The model optimization module is used to obtain an image to be identified with a correct recognition result, and optimize the target detection model according to the identified image to be identified.

9. The device according to any one of claims 6 to 8, characterized in that: The device also includes: The initial image processing module is used to determine, for each frame of initial image captured by the image acquisition device in each autonomous driving vehicle, whether the known object detection result of the initial image is different from the known object detection results of a preset number of frames of initial images before and after the initial image, and whether the known object detection results of the preset number of frames of initial images before and after the initial image are the same; if so, delete the initial image.

10. The device according to any one of claims 6 to 9, characterized in that: The annotated image determination module comprises: An element projection submodule, used to project the target element according to the three-dimensional coordinates of the target element and the position and posture of each candidate image, so as to obtain the coordinate information of the target element in each candidate image; The image determination submodule is used to determine, for any of the candidate images, whether the coordinate information of the target element in the candidate image is within the coordinate range of the candidate image; if so, determine the candidate image as a target image containing the target element.

Images