An image data processing method, apparatus and electronic device

Abstract

Embodiments of the present application disclose an image processing method, device and electronic equipment, through marking a first sensor image, determining a first image marking result, determining corresponding two-dimensional marking coordinates according to the first image marking result, pre-processing a second sensor image according to the two-dimensional marking coordinates, obtaining a pre-processed image, marking the pre-processed image, and determining a second image marking result of the second sensor image. Thus, in the embodiments, the first sensor image marking result is used as prior information to automatically mark the second sensor image, the same target object in different sensor images is paired automatically, and the efficiency of image marking and the accuracy of target detection result are improved.

Description

Technical Field

[0001] This invention relates to the field of computer technology, and more specifically, to an image processing method, apparatus, and electronic device. Background Technology

[0002] The more input data a deep learning model receives during training, the more accurate the output during inference. Therefore, a large amount of labeled data is required during model training. In complex and ever-changing application scenarios (such as autonomous driving), the amount of data acquired daily is enormous. Currently, data can only be labeled manually, which is time-consuming, labor-intensive, and slow, making it difficult to meet the needs of model updates and iterations. Summary of the Invention

[0003] In view of this, embodiments of the present invention provide an image processing method, apparatus, and electronic device that automatically annotates a second sensor image using the annotation results of a first sensor image as prior information, thereby enabling automatic pairing of the same target object in different sensor images, improving the efficiency of image annotation and the accuracy of target detection results.

[0004] In a first aspect, embodiments of the present invention provide an image processing method, the method comprising:

[0005] Obtain the first image annotation result of the first sensor image;

[0006] Determine the corresponding two-dimensional annotation coordinates based on the first image annotation results;

[0007] The second sensor image is preprocessed according to the two-dimensional labeled coordinates to obtain the corresponding preprocessed image;

[0008] Each of the preprocessed images is labeled to determine the second image labeling result of the second sensor image.

[0009] Optionally, the first sensor image is a three-dimensional image;

[0010] The first image annotation results obtained from the first sensor image include:

[0011] The first sensor image is annotated; wherein the annotation information of the first sensor image includes the coordinates of each vertex of the three-dimensional bounding box of each target object in the image;

[0012] The labeled first sensor image is mapped to a corresponding two-dimensional image based on the parameter information of the first sensor and the second sensor. The labeling information of the two-dimensional image includes the coordinates of each vertex of the two-dimensional label box corresponding to each target in the image. The coordinates of each vertex of the two-dimensional label box are determined based on the coordinates of each vertex of the corresponding three-dimensional label box.

[0013] The annotation result of the first image is determined based on the annotation information of the two-dimensional image.

[0014] Optionally, determining the corresponding two-dimensional annotation coordinates based on the second image annotation result includes:

[0015] The coordinates determined by the maximum value of each coordinate axis and the coordinates determined by the minimum value of each coordinate axis in the coordinates of each vertex of the two-dimensional annotation box are determined as the two-dimensional annotation coordinates.

[0016] Optionally, determining the corresponding two-dimensional annotation coordinates based on the second image annotation result includes:

[0017] Determine the maximum and minimum values ​​of each coordinate axis corresponding to the coordinates of each vertex of the two-dimensional annotation box;

[0018] The maximum value of each coordinate axis is increased by a first predetermined value, and the minimum value of each coordinate axis is reduced by a second predetermined value, so that the coordinates determined by the increased maximum value of each coordinate axis and the reduced minimum value of each coordinate axis are determined as the two-dimensional labeled coordinates.

[0019] Optionally, preprocessing the second sensor image based on the stated two-dimensional labeled coordinates to obtain the corresponding preprocessed image includes:

[0020] The second sensor image is cropped according to the two-dimensional labeled coordinates to obtain an initial image;

[0021] The initial image is placed on a predetermined background image to obtain the preprocessed image.

[0022] Optionally, cropping the second sensor image based on the two-dimensional labeled coordinates to obtain an initial image includes:

[0023] The second sensor image is cropped according to the two-dimensional labeled coordinates to obtain the cropped image;

[0024] The cropped image is magnified to obtain the initial image.

[0025] Optionally, enlarging the cropped image to obtain the initial image includes:

[0026] The magnification factor is determined based on the size parameters of the second sensor image and the size parameters of the cropped image;

[0027] The cropped image is magnified according to the magnification factor to obtain the initial image.

[0028] Optionally, determining the magnification factor based on the size parameters of the second sensor image and the size parameters of the cropped image includes:

[0029] Determine the length ratio of each side of the second sensor image to the cropped image;

[0030] The magnification factor is determined based on the length ratios of each of the aforementioned factors.

[0031] Optionally, the magnification factor of the cropped image is not greater than a predetermined value.

[0032] Optionally, labeling the first sensor image includes:

[0033] The first sensor image is input into the first image annotation model for processing in order to annotate the first sensor image.

[0034] Optionally, the annotation of each of the preprocessed images to determine the second image annotation result of the second sensor image includes:

[0035] Each of the preprocessed images is input into a pre-trained second image annotation model for annotation, in order to determine the second image annotation result.

[0036] Optionally, the first sensor image is a radar sensor image, and the second sensor image is a camera sensor image.

[0037] In a second aspect, embodiments of the present invention provide an image processing apparatus, the apparatus comprising:

[0038] The first annotation unit is configured to acquire the first image annotation result of the first sensor image;

[0039] The coordinate determination unit is configured to determine the corresponding two-dimensional annotation coordinates based on the first image annotation result;

[0040] The preprocessing unit is configured to preprocess the second sensor image according to each of the two-dimensional labeled coordinates to obtain the corresponding preprocessed image;

[0041] The second annotation unit is configured to annotate each of the preprocessed images to determine the second image annotation result of the second sensor image.

[0042] Thirdly, embodiments of the present invention provide an electronic device, including a memory and a processor, wherein the memory is used to store one or more computer program instructions, wherein the one or more computer program instructions are executed by the processor to implement the method described above.

[0043] Fourthly, embodiments of the present invention provide a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the method described above.

[0044] Fifthly, embodiments of the present invention provide a computer program product that, when run on a computer, causes the computer to perform the method described above.

[0045] This invention discloses an image processing method, apparatus, and electronic device. The method involves annotating a first sensor image to determine a first image annotation result, determining corresponding two-dimensional annotation coordinates based on the first image annotation result, preprocessing a second sensor image using the two-dimensional annotation coordinates to obtain a preprocessed image, and then annotating the preprocessed image to determine a second image annotation result for the second sensor image. Therefore, this embodiment automatically annotates the second sensor image using the annotation result of the first sensor image as prior information, achieving automatic pairing of the same target object in different sensor images, thus improving the efficiency of image annotation and the accuracy of target detection results. Attached Figure Description

[0046] The above and other objects, features and advantages of the present invention will become clearer from the following description of embodiments of the invention with reference to the accompanying drawings, in which:

[0047] Figure 1 This is a flowchart of an image processing method according to an embodiment of the present invention;

[0048] Figure 2 This is a schematic diagram of an annotated radar sensor image according to an embodiment of the present invention;

[0049] Figure 3 This is a schematic diagram of an annotated two-dimensional image according to an embodiment of the present invention;

[0050] Figure 4 This is a schematic diagram of a two-dimensional annotated image according to an embodiment of the present invention;

[0051] Figure 5 This is another schematic diagram of a two-dimensional annotated image according to an embodiment of the present invention;

[0052] Figure 6 This is a schematic diagram of the second sensor image according to an embodiment of the present invention;

[0053] Figure 7 This is a schematic diagram of an initial image according to an embodiment of the present invention;

[0054] Figure 8 This is a schematic diagram of another initial image according to an embodiment of the present invention;

[0055] Figure 9 This is a schematic diagram of a preprocessed image according to an embodiment of the present invention;

[0056] Figure 10 This is a schematic diagram of the image processing process according to an embodiment of the present invention;

[0057] Figure 11 This is a schematic diagram of an image processing apparatus according to an embodiment of the present invention;

[0058] Figure 12 This is a schematic diagram of an electronic device according to an embodiment of the present invention. Detailed Implementation

[0059] The present invention is described below based on embodiments, but the invention is not limited to these embodiments. In the detailed description of the invention below, certain specific details are described in detail. Those skilled in the art will fully understand the invention even without these details. To avoid obscuring the essence of the invention, well-known methods, processes, flows, elements, and circuits are not described in detail.

[0060] Furthermore, those skilled in the art should understand that the accompanying drawings provided herein are for illustrative purposes only and are not necessarily drawn to scale.

[0061] Unless the context explicitly requires it, words such as "including" or "contains" in the instruction manual should be interpreted as including rather than exclusive or exhaustive; that is, meaning "including but not limited to".

[0062] In the description of this invention, it should be understood that the terms "first," "second," etc., are used for descriptive purposes only and should not be construed as indicating or implying relative importance. Furthermore, in the description of this invention, unless otherwise stated, "a plurality of" means two or more.

[0063] Currently, in specific application scenarios, such as autonomous driving, joint annotation using radar (Lidar) sensors and camera (Camera) sensors is commonly used to improve the accuracy of target detection. The point cloud information acquired by radar sensors reflects real physical information; the size of the same object does not change, making it easy to annotate. However, the image information acquired by camera sensors exhibits the characteristic of objects appearing larger when closer and smaller when farther away, and the target object changes with each frame, making manual annotation more difficult. Related technologies train image annotation models based on partially annotated 2D images, and then use the trained model to automatically annotate the images to be annotated, thus detecting targets in the images. Due to the complexity and variability of application scenarios, the training data for manual annotation is insufficient, resulting in low annotation accuracy, i.e., low target detection accuracy. Furthermore, joint annotation of the same target object across different sensors requires pairing processing; manually pairing the same object would consume significant manpower and resources and be inefficient.

[0064] Based on this, this embodiment provides an image processing method. Specifically, this embodiment uses the annotation results of a first sensor image as prior information to automatically annotate a second sensor image, thereby achieving automatic pairing of the same target object in different sensor images, thus improving the efficiency of image annotation and the accuracy of target detection results.

[0065] Figure 1 This is a flowchart of an image processing method according to an embodiment of the present invention. Figure 1 As shown, the image processing method in this embodiment includes the following steps:

[0066] Step S110: Obtain the first image annotation result of the first sensor image. Optionally, the first sensor is a sensor that is easy to annotate. Further optionally, the first sensor image is a three-dimensional image, and the second sensor is a two-dimensional sensor.

[0067] In one optional implementation, the first sensor image is a radar sensor image, and the first sensor is a radar sensor. Since the point cloud information acquired by the radar sensor reflects real physical information, the size of the same object does not change, making it easy to annotate and producing relatively accurate annotation results.

[0068] In one optional implementation, this embodiment annotates the first sensor image and converts the annotated first sensor image into a corresponding two-dimensional image based on the parameter information of the first and second sensors. The annotation information of the first sensor image includes the vertex coordinates of the three-dimensional bounding boxes of each target object in the image, and the annotation information of the two-dimensional image includes the vertex coordinates of the two-dimensional bounding boxes corresponding to each target in the image. Furthermore, the vertex coordinates of the two-dimensional bounding boxes of the target objects are determined based on the vertex coordinates of the three-dimensional bounding boxes of the target objects. Therefore, this embodiment can determine the annotation information of the two-dimensional image as the annotation result of the first image.

[0069] Optionally, the target object is labeled with a cuboid in the first sensor image. Further, the coordinates of the vertices of the cuboid label can be determined as the coordinates of the corresponding target object. It should be understood that this embodiment does not limit the form in which the target object is labeled in the image; it can also be determined based on the shape of the target object.

[0070] In one alternative implementation, the image annotation information may also include category information for each target object, so as to facilitate the identification of target objects in the image or the acquisition of predetermined target objects in the image.

[0071] In this embodiment, the parameter information of the first sensor and the second sensor may include the relative position and orientation relationship of the first sensor and the second sensor, as well as the intrinsic and extrinsic parameters of the first sensor and the second sensor. Optionally, this embodiment may perform coordinate system transformation (mapping) based on the above parameter information, such as translation and rotation operations, to convert the labeled first sensor image into a corresponding two-dimensional image.

[0072] Taking a radar sensor as the first sensor and a camera as the second sensor as an example, the radar sensor can be labeled manually or using a pre-trained first image labeling model; this embodiment does not impose any limitations on this. The first sensor labeling model can be any network model for target detection based on 3D point cloud data, such as the MVF (End-to-End Multi-View Fusion) model, the LaserNet model, the BirdNet network framework for LiDAR data, the LMNet convolutional neural network, the YOLO3D model, etc. This embodiment does not limit the network model for 3D target detection.

[0073] In one alternative implementation, the first image annotation model is trained through the following steps: acquiring training data, which includes annotated first sensor images, and inputting the training data into the first image annotation model for processing to adjust the model parameters, thereby obtaining the trained first image annotation model.

[0074] Figure 2 This is a schematic diagram of an annotated radar sensor image according to an embodiment of the present invention. For example... Figure 2 As shown, 3D image 2 is an annotated radar sensor image. In 3D image 2, each cuboid 21 corresponds to a target object, and the coordinates of each vertex of cuboid 21 represent the coordinate information of the corresponding target object.

[0075] Figure 3 This is a schematic diagram of an annotated two-dimensional image according to an embodiment of the present invention. For example... Figure 3 As shown, this embodiment can convert the annotated 3D image 2 into a 2D image based on the parameter information of the radar sensor and camera. Wherein, Figure 3 The two-dimensional image 3 in the example only shows a portion of the two-dimensional image corresponding to the three-dimensional image 2, and it does not correspond one-to-one with the three-dimensional image 2.

[0076] Taking target object 31 as an example, after the 3D bounding box of target object 31 is converted to 2D, a 2D bounding box 31' is formed. The vertex ah of the 2D bounding box 31' corresponds to each vertex of the 3D bounding box in the 3D image, and is obtained based on the mapping of the corresponding 3D bounding box. In this embodiment, the vertex coordinate information of vertex ah of the 2D bounding box 31' in the 2D image 3 is determined as the image annotation result corresponding to target object 31. Therefore, this embodiment can determine the first image annotation result of the first sensor image by determining the vertex coordinate information of each vertex of the 2D bounding box of each target object in the 2D image corresponding to the first sensor image.

[0077] Step S120: Determine the corresponding two-dimensional annotation coordinates based on the annotation results of the first image.

[0078] In one optional implementation, this embodiment determines the coordinates of the two-dimensional annotation by taking the maximum value of each coordinate axis corresponding to the coordinates of each vertex of the two-dimensional annotation box and the coordinates of the minimum value of each coordinate axis.

[0079] Figure 4 This is a schematic diagram of a two-dimensional annotated image according to an embodiment of the present invention. Figure 3 For example, using an xy-axis coordinate system, with the top left corner of the image as the origin, the coordinates of the vertex ah of the two-dimensional annotation box 31' are (x...). a ,y a ), (x b ,y b ), (x c ,y c ), (x d ,y d ), (x e ,y e ), (x f ,y f ), (x g ,y g ), (x h ,y h Furthermore, determine the coordinates (x, y) of vertex ah. a ,y a ), (x b ,y b ), (x c ,y c ), (x d ,y d ), (x e ,y e ), (x f ,y f ), (x g ,y g ), (x h,y h The maximum value x on the x-axis in ) max Minimum value x min and the maximum value y on the y-axis max Minimum value y min .like Figure 3 As shown, it is clear that the x-axis coordinates of vertices a and c are the same and the smallest. The x-axis coordinate of vertex a (or vertex c) is x. min The x-axis coordinates of vertices f and h are the same and the largest. The x-axis coordinate of vertex f (vertices h) is x. max The y-coordinates of vertices d and h are the same and the largest. The y-coordinate of vertex d (or vertex h) is y max The y-axis coordinates of vertices a, b, e, and f are the same and the smallest. The y-axis coordinate of vertex a (or vertex b, e, and f) is y min Furthermore, the coordinates (x, y) corresponding to the maximum values ​​of each coordinate axis in the coordinates ah of each vertex are determined. min ,y min ), and the coordinates determined by the minimum value of each coordinate axis (x) max ,y max ) is determined as two-dimensional labeled coordinates, based on the two-dimensional labeled coordinates (x min ,y min ), (x max ,y max The rectangle formed is labeled with the target object 31, such as... Figure 4 The rectangle 41 shown.

[0080] Optionally, in this embodiment, the maximum value of each coordinate axis of the two-dimensional annotation box is increased by a first predetermined value, and the minimum value of each coordinate value is decreased by a second predetermined value, so that the coordinates determined by the increased maximum value and decreased minimum value of each coordinate axis are determined as the two-dimensional annotation coordinates. Thus, this embodiment can adjust the deviation between the three-dimensional image and the two-dimensional image, further ensuring that the rectangle formed by the obtained two-dimensional annotation coordinates can completely cover the target object.

[0081] It should be understood that in this embodiment, the first predetermined value corresponding to each coordinate axis may be the same or different, the second predetermined value corresponding to each coordinate axis may be the same or different, and the first predetermined value and the second predetermined value may be the same or different.

[0082] Figure 5 This is another schematic diagram of a two-dimensional annotated image according to an embodiment of the present invention. Taking the above embodiment as an example, as follows... Figure 4As shown, the rectangle 41 does not completely encompass the target object 31. Therefore, in this embodiment, the rectangle used to label the target object 31 can be enlarged by further reducing the minimum coordinate values ​​on the x-axis and y-axis and increasing the maximum coordinate values ​​on the x-axis and y-axis. Assuming the first predetermined value for the x-axis is mx and the second predetermined value is nx, and the first predetermined value for the y-axis is my and the second predetermined value is ny, then the final determined two-dimensional label coordinates are (x... min -nx,y min -ny), (x max +mx,y max +my), based on two-dimensional labeled coordinates (x min -nx,y min -ny), (x max +mx,y max The rectangle formed by +my) marks the target object 31. For example... Figure 5 The rectangular box 51 shown can completely encompass the target object 31 within the box, thereby further improving the accuracy of image annotation and thus improving the accuracy of target detection.

[0083] Step S130: Preprocess the second sensor image according to each two-dimensional labeled coordinate to obtain a preprocessed image.

[0084] In one optional implementation, the second sensor image is cropped based on the acquired two-dimensional labeled coordinates to obtain an initial image. This initial image is then placed on a predetermined background image to obtain the preprocessed image. Since images of the same size are easier to label, and cropping results in different image sizes, this embodiment facilitates automatic labeling by placing the cropped initial image on a predetermined background image. Optionally, this embodiment uses a black image as the background image. It should be understood that any image that does not affect target detection can be used as a background image, such as a white image; this embodiment is not limited to this.

[0085] Figure 6 This is a schematic diagram of the second sensor image according to an embodiment of the present invention. Taking a camera as an example, the camera sensor image 6 is... Figure 2 The image corresponds to the 3D radar point cloud image 2. In this embodiment, the annotated 3D image 2 has been converted to a 2D image 3 based on the intrinsic and extrinsic parameters of the radar sensor and camera sensor. Obviously, the 2D image 3 and the camera sensor image 6 are in the same coordinate system. Therefore, the 2D annotated coordinates of each target object obtained in step S120 can be used to crop the second sensor image to obtain the initial image corresponding to each target.

[0086] Figure 7It is a schematic diagram of an initial image in an embodiment of the present invention. Taking the target object 31 in the camera sensor image 6 as an example, based on the two-dimensional annotation coordinates corresponding to the target object 31 being (x min -nx, y min -ny), (x max +mx, y max +my), the camera sensor image 6 is sheared to obtain the initial image corresponding to the target object 31, such as the initial image 7 shown Figure 7 as follows.

[0087] Further optionally, in this embodiment, the second sensor image is sheared according to the two-dimensional annotation coordinates of each target object to obtain the sheared image, and each sheared image is enlarged to obtain the initial image corresponding to each target object respectively. Since the image size obtained based on the two-dimensional annotation coordinates of the target object is too small and not convenient for image annotation and target detection, therefore, the sheared image can be enlarged by different degrees to further improve the accuracy of image annotation and target detection.

[0088] In an optional implementation manner, the magnification factor is determined according to the size parameters of the second sensor image and the size parameters of the sheared image, and the sheared image is enlarged according to the magnification factor to obtain the corresponding initial image. Optionally, the length ratios of each side of the second sensor image and the sheared image are determined, and the magnification factor is determined according to each length ratio.

[0089] In an optional implementation manner, the length and width of the second sensor image are image_height and image_width respectively, and the length and width of the sheared image are H and W respectively. The length ratios of each side of the second sensor image and the sheared image, image_height / H and image_width / W, are obtained.

[0090] Assuming image_height / H < image_width / W, take as the magnification factor of this sheared image.

[0091] It should be understood that this embodiment does not limit the calculation method of the magnification factor, and it can be set according to the specific application scenario. Further optionally, in this embodiment, the magnification factor does not exceed a predetermined value to avoid the enlarged image being too large and causing image information loss due to mismatch with the background image. The optional predetermined value can be 3. It should be understood that this embodiment does not limit this, and it can be determined according to the specific application scenario.

[0092] Figure 8 It is a schematic diagram of another initial image in an embodiment of the present invention. Taking Figure 7For example, in the example above, since the initial image 7 is based on two-dimensional labeled coordinates (x... min -nx,y min -ny), (x max +mx,y max +my) was cut to obtain the image; therefore, the side length of the initial image 7 is: H = ((x) max +mx)-(x min -nx)), W=((y max +my)-(y min Next, the length and width of the second sensor image 6 are obtained as image_height and image_width, respectively. Based on image_height / H and image_width / W, the corresponding magnification factor is determined using the method described above. The initial image 7 is then magnified to obtain the initial image 8.

[0093] Figure 9 This is a schematic diagram of a preprocessed image according to an embodiment of the present invention. Furthermore, in this embodiment, the initial image 8 is placed within a predetermined background image to obtain a preprocessed image, as shown in preprocessed image 9. This embodiment uses the placement of the initial image based on the upper left corner of the background image as an example for illustration. It should be understood that this embodiment does not limit the placement position of the initial image; it can be placed based on the lower left corner, upper right corner, lower right corner, or any other position.

[0094] Therefore, this embodiment uses a background image to ensure that the preprocessed images to be labeled are of consistent size, which further facilitates image labeling.

[0095] Step S140: Label each of the preprocessed images to determine the second image labeling result of the second sensor image.

[0096] In one optional implementation, each preprocessed image is input into a second image annotation model for annotation to determine the second image annotation result. The second image annotation model can be built based on any two-dimensional image object detection model, such as the R-CNN model, the YOLO model, etc., and this embodiment is not limited to this.

[0097] In one alternative implementation, the second image annotation model is trained through the following steps: acquiring training data, which includes labeled two-dimensional image data, and inputting the training data into the second image annotation model for processing to adjust the model parameters, thereby obtaining the trained second image annotation model.

[0098] This embodiment annotates a first sensor image to determine a first image annotation result. Based on the first image annotation result, it determines the corresponding two-dimensional annotation coordinates. The second sensor image is then preprocessed using these two-dimensional annotation coordinates to obtain a preprocessed image. This preprocessed image is then annotated to determine the second image annotation result for the second sensor image. Therefore, this embodiment uses the relatively accurate annotation result of the first sensor image as prior information to automatically annotate the second sensor image, achieving automatic pairing of the same target object in different sensor images. This improves the efficiency of image annotation and the accuracy of target detection results.

[0099] Figure 10 This is a schematic diagram of the image processing process according to an embodiment of the present invention. For example... Figure 10 As shown, the first sensor image 101 is annotated manually or using a first image annotation model to obtain a three-dimensional annotated image 102. The three-dimensional annotated image 102 includes information such as the coordinates of each vertex of the three-dimensional bounding box of the annotated target object. Then, the three-dimensional annotated image 102 is mapped based on the intrinsic and extrinsic parameters of the first and second sensors to obtain corresponding two-dimensional annotation information 103. The two-dimensional annotated image 103 includes a two-dimensional bounding box of the annotated target object, where the coordinates of each vertex of the two-dimensional bounding box correspond to the coordinates of each vertex of the corresponding three-dimensional bounding box. Further, the two-dimensional annotation coordinates 104 of each target object are determined based on the two-dimensional annotation information 103. Specifically, the two-dimensional annotation coordinates 104 of the target object are determined based on the minimum and maximum coordinate values ​​of the coordinate axes of each vertex of the two-dimensional bounding box of the target object in the two-dimensional annotation information 103. For specific details, please refer to the above embodiment, which will not be repeated here.

[0100] Furthermore, the second sensor image 105 is cropped according to the two-dimensional labeled coordinates 104 of each target object, and each cropped image is magnified according to the corresponding magnification factor to obtain the initial image 106 of each target object. The magnification factor can be determined based on the size of the cropped image and the size of the second sensor image; for details, please refer to the above embodiments, which will not be repeated here.

[0101] Furthermore, the initial image 106 of each target object is placed in the background image to obtain the preprocessed image 107 of each target object. This ensures that the preprocessed images 107 of each target object are of the same size, facilitating automatic annotation using a model and improving image annotation and target detection efficiency.

[0102] Furthermore, the preprocessed images 107 of each target object are input into the second image annotation model 108 for processing to obtain the second image annotation result 109. The second image annotation result 109 includes the coordinates of the annotation boxes of each target object (e.g., the coordinates of the diagonal vertices of the rectangular annotation box) and the category information of the target object.

[0103] This embodiment annotates a first sensor image to determine a first image annotation result. Based on the first image annotation result, it determines the corresponding two-dimensional annotation coordinates. The second sensor image is then preprocessed using these two-dimensional annotation coordinates to obtain a preprocessed image. This preprocessed image is then annotated to determine the second image annotation result for the second sensor image. Therefore, this embodiment uses the relatively accurate annotation result of the first sensor image as prior information to automatically annotate the second sensor image, achieving automatic pairing of the same target object in different sensor images. This improves the efficiency of image annotation and the accuracy of target detection results.

[0104] Figure 11 This is a schematic diagram of an image processing apparatus according to an embodiment of the present invention. Figure 11 As shown, the image processing device 11 in this embodiment includes a first annotation unit 111, a coordinate determination unit 112, a preprocessing unit 113, and a second annotation unit 114.

[0105] The first annotation unit 111 is configured to acquire the first image annotation result of the first sensor image.

[0106] In one alternative implementation, the first sensor image is a three-dimensional image. The first annotation unit 111 includes an annotation subunit, a mapping subunit, and an annotation result determination subunit.

[0107] The annotation subunit is configured to annotate the first sensor image. The annotation information of the first sensor image includes the vertex coordinates of the 3D bounding boxes of each target object in the image. The annotation subunit is further configured to input the first sensor image into a first image annotation model for processing, thereby annotating the first sensor image.

[0108] The mapping subunit is configured to map the annotated first sensor image into a corresponding two-dimensional image based on the parameter information of the first and second sensors. The annotation information of the two-dimensional image includes the vertex coordinates of each vertex of the two-dimensional bounding box corresponding to each target in the image, and the vertex coordinates of each vertex of the two-dimensional bounding box are determined based on the vertex coordinates of the corresponding three-dimensional bounding box.

[0109] The annotation result determination subunit is configured to determine the annotation result of the first image based on the annotation information of the two-dimensional image.

[0110] The coordinate determination unit 112 is configured to determine the corresponding two-dimensional annotation coordinates based on the first image annotation result. In an optional implementation, the coordinate determination unit 112 includes a first coordinate determination subunit. The first coordinate determination subunit is configured to determine the coordinates determined by the maximum value of each coordinate axis and the coordinates determined by the minimum value of each coordinate axis among the coordinates of each vertex of the two-dimensional annotation box as the two-dimensional annotation coordinates.

[0111] In another optional implementation, the coordinate determination unit 112 includes an extremum determination subunit and a second coordinate determination subunit. The extremum determination subunit is configured to determine the maximum and minimum values ​​of each coordinate axis corresponding to the coordinates of each vertex of the two-dimensional annotation box. The second coordinate determination subunit is configured to increase the maximum values ​​of each coordinate axis by a first predetermined value and decrease the minimum values ​​of each coordinate axis by a second predetermined value, so that the coordinates determined by the increased maximum values ​​and decreased minimum values ​​of each coordinate axis are determined as the two-dimensional annotation coordinates.

[0112] The preprocessing unit 113 is configured to preprocess the second sensor image according to each of the two-dimensional labeled coordinates to obtain the corresponding preprocessed image.

[0113] In one alternative implementation, the preprocessing unit 113 includes a cutting subunit and a processing subunit.

[0114] The cropping subunit is configured to crop the second sensor image according to the two-dimensional labeled coordinates to obtain an initial image. The processing subunit is configured to place the initial image on a predetermined background image to obtain the preprocessed image.

[0115] In one alternative implementation, the shearing subunit includes a shearing module and an amplification module.

[0116] The cropping module is configured to crop the second sensor image according to the two-dimensional labeled coordinates to obtain a cropped image. The magnification module is configured to magnify the cropped image to obtain the initial image.

[0117] In one alternative implementation, the amplification module includes an amplification factor determination submodule and an amplification submodule.

[0118] The magnification factor determination submodule is configured to determine a magnification factor based on the size parameters of the second sensor image and the size parameters of the cropped image. The magnification submodule is configured to magnify the cropped image according to the magnification factor to obtain the initial image.

[0119] In one optional implementation, the magnification factor determination submodule is further configured to determine the length ratio of each side of the second sensor image and the cropped image, and determine the magnification factor based on each of the length ratios. Optionally, the magnification factor of the cropped image is not greater than a predetermined value.

[0120] The second annotation unit 114 is configured to annotate each of the preprocessed images to determine a second image annotation result for the second sensor image. Optionally, the second annotation unit 114 is further configured to input each of the preprocessed images into a pre-trained second image annotation model for annotation to determine the second image annotation result.

[0121] In one alternative implementation, the first sensor image is a radar sensor image, and the second sensor image is a camera sensor image.

[0122] This embodiment annotates a first sensor image to determine a first image annotation result. Based on the first image annotation result, it determines the corresponding two-dimensional annotation coordinates. The second sensor image is then preprocessed using these two-dimensional annotation coordinates to obtain a preprocessed image. This preprocessed image is then annotated to determine the second image annotation result for the second sensor image. Therefore, this embodiment uses the relatively accurate annotation result of the first sensor image as prior information to automatically annotate the second sensor image, achieving automatic pairing of the same target object in different sensor images. This improves the efficiency of image annotation and the accuracy of target detection results.

[0123] Figure 12 This is a schematic diagram of an electronic device according to an embodiment of the present invention. (For example...) Figure 12 As shown, electronic device 12 is a general-purpose data processing device, which includes a general-purpose computer hardware structure, including at least a processor 121 and a memory 122. The processor 121 and memory 122 are connected via a bus 123. The memory 122 is adapted to store instructions or programs executable by the processor 121. The processor 121 can be a standalone microprocessor or a collection of one or more microprocessors. Thus, the processor 121 executes the instructions stored in the memory 122 to perform the method flow of the embodiments of the present invention as described above, thereby realizing data processing and control of other devices. The bus 123 connects the aforementioned components together, and also connects the aforementioned components to the display controller 124, the display device, and the input / output (I / O) device 125. The input / output (I / O) device 125 can be a mouse, keyboard, modem, network interface, touch input device, motion-sensing input device, printer, and other devices known in the art. Typically, the input / output device 125 is connected to the system via the input / output (I / O) controller 125.

[0124] Those skilled in the art will understand that embodiments of this application can be provided as methods, apparatus (devices), or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-readable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0125] This application is described with reference to flowchart illustrations of methods, apparatus (devices), and computer program products according to embodiments of this application. It should be understood that each step in the flowchart can be implemented by computer program instructions.

[0126] These computer program instructions may be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including an instruction means, the implementation process of which is described in the instruction means. Figure 1 The function specified in one or more processes.

[0127] These computer program instructions may also be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing device to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing device, produce instructions for implementing processes. Figure 1 A device for a function specified in one or more processes.

[0128] Another embodiment of the present invention relates to a non-volatile storage medium for storing a computer-readable program for use by a computer to execute some or all of the above-described method embodiments.

[0129] That is, those skilled in the art will understand that all or part of the steps in the methods of the above embodiments can be implemented by a program specifying the relevant hardware. This program is stored in a storage medium and includes several instructions to cause a device (which may be a microcontroller, chip, etc.) or processor to execute all or part of the steps of the methods described in the embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, a portable hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0130] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. For those skilled in the art, the present invention can be modified and varied in various ways. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principle of the present invention should be included within the scope of protection of the present invention.

Claims

1. An image processing method, characterized in that, The method includes: Obtain the first image annotation result of the first sensor image; Determine the corresponding two-dimensional annotation coordinates based on the first image annotation results; The second sensor image is preprocessed according to the two-dimensional labeled coordinates to obtain the corresponding preprocessed image; Each of the preprocessed images is labeled to determine the second image labeling result of the second sensor image; Wherein, the first sensor image is a three-dimensional image; the first image annotation result for obtaining the first sensor image includes: The first sensor image is annotated; wherein the annotation information of the first sensor image includes the vertex coordinates of the three-dimensional bounding boxes of each target object in the image, and the annotation is performed by a pre-trained first image annotation model; The labeled first sensor image is mapped to a corresponding two-dimensional image based on the parameter information of the first sensor and the second sensor. The labeling information of the two-dimensional image includes the vertex coordinates of each vertex of the two-dimensional label box corresponding to each target object in the image. The vertex coordinates of each vertex of the two-dimensional label box are determined according to the vertex coordinates of the corresponding three-dimensional label box. The parameter information of the first sensor and the second sensor includes the relative position and orientation relationship of the first sensor and the second sensor, and the internal and external parameters of the first sensor and the second sensor. The annotation result of the first image is determined based on the annotation information of the two-dimensional image; Determining the corresponding two-dimensional annotation coordinates based on the first image annotation result includes: Determine the maximum and minimum values ​​of each coordinate value in the coordinates of each vertex of the two-dimensional annotation box; increase the maximum value of each coordinate value by a first predetermined value and decrease the minimum value of each coordinate value by a second predetermined value, so that the coordinates determined by the increased maximum value and decreased minimum value of each coordinate value are determined as the two-dimensional annotation coordinates; Preprocessing the second sensor image based on the stated two-dimensional labeled coordinates to obtain the corresponding preprocessed image includes: The second sensor image is cropped according to the two-dimensional labeled coordinates to obtain an initial image; The initial image is placed on a predetermined background image to obtain the preprocessed image.

2. The method according to claim 1, characterized in that, Cropping the second sensor image based on the two-dimensional labeled coordinates to obtain an initial image includes: The second sensor image is cropped according to the two-dimensional labeled coordinates to obtain the cropped image; The cropped image is magnified to obtain the initial image.

3. The method according to claim 2, characterized in that, Enlarging the cropped image to obtain the initial image includes: The magnification factor is determined based on the size parameters of the second sensor image and the size parameters of the cropped image; The cropped image is magnified according to the magnification factor to obtain the initial image.

4. The method according to claim 3, characterized in that, Determining the magnification factor based on the size parameters of the second sensor image and the size parameters of the cropped image includes: Determine the length ratio of each side of the second sensor image to the cropped image; The magnification factor is determined based on the length ratios of each of the aforementioned factors.

5. The method according to any one of claims 2-4, characterized in that, The magnification factor of the cropped image is no greater than a predetermined value.

6. The method according to claim 1, characterized in that, Annotating the first sensor image includes: The first sensor image is input into the first image annotation model for processing in order to annotate the first sensor image.

7. The method according to claim 1, characterized in that, The second image annotation result for determining the second sensor image by annotating each of the preprocessed images includes: Each of the preprocessed images is input into a pre-trained second image annotation model for annotation, in order to determine the second image annotation result.

8. The method according to claim 1, characterized in that, The first sensor image is a radar sensor image, and the second sensor image is a camera sensor image.

9. An image processing apparatus, characterized in that, The device includes: The first annotation unit is configured to acquire the first image annotation result of the first sensor image; The coordinate determination unit is configured to determine the corresponding two-dimensional annotation coordinates based on the first image annotation result; The preprocessing unit is configured to preprocess the second sensor image according to each of the two-dimensional labeled coordinates to obtain the corresponding preprocessed image; The second annotation unit is configured to annotate each of the preprocessed images to determine the second image annotation result of the second sensor image; Wherein, the first sensor image is a three-dimensional image; the first annotation unit is further configured to annotate the first sensor image; wherein, the annotation information of the first sensor image includes the vertex coordinates of the three-dimensional bounding boxes of each target object in the image, and the annotation is performed through a pre-trained first image annotation model; the annotated first sensor image is mapped to a corresponding two-dimensional image according to the parameter information of the first sensor and the second sensor, the annotation information of the two-dimensional image includes the vertex coordinates of the two-dimensional bounding boxes corresponding to each target object in the image, the vertex coordinates of the two-dimensional bounding boxes are determined according to the vertex coordinates of the corresponding three-dimensional bounding boxes, the parameter information of the first sensor and the second sensor includes the relative position and orientation relationship of the first sensor and the second sensor, and the intrinsic and extrinsic parameters of the first sensor and the second sensor; the annotation result of the first image is determined according to the annotation information of the two-dimensional image; The coordinate determination unit is further configured to determine the maximum value and minimum value of each coordinate value in the coordinates of each vertex of the two-dimensional annotation box; increase the maximum value of each coordinate value by a first predetermined value and decrease the minimum value of each coordinate value by a second predetermined value, so as to determine the coordinates determined by the increased maximum value and decreased minimum value of each coordinate value as the two-dimensional annotation coordinates; The preprocessing unit is further configured to cut the second sensor image according to the two-dimensional labeled coordinates to obtain an initial image; and to place the initial image on a predetermined background image to obtain the preprocessed image.

10. An electronic device comprising a memory and a processor, characterized in that, The memory is used to store one or more computer program instructions, wherein the one or more computer program instructions are executed by the processor to implement the method as described in any one of claims 1-8.

11. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the method as described in any one of claims 1-8.

12. A computer program product, characterized in that, When the computer program product is run on a computer, it causes the computer to perform the method as described in any one of claims 1-8.

Images