Vehicle environment image pre-labeling method and device, electronic equipment and storage medium

Abstract

The application provides a vehicle environment image pre-labeling method and device, electronic equipment and a storage medium. The method comprises the following steps: obtaining a vehicle environment image and a mirror image corresponding to the vehicle environment image; performing target detection on the vehicle environment image and the mirror image respectively to obtain a first detection result and a second detection result; comparing detection box data of detection boxes of the same category in the first detection result and the second detection result; determining detection box data of a plurality of candidate detection boxes; performing attribute recognition on a target object corresponding to each candidate detection box to obtain an attribute value of each candidate detection box; determining detection box data of a target detection box by using the attribute value; and determining the detection box data of the target detection box and the attribute value as a pre-labeling result of the vehicle environment image. The method can automatically perform target detection and attribute recognition on the vehicle environment image, remove low-quality detection box data, effectively improve labeling accuracy and labeling efficiency, and reduce workload and cost.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of vehicle data labeling, in particular to a vehicle environment image pre-labeling method and device, electronic equipment and a storage medium. BACKGROUND

[0002] With the continuous development of artificial intelligence and automatic driving car technology, automatic driving has gradually become a new competitive point of intelligent cars. For automatic driving, perception technology is particularly important. Training an automatic driving perception model requires a large amount of accurate and real data. Therefore, the quality and reliability of automatic driving algorithms are heavily dependent on high-quality labeled data, which is a necessary condition for the application of automatic driving technology. Data pre-labeling will become the next breakthrough point in the automotive industry.

[0003] At present, data labeling work relies heavily on manual work, and the labeling efficiency is low and the cost is high. Therefore, data pre-labeling method has become a research hotspot. Related data labeling technology is aimed at the case where the adhesion requirement is very high. For example, when there is a positioning error, the pre-labeled result has a certain deviation from the true result, which needs to be adjusted, resulting in a substantial increase in the workload of adjustment, and thus affecting the labeling efficiency. SUMMARY

[0004] In view of the above-mentioned shortcomings of the prior art, the present application provides a vehicle environment image pre-labeling method, device, electronic equipment and storage medium to solve the technical problems of large adjustment workload and low labeling efficiency when the pre-labeling result has a deviation.

[0005] The vehicle environment image pre-labeling method provided by the present application comprises: obtaining a vehicle environment image and a mirror image corresponding to the vehicle environment image; performing target detection on the vehicle environment image to obtain a first detection result, and performing target detection on the mirror image to obtain a second detection result, wherein the first detection result and the second detection result each include detection frame data of a detection frame corresponding to each target object in the vehicle environment image, and the detection frame data includes a detection frame category of the detection frame; comparing the detection frame data of each detection frame in the first detection result with the detection frame data of each detection frame of the same detection frame category in the second detection result, and determining detection frame data of a plurality of candidate detection frames according to the comparison result; performing attribute recognition on a target object corresponding to each candidate detection frame to obtain an attribute value of each candidate detection frame, using the attribute value to determine detection frame data of a target detection frame from the detection frame data of all candidate detection frames, and determining the detection frame data and the attribute value of the target detection frame as a pre-labeling result of the vehicle environment image.

[0006] In an embodiment of the present application, the comparison of the bounding box data of each bounding box in the first detection result with the bounding box data of each bounding box of the same bounding box category in the second detection result comprises: pairing the bounding box data of each bounding box in the first detection result with the bounding box data of each bounding box of the same bounding box category in the second detection result by using the bounding box positions, to obtain a plurality of pairing groups, the bounding box data further comprising a bounding box position of the bounding box; determining the bounding box data of the bounding box with greater category confidence in each pairing group as the bounding box data of the candidate bounding box, to obtain the bounding box data of a plurality of candidate bounding boxes, the bounding box data further comprising a category confidence corresponding to the bounding box category.

[0007] In an embodiment of the present application, the pairing of the bounding box data of each bounding box in the first detection result with the bounding box data of each bounding box of the same bounding box category in the second detection result by using the bounding box positions comprises: determining the bounding box data of the bounding box in the first detection result as the bounding box data of a first bounding box, and determining the bounding box data of the bounding box in the second detection result as the bounding box data of a second bounding box; if the bounding box category of the first bounding box is the same as the bounding box category of the second bounding box, then calculating the intersection over union of the first bounding box and the second bounding box according to the bounding box position of the first bounding box and the bounding box position of the second bounding box; and if the intersection over union of the first bounding box and the second bounding box is greater than or equal to a preset intersection over union threshold, then determining the bounding box data of the first bounding box and the bounding box data of the second bounding box as one pairing group.

[0008] In an embodiment of the present application, the determination of the bounding box data of the target bounding box from the bounding box data of all candidate bounding boxes by using the attribute value comprises: if the attribute value of the candidate bounding box satisfies a preset condition, then deleting the bounding box data of the candidate bounding box, to determine the bounding box data of the remaining candidate bounding box as the bounding box data of the target bounding box, the attribute value comprising at least one of an occlusion degree, a blur degree, and a truncation degree, and the preset condition comprising that the occlusion degree is greater than a preset occlusion degree, or the blur degree is greater than a preset blur degree, or the truncation degree is greater than a preset truncation degree.

[0009] In an embodiment of the present application, before target detection is performed on the vehicle environment image to obtain a first detection result and target detection is performed on the mirror image to obtain a second detection result, the vehicle environment image pre-labeling method comprises: obtaining a sample data set, the sample data set comprising a plurality of sample images, real box data of at least one real box corresponding to each sample image; marking each real box as a low-quality real box or a high-quality real box according to real box data of each real box to obtain real box data of each marked real box; establishing a target detection model based on an initial neural network and an initial target detection head, taking each sample image as input and taking real box data of each marked real box corresponding to each sample image as output, training the target detection model to obtain test box data of each test object corresponding to each sample image; matching a corresponding test box for each high-quality real box as a high-quality test box, and calculating a first loss according to real box data of each high-quality real box and test box data of the corresponding high-quality test box; if the first loss is greater than a preset first loss threshold, updating parameters of the target detection model according to the first loss until the first loss is less than or equal to the preset first loss threshold to obtain a trained target detection model.

[0010] In an embodiment of the present application, matching a corresponding test box for each high-quality real box as a high-quality test box and calculating a first loss according to real box data of each high-quality real box and test box data of the corresponding high-quality test box comprises: for each sample image, performing minimum loss matching on each real box and each test box according to real box data of each real box and test box data of each test box to obtain a test box corresponding to each real box, so as to take the test box corresponding to the high-quality real box as a high-quality test box, wherein the real box data comprises a real box position and a real box category of the real box, the test box data comprises a test box position and a test box category of the test box and a category confidence corresponding to the test box category; calculating an intersection over union loss by using the real box position of each high-quality real box and the test box position of the corresponding high-quality test box, and calculating a category loss by using the real box category of each high-quality real box and the test box category and the category confidence of the corresponding high-quality test box, and determining the first loss according to the intersection over union loss and the category loss.

[0011] In an embodiment of the present application, before attribute recognition is performed on the target object corresponding to each candidate detection frame to obtain the attribute value of each candidate detection frame, the vehicle environment image pre-labeling method comprises: obtaining an attribute recognition training set, the attribute recognition training set comprising sample feature maps of a plurality of sample images, real frame data of at least one real frame corresponding to each sample image, the sample feature map of each sample image being obtained by performing feature extraction on each sample image by the trained target detection model, the real frame data comprising a real frame position and an attribute value of the real frame of the real frame; establishing an attribute recognition model based on at least one initial attribute detection head, taking the sample feature map of each sample image and the real frame position of the real frame corresponding to each sample image as input, and taking the attribute value of the real frame of the real frame corresponding to each sample image as output, training the attribute recognition model to obtain a test attribute value of the real frame corresponding to each sample image; calculating a second loss based on the real attribute value and the test attribute value of each real frame, and if the second loss is greater than a preset second loss threshold, updating the parameters of the attribute recognition model until the second loss is less than or equal to the preset second loss threshold, to obtain a trained attribute recognition model.

[0012] In an embodiment of the present application, target detection is performed on the vehicle environment image to obtain a first detection result, and target detection is performed on the mirror image to obtain a second detection result, which comprises: performing positioning detection and category detection on the target objects in the vehicle environment image by the trained target detection model to obtain detection frame data of each target object corresponding detection frame in the vehicle environment image as the first detection result; performing positioning detection and category detection on the target objects in the mirror image by the trained target detection model to obtain detection frame data of each target object corresponding detection frame in the mirror image as a second initial detection result; performing mirror image processing on the second initial detection result to obtain the second detection result.

[0013] In an embodiment of the present application, attribute recognition is performed on the target object corresponding to each candidate detection frame to obtain the attribute value of each candidate detection frame, which comprises: inputting the detection frame data of all candidate detection frames and a feature map of the vehicle environment image into the trained attribute recognition model to obtain the attribute value of each candidate detection frame, the feature map being obtained by performing feature extraction on the vehicle environment image by the trained target detection model, the attribute recognition process of the trained attribute recognition model comprising: matching a target sub-feature map of a target object corresponding to each candidate detection frame from the feature map according to the detection frame position of each candidate detection frame, and performing attribute recognition on the target sub-feature map to determine the attribute value of each candidate detection frame.

[0014] In an embodiment of the present application, after obtaining the trained attribute recognition model, the vehicle environment image pre-labeling method comprises: dividing the trained target detection model into a basic network and a target detection head, and taking the trained attribute recognition model as an attribute detection head module; connecting the basic network with the target detection head and the attribute detection head module respectively to obtain a comprehensive pre-labeling model for target detection and attribute recognition of a target object.

[0015] In an embodiment of the present application, a vehicle environment image pre-labeling device is also provided, comprising: an acquisition module configured to acquire a vehicle environment image and a mirror image corresponding to the vehicle environment image; a target detection module configured to perform target detection on the vehicle environment image to obtain a first detection result, and perform target detection on the mirror image to obtain a second detection result, wherein the first detection result and the second detection result each comprise detection box data of a detection box corresponding to each target object in the vehicle environment image, and the detection box data comprises a detection box category of the detection box; a first screening module configured to compare the detection box data of each detection box in the first detection result with the detection box data of each detection box of the same detection box category in the second detection result, and determine detection box data of a plurality of candidate detection boxes according to a comparison result; an attribute recognition module configured to perform attribute recognition on a target object corresponding to each candidate detection box to obtain an attribute value of each candidate detection box; and a second screening module configured to determine detection box data of a target detection box from the detection box data of all candidate detection boxes by using the attribute value, and determine the detection box data and the attribute value of the target detection box as a pre-labeling result of the vehicle environment image.

[0016] In an embodiment of the present application, an electronic device is also provided, which comprises: one or more processors; a storage device configured to store one or more programs, which, when executed by the one or more processors, cause the electronic device to implement the vehicle environment image pre-labeling method as described above.

[0017] In an embodiment of the present application, a computer readable storage medium having a computer program stored thereon is also provided, which, when executed by a processor of a computer, causes the computer to perform the vehicle environment image pre-labeling method as described above.

[0018] Beneficial effects of the present application: the present application provides a vehicle environment image pre-labeling method and device, electronic equipment and storage medium, which can realize automatic target detection of vehicle environment image and attribute identification of detected target object, and can remove detection frame data with poor quality and accuracy through twice screening of detection frame data, effectively improve the accuracy and labeling efficiency of pre-labeling result, and reduce the workload and cost.

[0019] It should be understood that the above general description and the following detailed description are only exemplary and explanatory, and cannot limit the present application. BRIEF DESCRIPTION OF DRAWINGS

[0020] The drawings incorporated into the specification and forming a part thereof, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the application. It is clear that the drawings in the following description are only some embodiments of the present application, and those skilled in the art can obtain other drawings from these drawings without creative labor. In the drawings:

[0021] Figure 1 is a schematic diagram of an implementation environment of a vehicle environment image pre-labeling method according to an exemplary embodiment of the present application;

[0022] Figure 2 is a flowchart of a vehicle environment image pre-labeling method according to an exemplary embodiment of the present application;

[0023] Figure 3 is a training flowchart of a target detection model according to a specific embodiment of the present application;

[0024] Figure 4 is a training flowchart of an attribute identification model according to a specific embodiment of the present application;

[0025] Figure 5 is a schematic diagram of a comprehensive pre-labeling model network structure according to another specific embodiment of the present application;

[0026] Figure 6 is a brief flowchart of pre-training model training according to another specific embodiment of the present application;

[0027] Figure 7 is a flowchart of another vehicle environment image pre-labeling method according to another specific embodiment of the present application;

[0028] Figure 8 is a block diagram of a vehicle environment image pre-labeling device according to an example embodiment of the present application;

[0029] Figure 9 A structural schematic diagram of a computer system of an electronic device suitable for implementing the embodiments of the present application is shown. DETAILED DESCRIPTION

[0030] The embodiments of the present application will be described hereinafter with reference to the accompanying drawings and preferred embodiments in detail, and those skilled in the art can easily understand other advantages and functions of the present application from the contents disclosed in the present specification. The present application can be implemented or applied in other different specific embodiments, and the details in the present specification can be modified or changed based on different views and applications without departing from the spirit of the present application. It should be understood that the preferred embodiments are only for illustrating the present application, and not for limiting the protection scope of the present application.

[0031] It should be noted that the diagrams provided in the following embodiments only schematically illustrate the basic concept of the present application, and only the components related to the present application are shown in the diagrams, not the number, shape and size of the components when actually implemented. The type, number and ratio of the components when actually implemented can be arbitrarily changed, and the layout type of the components can be more complex.

[0032] It should be noted that in the present application, "first", "second", etc. are only for distinguishing similar objects, and are not limited in order or sequence. The described "include", "have" and the like mean that the subject covered by the word includes the scope of the word shown, and is not exclusive.

[0033] It can be understood that the various numbers, step numbers and the like labels described in the present application are distinguished for convenience of description, and are not used to limit the scope of the present application. The size of the labels in the present application does not mean the order of execution, and the execution order of the processes should be determined by its function and inherent logic.

[0034] In the following description, a large number of details are discussed to provide a more thorough explanation of the embodiments of the present application, however, it is obvious to those skilled in the art that the embodiments of the present application can be implemented without these specific details, and in other embodiments, the known structures and devices are shown in the form of block diagrams rather than in the form of details, to avoid making the embodiments of the present application difficult to understand.

[0035] Embodiments of the present application respectively propose a vehicle environment image pre-labeling method, a vehicle environment image pre-labeling device, an electronic device, a computer readable storage medium, and a computer program product, which will be described in detail below.

[0036] Please refer to Figure 1 , Figure 1 is an exemplary embodiment of the present application shows a vehicle environment image pre-labeling implementation environment schematic diagram.

[0037] As Figure 1 shown, the implementation environment can include an intelligent driving car 101 and a computer device 102. Among them, the computer device 102 can be at least one of a desktop graphic processing unit (GPU) computer, a GPU computing cluster, a microcomputer, an embedded computer, a neural network computer, etc. The computer device 102 can be configured in the intelligent driving car 101, and the computer device 102 can also be a stand-alone computer device, which is not limited here. The computer device 102 can be used to automatically pre-label the vehicle environment image. The intelligent driving car 101 collects the vehicle environment image through the sensor and provides it to the computer device 102 for processing.

[0038] Illustratively, the vehicle environment image and the mirror image corresponding to the vehicle environment image are obtained, the target detection is performed on the vehicle environment image to obtain a first detection result, the target detection is performed on the mirror image to obtain a second detection result, wherein the first detection result and the second detection result each include detection data of a detection box corresponding to each target object in the vehicle environment image, and the detection data of the detection box includes a detection box category of the detection box; the detection data of each detection box in the first detection result is compared with the detection data of each detection box of the same detection box category in the second detection result, and the detection data of a plurality of candidate detection boxes is determined according to the comparison result; the attribute recognition is performed on the target object corresponding to each candidate detection box to obtain an attribute value of each candidate detection box, the detection data of a target detection box is determined from the detection data of all candidate detection boxes by using the attribute value, and the detection data of the target detection box and the attribute value are determined as the pre-labeling result of the vehicle environment image. It can be seen that the technical scheme of the embodiments of the present application can realize the automatic target detection on the vehicle environment image and the attribute recognition on the detected target object, and the detection box data with poor quality and accuracy can be removed by twice screening of the obtained detection box data, thereby effectively improving the accuracy and labeling efficiency of the pre-labeling result, and reducing the workload and cost.

[0039] It should be noted that the vehicle environment image pre-labeling method provided in the embodiments of the present application is generally executed by the computer device 102, and correspondingly, the vehicle environment image pre-labeling apparatus is generally arranged in the computer device 102.

[0040] Please refer to Figure 2 , Figure 2 is a flowchart of a vehicle environment image pre-labeling method according to an example embodiment of the present application. The method can be applied to Figure 1 the implementation environment shown in the figure and executed by the computer device 102 in the implementation environment. It should be understood that the method can also be applied to other example implementation environments and executed by devices in other implementation environments, and the present embodiment does not limit the implementation environment to which the method is applied.

[0041] As shown in Figure 2 , in an example embodiment, the vehicle environment image pre-labeling method includes at least steps S210 to S240, which are described in detail as follows.

[0042] Step S210, obtaining a vehicle environment image and a mirror image corresponding to the vehicle environment image.

[0043] In an embodiment of the present application, the vehicle environment image refers to an image of the environment around the vehicle during the driving of the vehicle, and the vehicle environment image includes at least one of a vehicle, a pedestrian, a lane line, a road barrier, a road surface, a traffic signal, an animal, etc. In the process of automatic driving, the intelligent driving car can collect the vehicle environment image in real time through image collection devices such as cameras and cameras, and provide it to the computer device, so that the computer device automatically detects and filters the target objects in the vehicle environment image, and then obtains the pre-labeling result of the vehicle environment image, to provide a basis for the decision of automatic driving.

[0044] In order to improve the accuracy of target detection, the target object in the mirror image corresponding to the vehicle environment image can also be detected, and the more accurate and higher quality detection box data is screened by comparing the detection results of the two times, wherein the mirror image is obtained based on the mirror processing of the vehicle environment image. Illustratively, the mirror processing mode can be left-right flipping or up-down flipping, which is not limited here. It should be noted that for the vehicle environment image whose target object category changes after left-right flipping, the vehicle environment image is flipped up-down to obtain the mirror image corresponding to the vehicle environment image. For example, when the target object is a left-right turn indicator, if the vehicle environment image is flipped left-right, the category of the left-right turn indicator will change, that is, the left turn indicator is detected as a right turn indicator, and the right turn indicator is detected as a left turn indicator. Similarly, for the vehicle environment image whose target object category changes after up-down flipping, the vehicle environment image is flipped left-right to obtain the mirror image corresponding to the vehicle environment image.

[0045] In step S220, the vehicle environment image is subjected to target detection to obtain a first detection result, and the mirror image is subjected to target detection to obtain a second detection result.

[0046] In an embodiment of the present application, there are various ways of target detection. Taking the vehicle environment image as an example, the vehicle environment image can be subjected to feature extraction to obtain a feature map corresponding to the vehicle environment image, and the feature map corresponding to the vehicle environment image is matched according to the feature map of a preset object to be detected, so as to identify the position and category of the target object in the vehicle environment image, and further obtain the detection box data of the detection box corresponding to the target object as the first detection result. The detection box data includes the detection box position, the detection box category and the category confidence corresponding to the detection box category of the detection box. The detection box category represents the category of the target object corresponding to the detection box, for example, the category of the target object can include vehicles, pedestrians and traffic signal lights, the category of the vehicle can include cars, trucks, motorcycles and bicycles, and the category of the traffic signal light can include red light, green light and yellow light. The detection box is represented by the detection box data, rather than actually marked in the vehicle environment image.

[0047] The process of target detection on the mirror image is the same as above, and will not be repeated here. It should be understood that after target detection on the mirror image, the detection frame data of the target object in the mirror image corresponding to the detection frame also needs to be mirrored as the second detection result. The mirroring method of the detection frame data of the mirror image should be consistent with the mirroring method of the vehicle environment image to obtain the mirror image as recorded in the foregoing embodiments, i.e., both are left-right flipping or both are up-down flipping, for example: if the mirror image is obtained by left-right flipping of the vehicle environment image, then the detection frame data of the mirror image is left-right flipped; if the mirror image is obtained by up-down flipping of the vehicle environment image, then the detection frame data of the mirror image is up-down flipped.

[0048] Therefore, the first detection result and the second detection result both include detection frame data of each target object corresponding to the detection frame in the vehicle environment image, and the detection frame data of each detection frame in the first detection result and the detection frame data of each detection frame in the second detection result may be different.

[0049] In addition, the target detection model can also be used to detect targets in the vehicle environment image or the mirror image, which is not limited here.

[0050] In an embodiment of the present application, before step S220, the vehicle environment image pre-labeling method comprises: obtaining a sample data set, the sample data set comprising a plurality of sample images, real frame data of at least one real frame corresponding to each sample image; marking each real frame as a low-quality real frame or a high-quality real frame according to the real frame data of each real frame to obtain real frame data of each marked real frame; establishing a target detection model based on an initial neural network and an initial target detection head, taking each sample image as input and taking the real frame data of the marked real frame corresponding to each sample image as output, training the target detection model to obtain test frame data of each target object corresponding to each test frame in each sample image; matching each high-quality real frame to a corresponding test frame as a high-quality test frame, and calculating a first loss according to the real frame data of each high-quality real frame and the test frame data of the corresponding high-quality test frame; if the first loss is greater than a preset first loss threshold, updating the parameters of the target detection model according to the first loss until the first loss is less than or equal to the preset first loss threshold to obtain a trained target detection model.

[0051] In this embodiment, for each sample image, the real box data of the corresponding real box is obtained based on the position, category and attribute of the target object to be detected in the sample image. The real box is represented by the real box data, rather than being actually labeled in the sample image. The target object to be detected can be determined in the sample image, and each target object to be detected is required to have a correct category label as the real box category; the target object to be detected has a real box, which is referred to as a GT Box (Ground Truth Box), and the real box can be a rectangle or other shape, which is fitted to the edge of the target object to be detected, and the occluded part is completed to determine the real box position. Illustratively, if the real box is a rectangle, at least one set of corner points of opposite corners of the real box can be used as the real box position; the target object to be detected has a correct attribute value as the real attribute value. Illustratively, the real attribute value includes at least one of the occlusion degree, the blur degree and the truncation degree.

[0052] The sample data set is subjected to data processing, and high-quality GT Boxes and low-quality GT Boxes are labeled to obtain a target detection training set, wherein any one of the following conditions satisfies the low-quality GT Box:

[0053] 1, h box <h thr or w box <w thr The GT Box with smaller size, wherein h box is the height of the GT Box, w box is the width of the GT Box, h thr is a preset height threshold, w thr is a preset width threshold, illustratively, h thr and w thr may be 5px or other numerical values, which are not limited here;

[0054] 2, the GT Box is blurred, the GT Box is severely occluded, and the GT Box is truncated at the edge of the image.

[0055] wherein the size of h box and w box may be calculated according to the real box position of the GT Box, and the blur, occlusion and truncation are obtained according to the real attribute value.

[0056] Based on the initial neural network and the initial target detection head, a target detection deep neural network based on a visual large model, i.e., a target detection model, is constructed. The target detection deep neural network is trained using a target detection training set to obtain a trained target detection model. Using the visual large model, the positioning accuracy of the model can be greatly enhanced. Illustratively, the target detection model can be a pre-trained model or other types of neural network models.

[0057] Before training, the target detection training set can also be data enhanced in the following ways: random flipping, adding random white noise, adding random disturbance to color brightness, etc., random cropping enhancement, to obtain an enhanced target detection training set. Among them, for the classes that exist label property change after left-right flipping, such as turning left and right to the traffic light, only random up-down flipping is performed, for the classes that exist label property change after up-down flipping, only random left-right flipping is performed, and for the classes that do not exist label property change after flipping, random up-down and left-right flipping can be performed.

[0058] The enhanced target detection training set is sent into the target detection deep neural network to obtain the detection frame data of the recognized detection frame, which can be referred to as the test frame data of the test frame. The real frame data of the real frame and the test frame data of the test frame can be compared, each high-quality real frame is matched with a corresponding test frame as a high-quality test frame, and a first loss is calculated according to the real frame data of each high-quality real frame and the test frame data of the corresponding high-quality test frame. If the first loss is greater than a preset first loss threshold, the parameters of the target detection model are updated according to the first loss until the first loss is less than or equal to the preset first loss threshold, or until a preset training number is reached, to obtain a trained target detection model. Illustratively, the preset first loss threshold can be 0.01 or other numerical values, and the preset training number can be 50 times or other number of times, which are not limited herein.

[0059] In one embodiment of the present application, a corresponding test box is matched for each high-quality real box as a high-quality test box, a first loss is calculated according to real box data of each high-quality real box and test box data of the corresponding high-quality test box, including: for each sample image, each real box and each test box are matched for minimum loss according to real box data of each real box and test box data of each test box, to obtain a test box corresponding to each real box, and the test box corresponding to the high-quality real box is taken as the high-quality test box, the real box data includes real box position and real box category of the real box, and the test box data includes test box position and test box category of the test box and category confidence corresponding to the test box category; an intersection over union loss is calculated by using the real box position of each high-quality real box and the test box position of the corresponding high-quality test box, and a category loss is calculated by using the real box category of each high-quality real box and the test box category and the category confidence of the corresponding high-quality test box, and the first loss is determined according to the intersection over union loss and the category loss.

[0060] In this embodiment, the Hungarian algorithm can be used to match the test box with the GT Box to obtain an optimal one-to-one matching relationship, so as to minimize the matching loss, and the matching loss includes the category loss and the IOU loss between the test box and the GT Box, and the optimal matching is denoted as The calculation method is as follows:

[0061]

[0062] wherein, is the optimal matching, is the matching loss between the true value and the test value, representing the similarity between the test box and the real box, and the calculation method of the matching loss is as follows:

[0063]

[0064] wherein, is the matching loss, is no target object, c i is the test box category of the i th target object, is the category confidence of the test box category, is the intersection over union between the real box and the test box, and the calculation method of the intersection over union is as follows:

[0065]

[0066] wherein, IOU is the intersection over union, area_inter is the intersection of the real box and the test box, and area_union is the union of the real box and the test box.

[0067] The index of the low-quality GT Box is removed from to obtain a new index That is, the low-quality GT Box is neither a positive sample nor a negative sample, and the test box matched by the new index is a high-quality test box. Finally, the first loss is calculated in the following manner:

[0068]

[0069] wherein loss1 is the first loss, is a category loss, is an intersection over union loss, a is a weight coefficient of the category loss, and b is a weight coefficient of the intersection over union loss. Illustratively, a can be 1, b can be 2, and a and b can also be other numerical values, which are not limited herein.

[0070] Please refer to Figure 3 , Figure 3 is a training flowchart of the target detection model according to an embodiment of the present application. As shown in Figure 3 , the training flowchart of the target detection model is as follows:

[0071] The target detection training set is obtained, each sample image in the target detection training set is subjected to data enhancement through mirroring, adding noise disturbance, color enhancement, etc., the target detection model is trained using the enhanced target detection training set to perform forward inference, and loss calculation is performed, including calculating the optimal match according to formula (1), formula (2) and formula (3) to one-by-one match the real frame and the test frame, and then remove the test frame corresponding to the low-quality real frame, that is, remove the low-quality box (frame), and retain the test frame corresponding to the high-quality real frame, so as to calculate the first loss according to formula (4), update the parameters of the target detection model, and continue training. When the number of training times reaches 50 or the first loss is less than or equal to 0.01, the forward inference is stopped, and the trained target detection model is obtained.

[0072] In an embodiment of the present application, step S220 includes: performing positioning detection and category detection on the target objects in the vehicle environment image by using the trained target detection model to obtain the detection frame data of each target object in the vehicle environment image, as a first detection result; performing positioning detection and category detection on the target objects in the mirror image by using the trained target detection model to obtain the detection frame data of each target object in the mirror image, as a second initial detection result; and performing mirror processing on the second initial detection result to obtain a second detection result.

[0073] In this embodiment, the vehicle environment image can be normalized and scaled to obtain a first image, the first image is input into the trained target detection model, so that the trained target detection model performs target detection on the first image, and outputs detection box data of a detection box corresponding to a target object in the first image as a first initial detection result. The first initial detection result is scaled and restored according to the size of the vehicle environment image to obtain a first detection result. The mirror image can be normalized and scaled to obtain a second image, or the first image can be mirrored to obtain a second image. The second image is input into the trained target detection model, so that the trained target detection model performs target detection on the second image, and outputs detection box data of a detection box corresponding to a target object in the second image as a second initial detection result. The second initial detection result is mirrored and scaled and restored according to the size of the vehicle environment image to obtain a second detection result.

[0074] The trained target detection model includes a basic network and a target detection head. The process of target detection performed by the trained target detection model can include: performing feature extraction on the vehicle environment image or the mirror image by the basic network to obtain a feature map of the vehicle environment image or the mirror image, and performing positioning detection and category detection on a target sub-feature map of a target object in the feature map by the target detection head to obtain detection box data of a detection box corresponding to the target object. The target detection performed by the trained target detection model on the vehicle environment image and the mirror image of the vehicle environment image can effectively improve the efficiency of target detection.

[0075] In step S230, the detection box data of each detection box in the first detection result is compared with the detection box data of each detection box of the same detection box category in the second detection result, and the detection box data of the plurality of candidate detection boxes is determined according to the comparison result.

[0076] In an embodiment of the present application, the detection box data in the first detection result and the detection box data in the second detection result can be matched to find the detection box data corresponding to the same target object in the first detection result and the detection box data corresponding to the same target object in the second detection result. The detection box data corresponding to the target object with higher quality and better accuracy is screened out by comparison and is used as the detection box data of the candidate detection box. When the detection box category of the same target object in the two detection results is different, it indicates that the quality and accuracy of the corresponding detection box data are poor. Therefore, the detection box data in the first detection result and the detection box data in the second detection result can be matched and screened according to each detection box category. This can not only reduce the calculation amount, but also screen out the detection box data of the same target object with different detection box categories in the two detection results, further ensure the accuracy of the detection box data, and also help to improve the accuracy of the pre-labeled result of the vehicle environment image.

[0077] In an embodiment of the present application, the step S230 comprises: pairing the bounding box data of each bounding box in the first detection result with the bounding box data of each bounding box of the same bounding box category in the second detection result by using the bounding box positions to obtain a plurality of pairing groups, the bounding box data further comprising the bounding box position of the bounding box; determining the bounding box data of the bounding box with greater category confidence in each pairing group as the bounding box data of the candidate bounding box to obtain the bounding box data of a plurality of candidate bounding boxes, the bounding box data further comprising the category confidence corresponding to the bounding box category.

[0078] In this embodiment, the low-quality and less-accurate initial bounding boxes are removed. For each bounding box category, the bounding box data of the corresponding bounding box in the first detection result and the bounding box data of the corresponding bounding box in the second detection result can be found by comparing the bounding box positions in the bounding box data of the bounding boxes in the first detection result with the bounding box positions in the bounding box data of the bounding boxes in the second detection result, and the bounding box data of the two bounding boxes is taken as a pairing group, thereby obtaining a plurality of pairing groups. The bounding box position comprises the corner point positions of at least one set of opposite corners of the bounding box.

[0079] The higher the category confidence corresponding to the bounding box category, the more accurate the bounding box category, and the better the quality and accuracy of the bounding box data. Conversely, the lower the category confidence corresponding to the bounding box category, the less accurate the bounding box category, and the worse the quality and accuracy of the bounding box data. Therefore, the category confidences in the two bounding box data in the same pairing group can be compared, the bounding box data of the bounding box with greater category confidence in the pairing group is determined as the bounding box data of the candidate bounding box, and the bounding box data of the bounding box with smaller category confidence is removed, thereby achieving the purpose of removing low-quality bounding boxes, further improving the accuracy of the pre-labeled result of the vehicle environment image, and using the category confidence to screen the bounding box data in each pairing group can effectively reduce the amount of calculation and improve the screening efficiency.

[0080] If the category confidences in the bounding box data of the two bounding boxes in a pairing group are the same, the bounding box data of one of the two bounding boxes is selected as the bounding box data of the candidate bounding box, and the bounding box data of the other bounding box is removed.

[0081] In an embodiment of the present application, the pairing of the bounding box data of each bounding box in the first detection result with the bounding box data of each bounding box of the same bounding box category in the second detection result by using the bounding box positions comprises: determining the bounding box data of the bounding box in the first detection result as the bounding box data of the first bounding box, and determining the bounding box data of the bounding box in the second detection result as the bounding box data of the second bounding box; if the bounding box category of the first bounding box is the same as the bounding box category of the second bounding box, calculating the intersection-over-union of the first bounding box and the second bounding box according to the bounding box position of the first bounding box and the bounding box position of the second bounding box; and if the intersection-over-union of the first bounding box and the second bounding box is greater than or equal to a preset intersection-over-union threshold, determining the bounding box data of the first bounding box and the bounding box data of the second bounding box as a pairing group.

[0082] In this embodiment, if the intersection-over-union of a first bounding box and a second bounding box is greater than or equal to a preset intersection-over-union threshold, it indicates that the target object corresponding to the first bounding box and the target object corresponding to the second bounding box are the same target object, and thus the bounding box data of the first bounding box and the bounding box data of the second bounding box can be determined as a pairing group; otherwise, it indicates that the target object corresponding to the first bounding box and the target object corresponding to the second bounding box are different target objects, or it indicates that the bounding box data of the first bounding box corresponding to the same target object and the bounding box data of the second bounding box corresponding to the same target object are quite different, which are low-quality or low-accuracy bounding box data, and thus pairing is not performed. Illustratively, the preset intersection-over-union threshold can be 0.95, or 1, or other values less than 1, which are not limited herein. By comparing the intersection-over-union of each first bounding box and each second bounding box with the preset intersection-over-union threshold, the bounding box data of the first bounding box and the bounding box data of the second bounding box corresponding to the same target object are paired, which can effectively improve the pairing efficiency and effectively remove the low-quality or low-accuracy bounding box data of the same target object in two target detections, further improving the accuracy of the pre-labeled result of the vehicle environment image.

[0083] In step S240, the attribute of the target object corresponding to each candidate bounding box is identified to obtain the attribute value of each candidate bounding box, the bounding box data of the target bounding box is determined from the bounding box data of all candidate bounding boxes by using the attribute value, and the bounding box data of the target bounding box and the attribute value are determined as the pre-labeled result of the vehicle environment image.

[0084] In an embodiment of the present application, since the target object corresponding to the candidate bounding box may have a serious occlusion, a serious truncation or a serious blur, the detection box data of the candidate bounding box has a low quality and a poor accuracy, and the detection box data of the candidate bounding box with the above conditions needs to be deleted. The attribute of the target object corresponding to the candidate bounding box can be recognized to determine the attribute value of the candidate bounding box, including at least one of the occlusion degree, the truncation degree and the blur degree. According to the attribute value, the quality level of the candidate bounding box is judged, the detection box data of the candidate bounding box with a low quality level is removed, and the detection box data of the candidate bounding box with a high quality level is determined as the detection box data of the target bounding box, which is beneficial to further improve the accuracy of the pre-labeled result of the vehicle environment image. Illustratively, the target sub-feature map of the corresponding target object can be found from the feature map of the vehicle environment image according to the detection box position in the detection box data of the candidate bounding box, or the corresponding target object can be found from the vehicle environment image according to the detection box position in the detection box data of the candidate bounding box, and then the target sub-feature map of the target object is obtained by feature extraction of the target object. Here, no limitation is made, and the attribute of the target sub-feature map of the target object is recognized to determine the attribute value of the candidate bounding box corresponding to the target object.

[0085] In an embodiment of the present application, before step S240, the vehicle environment image pre-labeling method comprises: obtaining an attribute recognition training set, the attribute recognition training set comprising sample feature maps of a plurality of sample images, real box data of at least one real bounding box corresponding to each sample image, the sample feature map of each sample image being obtained by feature extraction of each sample image by the trained target detection model, and the real box data comprising a real box position and a real box attribute value of the real bounding box; establishing an attribute recognition model based on at least one initial attribute detection head, taking the sample feature map of each sample image and the real box position of the real bounding box corresponding to each sample image as input, and taking the real box attribute value of the real bounding box corresponding to each sample image as output, training the attribute recognition model to obtain a test attribute value of the real bounding box corresponding to each sample image; calculating a second loss based on the real attribute value and the test attribute value of each real bounding box, updating the parameters of the attribute recognition model if the second loss is greater than a preset second loss threshold, until the second loss is less than or equal to the preset second loss threshold, and obtaining the trained attribute recognition model.

[0086] In this embodiment, feature extraction can be performed on each sample image by the trained target detection model or other manners to obtain a corresponding sample feature map. The sample feature map of each sample image is merged with the sample data set to obtain an attribute recognition training set. A visual large model is used as a basis, including but not limited to a visual model of a CNN (Convolutional Neural Network) structure or a Transformer (a sequence model based on an attention mechanism) structure, an ROI Pooling (Region of Interest Pooling) layer is added, and N attribute detection heads are added, where N is the number of detected attributes, to establish an attribute recognition model.

[0087] In one specific embodiment of the present application, a DETR (Detection Transformer) algorithm can be used as a target detection neural network to train the target detection neural network to obtain a trained target detection model. Based on the trained target detection model, the target detection head is deleted, the ROI Pooling layer is added, and three attribute detection heads are added to detect the occlusion, blur, and truncation attributes, respectively.

[0088] The attribute recognition training set is input into the attribute recognition model for training, which can shorten the training time of the attribute recognition model. Of course, the attribute recognition model can also be directly trained using the sample data set, but the training time is longer, and the parameters of the base network will change, which is easy to cause the feature maps extracted by the trained target detection model and the trained attribute recognition model for the same vehicle environment image to be inconsistent.

[0089] During the training process, the attribute recognition model outputs the attribute values of the real boxes corresponding to each sample image as the test attribute values of the real boxes, and then calculates a second loss based on the real attribute values and the test attribute values of each real box. The calculation method of the second loss is as follows:

[0090] loss2=ω1L p1 +ω2L p2 +…+ω N L pN Equation (5),

[0091] wherein, loss2 is the second loss, L pi is the loss function of the i-th attribute detection head, ω i is the loss function weight of the i-th attribute detection head, wherein the loss function of the attribute detection head adopts a cross-entropy loss function. Illustratively, ω i may take a value of 1 or other numerical values, which are not limited here.

[0092] If the second loss is greater than the preset second loss threshold, the parameters of the attribute recognition model are updated according to the second loss until the second loss is less than or equal to the preset second loss threshold, or until the preset training number is reached, to obtain the trained attribute recognition model. Illustratively, the preset second loss threshold can be 0.01, or other numerical values, which are not limited here.

[0093] Please refer to Figure 4 , Figure 4 is a training flowchart of the attribute recognition model according to an embodiment of the present application. As shown in Figure 4 , the training flowchart of the attribute recognition model is as follows:

[0094] The trained target detection model is used to process each sample image in the sample data set to obtain a sample feature map corresponding to each sample image. The sample feature map corresponding to each sample image is merged with the sample data set to obtain an attribute recognition training set. The attribute recognition model is trained using the attribute recognition training set for forward reasoning, and the loss is calculated according to formula (5) as the second loss to update the parameters of the attribute recognition model and continue training. When the training number reaches 50 times, or the second loss is less than or equal to 0.01, the forward reasoning is stopped, and the trained attribute recognition model is obtained.

[0095] In an embodiment of the present application, the attribute recognition is performed on the target object corresponding to each candidate detection box to obtain the attribute value of each candidate detection box, including: inputting the detection box data of all candidate detection boxes and the feature map of the vehicle environment image into the trained attribute recognition model to obtain the attribute value of each candidate detection box, the feature map being obtained by the trained target detection model performing feature extraction on the vehicle environment image, and the attribute recognition process of the trained attribute recognition model including: matching the target sub-feature map of the target object corresponding to each candidate detection box from the feature map according to the detection box position of each candidate detection box, and performing attribute recognition on the target sub-feature map to determine the attribute value of each candidate detection box.

[0096] In this embodiment, the attribute value of the candidate detection frame is determined by the trained attribute recognition model, which can effectively improve the efficiency of attribute recognition. Since the trained target detection model includes a basic network and a target detection head, the basic network is used to extract features of the vehicle environment image to obtain a feature map of the vehicle environment image, so that the target detection head performs target detection on the feature map. Therefore, the feature map of the vehicle environment image and the detection frame data of the candidate detection frame extracted by the trained target detection model can be input into the trained attribute recognition model, so that the trained attribute recognition model does not need to repeatedly extract the feature map, directly uses the detection position in the detection frame data of the candidate detection frame to cut out the input feature map, obtains the target sub-feature map corresponding to each candidate detection frame and performs attribute recognition, which can shorten the time of attribute recognition and further improve the efficiency of attribute recognition.

[0097] In an embodiment of the present application, the detection frame data of the target detection frame is determined from the detection frame data of all candidate detection frames using the attribute value, including: if the attribute value of the candidate detection frame meets the preset condition, deleting the detection frame data of the candidate detection frame, and determining the detection frame data of the retained candidate detection frame as the detection frame data of the target detection frame, the attribute value including at least one of the occlusion degree, the blur degree and the truncation degree, and the preset condition including: the occlusion degree meets the preset occlusion degree; or, the blur degree meets the preset blur degree; or, the truncation degree meets the preset threshold.

[0098] In this embodiment, the detection frame data of the candidate detection frame corresponding to the target object with serious occlusion, serious blur or serious truncation is removed using the attribute value of the candidate detection frame, which can further improve the accuracy of the pre-annotation result. If the attribute value of a certain candidate detection frame meets the preset condition, the candidate detection frame is considered to be a low-quality candidate detection frame, and accordingly, the detection frame data of the candidate detection frame is detection frame data with poor quality and accuracy. Removing the detection frame data of all low-quality candidate detection frames, and determining the detection frame data of the retained candidate detection frame as the detection frame data of the target detection frame.

[0099] Illustratively, the blur degree or the occlusion degree or the truncation degree can be represented by three levels of high, medium and low, or by specific numerical values to represent the degree, which is not limited here. For example: the blur degree can be represented by a numerical value between 0 and 1, and the larger the numerical value, the higher the blur degree.

[0100] It should be noted that the trained target detection model and the trained attribute recognition model can be two independent models, or two sub-models in the comprehensive pre-annotation model.

[0101] In an embodiment of the present application, the trained target detection model and the trained attribute recognition model are two independent models. The trained target detection model is pre-configured to output a feature map of the vehicle environment image while outputting the first detection result.

[0102] In another embodiment of the present application, after obtaining the trained attribute recognition model, the vehicle environment image pre-labeling method comprises: dividing the trained target detection model into a base network and a target detection head, and taking the trained attribute recognition model as an attribute detection head module; connecting the base network with the target detection head and the attribute detection head module respectively to obtain a comprehensive pre-labeling model for target detection and attribute recognition of the target object.

[0103] In this embodiment, the trained target detection model and the trained attribute recognition model are two sub-models in the comprehensive pre-labeling model. Please refer to Figure 5 , Figure 5 is a schematic diagram of a comprehensive pre-labeling model network structure according to another specific embodiment of the present application. As shown in Figure 5 , the comprehensive pre-labeling model comprises a trained target detection model and a trained attribute recognition model, wherein the trained target detection model is divided into a base network and a target detection head, the trained attribute recognition model is taken as an attribute detection head module, the base network is connected with the target detection head and the attribute detection head module respectively, the base network is used to extract a feature map, the target detection head is used to detect the category and position of the target object, and outputs class / box, i.e. the detection frame data of the detection frame, including the detection frame position, the detection frame category and the category confidence. The attribute detection head module comprises an ROIPooling layer and N attribute detection heads, and is used to recognize the attribute of the target object, N being an integer greater than or equal to 1.

[0104] In this embodiment, the vehicle environment image / mirrored image is input into the comprehensive pre-labeling model respectively, the vehicle environment image / mirrored image is feature extracted by the basic network to obtain the feature map of the vehicle environment image / mirrored image, the feature map of the vehicle environment image / mirrored image is target detected by the target detection head to output the first detection result / second initial detection result; the second initial detection result is mirrored to obtain the second detection result, and the first detection result and the second detection result are paired and screened to obtain the detection box data of a plurality of candidate detection boxes. The detection box data of all candidate detection boxes is input into the comprehensive pre-labeling model, the ROIPooling layer in the attribute detection head module extracts the target sub-feature map of the target object corresponding to the candidate detection box from the feature map of the vehicle environment image by using the detection box position in the detection box data of the candidate detection box, so that each attribute detection head performs different type of attribute recognition on the target sub-feature map to output the attribute value of the candidate detection box, including at least one of the blur degree, the occlusion degree and the truncation degree. It is judged whether the attribute value of the candidate detection box meets the preset condition to determine the target detection box data of the target detection box, and the target detection box data and the attribute value of the target detection box are taken as the pre-labeling result of the vehicle environment image. The technical scheme of the embodiment of the application constructs a comprehensive pre-labeling model based on the trained target detection model and the trained attribute recognition model, without repeatedly extracting features from the vehicle environment image to perform target detection and attribute recognition on the vehicle environment image, thereby improving the accuracy of pre-labeling, reducing the amount of calculation, and improving the pre-labeling efficiency.

[0105] Please refer to Figure 6 , Figure 6 is a brief flowchart of pre-training model training according to another specific embodiment of the application, as shown in Figure 6 , the brief flowchart of model training is as follows:

[0106] A target detection deep neural network based on a visual large model is constructed by using the DETR algorithm, the sample data set with labels is processed, the high-quality GT Box and the low-quality GT Box are labeled respectively, and are denoted as a target detection training set. The target detection deep neural network is trained using the target detection training set to obtain a trained target detection model, which is a first pre-training model.

[0107] The first pre-training model comprises a base network and a target detection head. The target detection head in the first pre-training model can be removed, and an ROIPooling layer and N attribute detection heads are added to constitute an attribute recognition model, which is used to identify the attributes of the target object and determine the corresponding attribute values. A base model is established according to the base network in the first pre-training model, and the base model is used to extract features of each frame of data (each sample image) in the target detection training set, to obtain a sample feature map corresponding to each frame of data. The sample feature map corresponding to each frame of data is merged with the sample data set to obtain an attribute recognition training set. The attribute recognition training set is used to train the attribute recognition model, to obtain a trained attribute recognition model, which is a second pre-training model.

[0108] The first pre-training model and the second pre-training model are used to perform pre-labeling inference on the vehicle environment image, to obtain a pre-labeling result of the vehicle environment image.

[0109] Referring to Figure 7 , Figure 7 is a flowchart of another vehicle environment image pre-labeling method according to another specific embodiment of the present application. As shown in Figure 7 , the flow of the vehicle environment image pre-labeling method is as follows:

[0110] First, the vehicle environment image is normalized to obtain a first image, and the first image is mirror-inverted to obtain a second image. The target object in the first image is identified by the first pre-training model, including positioning detection and category detection, to obtain a first result (first detection result). The target object in the second image is identified by the first pre-training model, and the identification result is mirror-inverted to obtain a second result (second detection result).

[0111] Secondly, the first result and the second result are matched based on IOU to obtain a plurality of paired groups. The detection box data of low-quality detection boxes in each paired group is determined and removed according to the category confidence, i.e., the first low-quality box is removed in Figure 7 . The detection box data of the remaining detection boxes is used as the detection box data of candidate detection boxes.

[0112] Finally, the detection box data of the candidate detection boxes and the feature map of the vehicle environment image are input into the second pre-training model. The second pre-training model identifies the attributes of the target objects corresponding to the candidate detection boxes, to determine the attribute values of each candidate detection box pair. The detection box data of low-quality candidate detection boxes is determined and removed according to the attribute values, i.e., the second low-quality box is removed in Figure 7 . The detection box data of the remaining candidate detection boxes and the attribute values are used as the pre-labeling result of the vehicle environment image.

[0113] For details, please refer to the description in the foregoing embodiments, which will not be repeated here.

[0114] Figure 8 is a block diagram of a vehicle environment image pre-labeling device according to an example embodiment of the present application. The device can be applied to Figure 1 the implementation environment shown and specifically configured in the computer device 102. The device can also be applied to other example implementation environments and specifically configured in other devices, and the implementation environment to which the device is applied is not limited by the present embodiment.

[0115] As Figure 8 shown, the example vehicle environment image pre-labeling device includes:

[0116] The acquisition module 810 is configured to acquire a vehicle environment image and a mirror image corresponding to the vehicle environment image. The target detection module 820 is configured to perform target detection on the vehicle environment image to obtain a first detection result, and perform target detection on the mirror image to obtain a second detection result. The first detection result and the second detection result each include detection box data of a detection box corresponding to each target object in the vehicle environment image, and the detection box data includes a detection box category of the detection box. The first screening module 830 is configured to compare the detection box data of each detection box in the first detection result with the detection box data of each detection box of the same detection box category in the second detection result, and determine detection box data of a plurality of candidate detection boxes according to a comparison result. The attribute recognition module 840 is configured to perform attribute recognition on a target object corresponding to each candidate detection box to obtain an attribute value of each candidate detection box. The second screening module 850 is configured to determine detection box data of a target detection box from the detection box data of all candidate detection boxes using the attribute value, and determine the detection box data of the target detection box and the attribute value as a pre-labeling result of the vehicle environment image.

[0117] It should be noted that the vehicle environment image pre-labeling device provided by the above embodiments and the vehicle environment image pre-labeling method provided by the above embodiments belong to the same concept, and the specific manner in which each module and unit performs operations has been described in detail in the method embodiments, which will not be repeated here. The vehicle environment image pre-labeling device provided by the above embodiments can be completed by different functional modules according to the need in actual application, i.e., the internal structure of the device is divided into different functional modules to complete all or part of the functions described above, and this will not be limited herein.

[0118] Embodiments of the present application also provide an electronic device, comprising: one or more processors; a storage device for storing one or more programs, which, when executed by the one or more processors, cause the electronic device to implement the vehicle environment image pre-labeling method provided in each of the above embodiments.

[0119] Figure 9 A structural schematic diagram of a computer system of an electronic device suitable for implementing embodiments of the present application is shown. It should be noted that, Figure 9 The computer system 900 of the electronic device shown is only an example and should not impose any limitation on the functions and use range of embodiments of the present application.

[0120] As Figure 9 shown, the computer system 900 includes a central processing unit (CPU) 901, which can perform various appropriate actions and processes according to programs stored in a read-only memory (ROM) 902 or programs loaded from a storage portion 908 into a random access memory (RAM) 903, such as performing the methods described in the above embodiments. In the RAM 903, various programs and data required for system operation are also stored. The CPU 901, the ROM 902, and the RAM 903 are connected to each other through a bus 904. An input / output (I / O) interface 905 is also connected to the bus 904.

[0121] The following components are connected to the I / O interface 905: an input portion 906 including a keyboard, a mouse, and the like; an output portion 907 including a display such as a cathode ray tube (CRT), a liquid crystal display (LCD), and the like, and a speaker, and the like; a storage portion 908 including a hard disk, and the like; and a communication portion 909 including a network interface card such as a LAN (Local Area Network) card, a modem, and the like. The communication portion 909 performs communication processing via a network such as the Internet. A drive 910 is also connected to the I / O interface 905 as needed. A removable recording medium 911 such as a magnetic disk, an optical disc, a magneto-optical disc, a semiconductor memory, and the like is attached to the drive 910 as needed, so that a computer program read therefrom is installed in the storage portion 908 as needed.

[0122] In particular, the processes described above with reference to the flow charts can be implemented as computer software programs in accordance with the embodiments of the present application. For example, the embodiments of the present application include a computer program product comprising a computer program carried on a computer readable medium, the computer program containing computer programs for executing the methods shown in the flow charts. In such embodiments, the computer program can be downloaded and installed from a network by the communication section 909, and / or installed from the detachable medium 911. When the computer program is executed by the central processing unit (CPU) 901, various functions defined in the system of the present application are executed.

[0123] It should be noted that the computer readable medium shown in the embodiments of the present application can be a computer readable signal medium or a computer readable storage medium or any combination of the two. The computer readable storage medium may, for example, be an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or apparatus, or any combination of the above. More specific examples of the computer readable storage medium can include, but are not limited to, an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM), a flash memory, an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present application, the computer readable signal medium can include a data signal propagated in a baseband or as part of a carrier wave, in which the computer readable computer program is carried. Such a propagated data signal can take on many forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination of the above. The computer readable signal medium can also be any computer readable medium other than the computer readable storage medium that can send, propagate, or transfer the program for use by or in connection with the instruction execution system, apparatus, or device. The computer program contained on the computer readable medium can be transmitted in any suitable medium, including but not limited to wireless, wired, or the like, or any suitable combination of the above.

[0124] The flow and block diagrams in the drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present application. In this regard, each block in the flow and block diagrams can represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical functions. It should also be noted that in some alternative implementations, the functions noted in the blocks can occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently or the blocks may be executed in the reverse order, depending on the functionality involved. It will also be noted that each block of the block diagrams and / or flowchart illustrations, and combinations thereof, can be implemented by special purpose hardware-based systems that perform the specified functions or operations, or combinations of special purpose hardware and computer instructions.

[0125] The units described in the embodiments of the present application can be implemented by software, or by hardware, or by a combination of software and hardware. The units described may

[0126] Another aspect of the present application provides a computer readable storage medium, which stores a computer program. The computer program is executed by a processor of a computer, and causes the computer to perform the vehicle environment image pre-labeling method described above. The computer readable storage medium can be included in the electronic device described in the embodiments above, or can exist separately and not be assembled into the electronic device.

[0127] Another aspect of the present application provides a computer program product or a computer program, which includes computer instructions stored in a computer readable storage medium. A processor of a computer device reads the computer instructions from the computer readable storage medium, and the processor executes the computer instructions, so that the computer device performs the vehicle environment image pre-labeling method provided in the embodiments above.

[0128] The above embodiments are only illustrative of the principles and effects of the present application, and are not intended to limit the present application. Any person skilled in the art can modify or change the above embodiments without departing from the spirit and scope of the present application. Therefore, all equivalent modifications or changes made by those skilled in the art without departing from the spirit and technical ideas of the present application should be covered by the claims of the present application.

Claims

1. A vehicle environment image pre-labeling method, characterized in that, The vehicle environment image pre-labeling method comprises: acquiring a vehicle environment image and a mirror image corresponding to the vehicle environment image; performing target detection on the vehicle environment image to obtain a first detection result, and performing target detection on the mirror image to obtain a second detection result, wherein the first detection result and the second detection result each comprise detection box data of a detection box corresponding to each target object in the vehicle environment image, and the detection box data comprises a detection box category of the detection box; comparing the detection box data of each detection box in the first detection result with the detection box data of each detection box of the same detection box category in the second detection result, and determining detection box data of a plurality of candidate detection boxes according to a comparison result; performing attribute recognition on a target object corresponding to each candidate detection box to obtain an attribute value of each candidate detection box, determining detection box data of a target detection box from the detection box data of all candidate detection boxes by using the attribute value, and determining the detection box data and the attribute value of the target detection box as a pre-labeling result of the vehicle environment image, wherein the attribute value comprises at least one of an occlusion degree, a blur degree and a truncation degree.

2. The vehicle environmental image pre-labeling method of claim 1, wherein, comparing the detection box data of each detection box in the first detection result with the detection box data of each detection box of the same detection box category in the second detection result comprises: pairing the detection box data of each detection box in the first detection result with the detection box data of each detection box of the same detection box category in the second detection result by using detection box positions to obtain a plurality of pairing groups, wherein the detection box data further comprises a detection box position of the detection box; determining the detection box data of a detection box with greater category confidence in each pairing group as the detection box data of the candidate detection box to obtain the detection box data of the plurality of candidate detection boxes, wherein the detection box data further comprises a category confidence corresponding to the detection box category.

3. The vehicle environmental image pre-labeling method of claim 2, wherein, pairing the detection box data of each detection box in the first detection result with the detection box data of each detection box of the same detection box category in the second detection result by using detection box positions comprises: determining the detection box data of the detection box in the first detection result as detection box data of a first detection box, and determining the detection box data of the detection box in the second detection result as detection box data of a second detection box; if the detection box category of the first detection box is the same as the detection box category of the second detection box, calculating an intersection over union of the first detection box and the second detection box according to the detection box position of the first detection box and the detection box position of the second detection box; if the intersection over union of the first detection box and the second detection box is greater than or equal to a preset intersection over union threshold, determining the detection box data of the first detection box and the detection box data of the second detection box as one pairing group.

4. The vehicle environmental image pre-labeling method according to any one of claims 1-3, characterized in that, determining the detection box data of a target detection box from the detection box data of all candidate detection boxes by using the attribute value comprises: If the attribute value of the candidate bounding box meets a preset condition, the bounding box data of the candidate bounding box is deleted, the bounding box data of the retained candidate bounding box is determined as the bounding box data of the target bounding box, and the preset condition includes that The occlusion degree is greater than a preset occlusion degree; Or, The blur degree is greater than a preset blur degree; Or, The truncation degree is greater than a preset truncation degree.

5. The vehicle environmental image pre-labeling method of claim 1, wherein, Before target detection is performed on the vehicle environment image to obtain a first detection result and target detection is performed on the mirror image to obtain a second detection result, the vehicle environment image pre-labeling method includes: Obtaining a sample data set, the sample data set including a plurality of sample images, real bounding box data of at least one real bounding box corresponding to each sample image; Marking each real bounding box as a low-quality real bounding box or a high-quality real bounding box according to the real bounding box data of each real bounding box to obtain real bounding box data of each marked real bounding box; Establishing a target detection model based on an initial neural network and an initial target detection head, taking each sample image as input and taking the real bounding box data of the marked real bounding box corresponding to each sample image as output, training the target detection model to obtain test bounding box data of each target object in each sample image corresponding to each test bounding box; Matching a corresponding test bounding box to each high-quality real bounding box as a high-quality test bounding box, and calculating a first loss according to the real bounding box data of each high-quality real bounding box and the test bounding box data of the corresponding high-quality test bounding box; If the first loss is greater than a preset first loss threshold, updating the parameters of the target detection model according to the first loss until the first loss is less than or equal to the preset first loss threshold to obtain a trained target detection model.

6. The vehicle environmental image pre-labeling method according to claim 5, characterized in that, Matching a corresponding test bounding box to each high-quality real bounding box as a high-quality test bounding box, and calculating a first loss according to the real bounding box data of each high-quality real bounding box and the test bounding box data of the corresponding high-quality test bounding box, including: For each sample image, performing minimum loss matching on each real bounding box and each test bounding box according to the real bounding box data of each real bounding box and the test bounding box data of each test bounding box to obtain a test bounding box corresponding to each real bounding box, taking the test bounding box corresponding to the high-quality real bounding box as a high-quality test bounding box, the real bounding box data including a real bounding box position and a real bounding box category, and the test bounding box data including a test bounding box position and a test bounding box category and a category confidence corresponding to the test bounding box category; Calculating an intersection over union loss using the real bounding box position of each high-quality real bounding box and the test bounding box position of the corresponding high-quality test bounding box, and calculating a category loss using the real bounding box category of each high-quality real bounding box and the test bounding box category and the category confidence of the corresponding high-quality test bounding box, and determining the first loss according to the intersection over union loss and the category loss.

7. The vehicle environmental image pre-labeling method of any one of claims 5 or 6, characterized in that, Before attribute recognition is performed on a target object corresponding to each candidate bounding box to obtain an attribute value of each candidate bounding box, the vehicle environment image pre-labeling method includes: obtain an attribute recognition training set, the attribute recognition training set comprising sample feature maps of a plurality of sample images, and real box data of at least one real box corresponding to each sample image, the sample feature map of each sample image being obtained by performing feature extraction on each sample image by using the trained target detection model, and the real box data comprising a real box position and a real box attribute value of the real box; establish an attribute recognition model based on at least one initial attribute detection head, input the sample feature map of each sample image and the real box position of the real box corresponding to each sample image, and output the real box attribute value of the real box corresponding to each sample image, train the attribute recognition model, and obtain a test attribute value of the real box corresponding to each sample image; calculate a second loss based on the real attribute value and the test attribute value of each real box, update the parameters of the attribute recognition model if the second loss is greater than a preset second loss threshold, until the second loss is less than or equal to the preset second loss threshold, and obtain a trained attribute recognition model.

8. The vehicle environmental image pre-labeling method of any one of claims 5 or 6, characterized in that, perform target detection on the vehicle environment image to obtain a first detection result, and perform target detection on the mirror image to obtain a second detection result, comprising: perform positioning detection and category detection on the target object in the vehicle environment image by using the trained target detection model to obtain detection box data of each target object corresponding detection box in the vehicle environment image as the first detection result; perform positioning detection and category detection on the target object in the mirror image by using the trained target detection model to obtain detection box data of each target object corresponding detection box in the mirror image as a second initial detection result; perform mirror processing on the second initial detection result to obtain the second detection result.

9. The vehicle environmental image pre-labeling method of claim 7, wherein, perform attribute recognition on the target object corresponding to each candidate detection box to obtain an attribute value of each candidate detection box, comprising: input the detection box data of all candidate detection boxes and a feature map of the vehicle environment image into the trained attribute recognition model to obtain an attribute value of each candidate detection box, the feature map being obtained by performing feature extraction on the vehicle environment image by using the trained target detection model, and the attribute recognition process of the trained attribute recognition model comprising, match a target sub-feature map of the target object corresponding to each candidate detection box from the feature map according to the detection box position of each candidate detection box, and perform attribute recognition on the target sub-feature map to determine the attribute value of each candidate detection box.

10. The vehicle environmental image pre-labeling method of claim 7, wherein, After obtaining the trained attribute recognition model, the vehicle environment image pre-labeling method comprises: divide the trained target detection model into a basic network and a target detection head, and use the trained attribute recognition model as an attribute detection head module; connect the basic network with the target detection head and the attribute detection head module respectively to obtain a comprehensive pre-labeling model for performing target detection and attribute recognition on the target object.

11. A vehicle environment image pre-labeling apparatus, characterized by comprising: The vehicle environment image pre-labeling device comprises: An acquisition module is configured to acquire a vehicle environment image and a mirror image corresponding to the vehicle environment image; A target detection module is configured to perform target detection on the vehicle environment image to obtain a first detection result, and perform target detection on the mirror image to obtain a second detection result. The first detection result and the second detection result each include detection box data of a detection box corresponding to each target object in the vehicle environment image. The detection box data includes a detection box category of the detection box. A first screening module is configured to compare detection box data of each detection box in the first detection result with detection box data of each detection box of the same detection box category in the second detection result, and determine detection box data of a plurality of candidate detection boxes according to a comparison result. An attribute identification module is configured to perform attribute identification on a target object corresponding to each candidate detection box to obtain an attribute value of each candidate detection box. The attribute value includes at least one of an occlusion degree, a blur degree, and a truncation degree. A second screening module is configured to determine detection box data of a target detection box from the detection box data of all candidate detection boxes by using the attribute value, and determine the detection box data and the attribute value of the target detection box as a pre-labeling result of the vehicle environment image.

12. An electronic device, comprising: The electronic device includes: One or more processors; A storage device configured to store one or more programs, which, when executed by the one or more processors, cause the electronic device to implement the vehicle environment image pre-labeling method according to any one of claims 1 to 10.

13. A computer-readable storage medium, characterized in that, A computer program is stored thereon, which, when executed by a processor of a computer, causes the computer to perform the vehicle environment image pre-labeling method according to any one of claims 1 to 10.

Images