Data annotation method and device, model training method and device, equipment and medium

Abstract

The invention relates to a model training method and device, equipment and a medium, and the method comprises the steps: determining a first sub-image and a second sub-image from a driving environment image; affine transformation is carried out on the first sub-graph and the second sub-graph; respectively carrying out image segmentation processing on the first sub-image after affine transformation and the second sub-image after affine transformation by adopting a segmentation network to generate a first segmentation result and a second segmentation result; and training a segmentation network according to the first segmentation result and the second segmentation result. According to the technical scheme disclosed by the invention, automatic labeling of the data can be realized, the manual labeling cost is reduced, the training data set of the model is expanded, and meanwhile, the model precision is ensured.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence technology, and in particular to a model training method, device, equipment and medium. Background Art

[0002] Intelligent driving technology is a technology that uses machines to assist or replace humans in driving. In intelligent driving scenarios, there is a need to segment the driving environment at a semantic level to identify various objects in the driving environment, such as vehicles, pedestrians, and drivable areas in the driving environment. The method used is image semantic segmentation in the field of computer vision.

[0003] The training of image segmentation models relies on a massive amount of labeled data. For a large amount of unlabeled data in intelligent driving scenarios, it is costly to manually label the data. If the image segmentation model is trained with only a small amount of manually labeled data, the accuracy of the model is relatively low and it is difficult to meet the application requirements of actual scenarios. Therefore, how to train a model that meets the application requirements of actual scenarios with only a small amount of labeled data has become an urgent problem to be solved. Summary of the invention

[0004] In order to solve the above technical problems, the present disclosure provides a model training method, device, equipment and storage medium.

[0005] In a first aspect, an embodiment of the present disclosure provides a model training method, comprising:

[0006] Determining a first sub-image and a second sub-image from the driving environment image, wherein the first sub-image and the second sub-image contain the object to be segmented and have an overlapping area;

[0007] Performing affine transformation on the first sub-image and the second sub-image respectively;

[0008] Using a segmentation network to perform image segmentation processing on the first sub-image after affine transformation and the second sub-image after affine transformation, respectively, to generate a first segmentation result and a second segmentation result;

[0009] The segmentation network is trained according to the first segmentation result and the second segmentation result, wherein a training goal is that in the first segmentation result and the second segmentation result, a feature distance between two positions belonging to the same segmentation category is smaller than a feature distance between two positions belonging to different segmentation categories.

[0010] In a second aspect, the present disclosure provides a model training device, including:

[0011] A determination module, configured to determine a first sub-image and a second sub-image from the driving environment image, wherein the first sub-image and the second sub-image contain the object to be segmented and have an overlapping area;

[0012] A transformation module, used for performing affine transformation on the first sub-image and the second sub-image respectively;

[0013] A segmentation module, used to perform image segmentation processing on the first sub-image after affine transformation and the second sub-image after affine transformation by using a segmentation network, to generate a first segmentation result and a second segmentation result;

[0014] A training module is used to train the segmentation network according to the first segmentation result and the second segmentation result, wherein the training goal is: in the first segmentation result and the second segmentation result, the feature distance between two positions belonging to the same segmentation category is smaller than the feature distance between two positions belonging to different segmentation categories.

[0015] In a third aspect, an embodiment of the present disclosure provides an electronic device, comprising: a processor; a memory for storing executable instructions of the processor; the processor is used to read the executable instructions from the memory and execute the instructions to implement the model training method described in the first aspect above.

[0016] In a fourth aspect, an embodiment of the present disclosure provides a computer-readable storage medium, wherein the storage medium stores a computer program, and when the computer program is executed by a processor, the model training method described in the first aspect is implemented.

[0017] In a fifth aspect, an embodiment of the present disclosure provides a vehicle, comprising the model training device as described in the second aspect above, or the electronic device as described in the third aspect above.

[0018] In a sixth aspect, an embodiment of the present disclosure provides a data labeling method, including:

[0019] Image data to be annotated; inputting the image data to be annotated into the segmentation network to generate data annotation results;

[0020] The segmentation network is a segmentation network trained by the model training method described in the first aspect.

[0021] Compared with the prior art, the technical solution provided by the disclosed embodiment has the following advantages: by performing affine transformation on the first sub-image and the second sub-image in the driving environment image, respectively, a segmentation network is used to perform image segmentation processing on the first sub-image after affine transformation and the second sub-image after affine transformation, respectively, to generate a segmentation result, and then, based on the segmentation result, the segmentation network is trained according to certain conditions, and data labeling is achieved through the trained segmentation network. Thus, the model performance is improved through self-supervision, and the problem of how to train a model that meets the application requirements of intelligent driving scenarios with only a small amount of labeled data is solved. Automatic data labeling is achieved while ensuring accuracy, and the training data set of the model is quickly expanded, reducing the cost of manual labeling. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0023] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0024] Figure 1 A flowchart of a model training method provided by an embodiment of the present disclosure;

[0025] Figure 2 A flowchart of another model training method provided by an embodiment of the present disclosure;

[0026] Figure 3 A schematic diagram of an algorithm flow provided by an embodiment of the present disclosure;

[0027] Figure 4 A schematic diagram of the structure of a model training device provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0028] In order to more clearly understand the above-mentioned objectives, features and advantages of the present disclosure, the scheme of the present disclosure will be further described below. It should be noted that the embodiments of the present disclosure and the features in the embodiments can be combined with each other without conflict.

[0029] In the following description, many specific details are set forth to facilitate a full understanding of the present disclosure, but the present disclosure may also be implemented in other ways different from those described herein; it is obvious that the embodiments in the specification are only part of the embodiments of the present disclosure, rather than all of the embodiments.

[0030] Figure 1 This is a flow chart of a model training method provided in an embodiment of the present disclosure. The method provided in an embodiment of the present disclosure can be executed by a model training device, which can be implemented in software and / or hardware and can be integrated on any electronic device with computing capabilities, such as a vehicle-mounted terminal.

[0031] like Figure 1 As shown, the model training method provided by the embodiment of the present disclosure may include:

[0032] Step 101: determine a first sub-image and a second sub-image from a driving environment image.

[0033] In this embodiment, the driving environment image may be an image obtained by capturing the environment surrounding the vehicle, the first sub-image and the second sub-image contain objects to be segmented and the first sub-image and the second sub-image have an overlapping area, and the objects to be segmented include but are not limited to predetermined targets such as vehicles and people.

[0034] Among them, there are multiple ways to determine the first sub-image and the second sub-image. Optionally, a target detection network is used to perform target detection on the object to be segmented in the driving environment image, and the foreground area where the object to be segmented is located is determined from the driving environment image. The foreground area where the object to be segmented is located is intercepted to obtain a target image containing the object to be segmented, and then, the first sub-image and the second sub-image with overlapping areas are randomly selected from the target image. The target detection network can be trained using existing labeled data, and the network structure of the target detection network can adopt existing technology, which is not specifically limited here. Optionally, two sub-images with overlapping areas are directly intercepted from the driving environment image. If both sub-images contain the object to be segmented, the two sub-images are used as the first sub-image and the second sub-image respectively. If any sub-image does not contain the object to be segmented, the above steps are re-executed.

[0035] Step 102: Perform affine transformation on the first sub-image and the second sub-image respectively.

[0036] In this embodiment, the affine transformation includes random flipping, blurring, cropping, and color transformation. The affine transformation used for the first sub-image and the second sub-image can be randomly selected, or can be determined according to certain rules, or can be performed using other existing methods, which is not limited here. Optionally, a method is randomly selected from flipping, blurring, cropping, and color transformation to process the first sub-image, and a method is randomly selected from flipping, blurring, cropping, and color transformation to process the second sub-image.

[0037] As an example, the first sub-image is randomly flipped, and the second sub-image is blurred. It should be noted that the first sub-image and the second sub-image may be processed in more than one manner. As another example, the first sub-image is randomly flipped and blurred, and the second sub-image is cropped and color-changed.

[0038] Step 103: Use a segmentation network to perform image segmentation processing on the first sub-image after affine transformation and the second sub-image after affine transformation, respectively, to generate a first segmentation result and a second segmentation result.

[0039] In this embodiment, a segmentation network is used to perform image segmentation processing on the first sub-image after affine transformation to generate a first segmentation result, and a segmentation network is used to perform image segmentation processing on the second sub-image after affine transformation to generate a second segmentation result. The output of the segmentation network includes masks corresponding to each category obtained by segmenting the input image, and feature maps of the same size as the masks of each category, wherein the feature map is used to represent feature vectors at each position in the mask, for example, the first segmentation result includes the feature map obtained by segmenting the first sub-image after affine transformation, and the masks of each category obtained by segmenting the first sub-image after affine transformation, and the second segmentation result includes the feature map obtained by segmenting the second sub-image after affine transformation, and the masks of each category obtained by segmenting the first sub-image after affine transformation.

[0040] The segmentation network can be pre-trained using existing labeled segmentation data, the existing labeled segmentation data can be training data from a labeling platform, and the network structure of the segmentation network can adopt existing technology, which is not specifically limited here.

[0041] Step 104: training a segmentation network according to the first segmentation result and the second segmentation result.

[0042] In this embodiment, in the first segmentation result and the second segmentation result, the feature distance between two positions belonging to the same segmentation category is smaller than the feature distance between two positions belonging to different segmentation categories, so as to train the segmentation network, wherein whether the two positions belong to the same segmentation category can be obtained through the category masks in the first segmentation result and the second segmentation result, and the feature distance between the two positions can be obtained through the feature maps in the first segmentation result and the second segmentation result to obtain the feature vectors of the two positions, and then calculate the distance between the two feature vectors.

[0043] As an example, based on the condition that the feature distance between two positions belonging to the same segmentation category in the first segmentation result and the second segmentation result is less than the feature distance between two positions belonging to different segmentation categories, a loss function is constructed, and the segmentation network is optimized with the goal of minimizing the loss function, wherein the loss function can adopt contrast loss, or it can be a loss function constructed as needed to meet the above conditions, and there is no limitation here.

[0044] In the intelligent driving scenario, there is a need to perform semantic-level segmentation on the driving environment to identify various objects in the driving environment. A large amount of training data is required to train the model for this scenario. The method disclosed in the present invention is to automatically annotate a large amount of unlabeled data in this scenario to expand the training data set. However, the segmentation network obtained by pre-training with the existing labeled segmentation data is difficult to meet the automatic annotation accuracy requirements in this scenario. Therefore, in this embodiment, a random affine transformation is performed on the first sub-image and the second sub-image respectively, and then the image segmentation is performed on the sub-image after the affine transformation. Feature maps from different perspectives of the same image can be obtained. The feature maps based on the same object are consistent under different perspectives, so as to train the segmentation network and meet the accuracy requirements of the segmentation network for data annotation of the driving environment image.

[0045] In one embodiment of the present disclosure, after the segmentation network is trained, the image data to be labeled is input into the segmentation network to generate a data labeling result.

[0046] In this embodiment, the segmentation network is iteratively trained until the segmentation network converges. Then, the segmentation network is used to perform image segmentation processing on the image data to be labeled, and the image data is labeled according to the result of the image segmentation processing. The image data to be labeled can be driving environment image data related to the intelligent driving scenario, thereby realizing automatic data labeling and expanding the image segmentation training data set for the intelligent driving scenario.

[0047] According to the technical solution of the embodiment of the present disclosure, by performing affine transformation on the first sub-image and the second sub-image in the driving environment image, a segmentation network is used to perform image segmentation processing on the first sub-image after affine transformation and the second sub-image after affine transformation, respectively, to generate a segmentation result, and then, based on the segmentation result, the segmentation network is trained according to certain conditions, and data labeling is achieved through the trained segmentation network. Thus, the model performance is improved through self-supervision, and the problem that the existing segmentation network is difficult to meet the automatic labeling accuracy requirements when used in intelligent driving scenarios is solved. Automatic data labeling is achieved while ensuring accuracy, and the training data set of the model is quickly expanded, reducing the cost of manual labeling.

[0048] Based on the above embodiments, Figure 2 A flow chart of another model training method provided by an embodiment of the present disclosure is shown as follows: Figure 2 As shown, the method includes:

[0049] Step 201 : perform target detection on the driving environment image, obtain a target image containing the object to be segmented, and randomly select a first sub-image and a second sub-image having an overlapping area from the target image.

[0050] Among them, for example Figure 3 As shown, Figure 3 The left image is a driving environment image, and the right image represents the target image, the first sub-image, and the second sub-image from top to bottom. In this example, a target detection network is used to detect the objects to be segmented in the driving environment image. The objects to be segmented include but are not limited to predetermined targets such as vehicles and people. After the driving environment image is detected, the image is captured at the location of the object to be segmented, and the foreground area where the object to be segmented is located is captured to obtain a target image containing the object to be segmented. For example, a region with a size of 3 / 4 of the full image is captured to ensure that the foreground area is included. Then, the first sub-image and the second sub-image with overlapping areas are randomly selected from the target image.

[0051] Step 202: Perform affine transformation on the first sub-image and the second sub-image respectively.

[0052] The affine transformation includes random flipping, blurring, cropping, and color transformation. This process is similar to step 102, so it will not be described in detail.

[0053] Step 203, using a segmentation network to perform image segmentation on the target image, and obtaining masks corresponding to K categories and feature maps corresponding to the masks.

[0054] In this embodiment, the output of the segmentation network includes masks of various categories obtained by segmenting the input image, and a feature map with the same size as the mask. The target image is segmented by the segmentation network to obtain K categories of objects in the target image. The K categories of objects correspond to K categories of masks. After the target image is segmented, masks and feature maps corresponding to the K categories can be obtained. According to the masks of various categories obtained by segmentation, the segmentation category to which each position (pixel point) in the target image belongs can be determined, and the features of each position in the feature map can correspond to the feature vector representing each pixel point in the target image.

[0055] In practical applications, the segmentation network is pre-trained. The segmentation network trained with a small amount of labeled data has low accuracy and is difficult to meet the application requirements of actual scenarios. Therefore, in this embodiment, the segmentation network needs to be further trained to improve the accuracy of the segmentation network.

[0056] Step 204, clustering the features at each position in the feature map with K as the initial value, generating updated masks corresponding to the K categories, and performing the same affine transformation as the first sub-map and the second sub-map on the updated masks corresponding to the K categories to generate a first mask corresponding to the first sub-map and a second mask corresponding to the second sub-map.

[0057] In this embodiment, a clustering algorithm is used to cluster the features at each position in the feature map, and a new mask (denoted as m0) is determined by the clustering result to update the masks corresponding to the K categories output by the segmentation network, wherein the clustering algorithm includes the k-means algorithm, and the features belonging to the same category in the clustering result indicate that the corresponding positions belong to the same segmentation category. For example, the target image is segmented by the segmentation network to obtain background, human, vehicle 1 and vehicle 2, that is, 4 categories of masks are obtained, and the features at each position in the feature map are clustered with 4 as the initial value, and the 4 categories of masks are updated according to the clustering result.

[0058] Among them, m0 is subjected to the same affine transformation as the first sub-image and the second sub-image, including: m0 is subjected to the same flipping and / or cropping as the first sub-image and the second sub-image, so as to obtain a first mask (denoted as m1) corresponding to the first sub-image and a second mask (denoted as m2) corresponding to the second sub-image. Since the first sub-image and the second sub-image are selected from the target image, the part corresponding to the first sub-image and the part corresponding to the second sub-image in m0 can be determined according to the positions of the first sub-image and the second sub-image in the target image, and then the same flipping and / or cropping is performed on the corresponding parts to obtain m1 and m2.

[0059] As an example, the affine transformation of the first sub-image is flipping operation one, and the affine transformation of the second sub-image is cropping operation one, then m0 is flipped by one to obtain a first mask, and m0 is cropped by one to obtain a second mask.

[0060] As another example, the affine transformation of the first sub-image is flip operation two, and the affine transformation of the second sub-image is blur operation, then m0 is flipped two to obtain the first mask, and the second mask is directly determined without affine transformation of m0.

[0061] Step 205: Use a segmentation network to perform image segmentation on the first sub-image after affine transformation and the second sub-image after affine transformation, respectively, to obtain a first feature map corresponding to the first sub-image and a second feature map corresponding to the second sub-image.

[0062] In this embodiment, the output of the segmentation network includes masks of various categories obtained by segmenting the input image, and feature maps of the same size as each mask. In this step, when image segmentation is performed on the first sub-image after affine transformation and the second sub-image after affine transformation, the segmentation network output only retains the feature map part, namely the first feature map F1 and the second feature map F2.

[0063] It should be noted that the order between step 201, step 205 and step 203, step 204 can be adjusted as needed and is not limited here.

[0064] Step 206 , calculating the contrast loss based on the first segmentation result and the second segmentation result, and training the segmentation network with the goal of minimizing the contrast loss value.

[0065] In this embodiment, the first mask m1 and the first feature map F1 are used as the first segmentation result, and the second mask m2 and the second feature map F2 are used as the second segmentation result, wherein the contrast loss is calculated for F1 and F2 based on m1 and m2. For any first pixel in the first segmentation result and any second pixel in the second segmentation result, if the first pixel and the second pixel belong to the same segmentation category, the contrast loss is calculated based on the feature distance between the first pixel and the second pixel and the preset first sample pair label value. If the first pixel and the second pixel belong to different segmentation categories, the contrast loss is calculated based on the feature distance between the first pixel and the second pixel and the preset second sample pair label value, and then the segmentation network is trained with the minimum contrast loss value as the goal. The contrast loss formula is as follows:

[0066]

[0067] Among them, L c represents the loss function value, y represents the sample pair label, a represents any position (pixel point) in the first segmentation result, b represents any position in the second segmentation result, and m1 and m2 can be used to determine whether a and b belong to the same segmentation category. If a and b belong to the same segmentation category, y = 1, otherwise y = 0, Ia represents the feature vector of a, and Ib represents the feature vector of b, which can be obtained from the feature map, d Ia，Ib It represents the L2 distance (Euclidean distance) between the feature vector of a and the feature vector of b. α is a manually set hyperparameter, and + means that negative values ​​are taken as zero. The network optimization goal is to minimize the loss function. Specifically, when inputting positive sample pairs, the more similar the inputs are, the more similar the features are, the smaller the L2 distance is, and d Ia，Ib The smaller the distance is, the smaller the distance is. c It is positively correlated. When a negative sample pair is input, y=0. At this time, the L2 distance is greater than the L2 distance of the positive sample. In order to minimize the loss function, α-d is made less than 0, that is, the feature distance between the negative sample pairs is greater than α.

[0068] Optionally, segmentation networks S1 and S2 with the same network structure and parameters are used, wherein S1 is used to segment the target image, and S2 is used to segment the first sub-image after affine transformation and the second sub-image after affine transformation. When training the segmentation network, S2 is trained based on contrast loss, and S1 is updated based on the parameters of S2 using EMA (Exponential Moving Average, weighted sliding average), so that the parameters of S1 and S2 are updated synchronously to ensure model stability, and iterative training is performed until the model converges.

[0069] Step 207, input the image data to be annotated into the segmentation network to implement data annotation, divide each mask in the annotated data into confident samples and uncertain samples, and fine-tune the segmentation network according to the divided annotated data.

[0070] In this embodiment, the image data to be labeled is input into the segmentation network to generate the third segmentation result to realize automatic labeling of the overall unlabeled data. At this time, some noise will be introduced by automatic labeling, especially the junction of each category mask, which causes the self-supervised method to have mislabeling in the object junction area, while the features of the internal area of ​​the object are more obvious and easy to distinguish. Therefore, in this embodiment, after the segmentation network automatically labels the overall unlabeled data, the automatically labeled samples are divided into confident samples and unsure samples. By determining the edge area and the central area of ​​each mask in the third segmentation result, the central area is used as a confident sample and the edge area is used as an unsure sample. The edge area refers to the area formed by points within a certain range of distance from the edge of the mask, and the range can be set by itself. Then, the manually labeled data is mixed with the automatically labeled data, and the segmentation network is trained again to realize model fine-tuning. In the training process, the unsure samples are ignored when calculating the loss, for example, the loss at the unsure samples is set to zero, and the loss can be a cross entropy loss. Thus, a fine-tuned segmentation network is obtained, and the fine-tuned segmentation network can be used for automatic labeling of image data of the corresponding scene.

[0071] According to the technical solution of the embodiment of the present disclosure, the target image obtained by target detection is segmented, and the features in the output feature map are clustered to update the mask, and then the contrast loss is calculated according to the mask. Since the accuracy of the existing segmentation network is difficult to meet the needs of intelligent driving scenarios, the mask obtained by segmentation is updated by feature clustering, which improves the accuracy of the mask obtained by segmentation and further improves the accuracy of subsequent training. In addition, the model performance is improved through data hybrid training, the labeled data is divided into confident samples and uncertain samples, and the loss at the uncertain samples is ignored when the model is fine-tuned, further improving the accuracy of automatic labeling.

[0072] The disclosed embodiment also provides a model training device.

[0073] Figure 4 A schematic diagram of the structure of a model training device provided by an embodiment of the present disclosure is shown in FIG. Figure 4 As shown, the model training device includes: a determination module 41, a transformation module 42, a segmentation module 43, and a training module 44.

[0074] The determination module 41 is used to determine a first sub-image and a second sub-image from the driving environment image, wherein the first sub-image and the second sub-image contain the object to be segmented and have an overlapping area.

[0075] The transformation module 42 is used to perform affine transformation on the first sub-image and the second sub-image respectively.

[0076] The segmentation module 43 is used to use a segmentation network to perform image segmentation processing on the first sub-image after affine transformation and the second sub-image after affine transformation, respectively, to generate a first segmentation result and a second segmentation result.

[0077] A training module 44 is used to train the segmentation network according to the first segmentation result and the second segmentation result, wherein a training goal is that in the first segmentation result and the second segmentation result, a feature distance between two positions belonging to the same segmentation category is smaller than a feature distance between two positions belonging to different segmentation categories.

[0078] In one embodiment of the present disclosure, the determination module 41 is specifically used to: perform target detection on the driving environment image to obtain a target image containing the object to be segmented; and randomly select the first sub-image and the second sub-image having overlapping areas from the target image.

[0079] In one embodiment of the present disclosure, the segmentation module 43 is specifically used to: use the segmentation network to perform image segmentation on the target image to obtain masks corresponding to K categories and feature maps corresponding to the masks; cluster the features at each position in the feature map with K as the initial value to generate updated masks corresponding to K categories; perform the same affine transformation as the first sub-image and the second sub-image on the updated masks corresponding to the K categories to generate a first mask corresponding to the first sub-image and a second mask corresponding to the second sub-image; use the segmentation network to perform image segmentation on the first sub-image after affine transformation and the second sub-image after affine transformation to obtain a first feature map corresponding to the first sub-image and a second feature map corresponding to the second sub-image; wherein the first mask and the first feature map are the first segmentation results, and the second mask and the second feature map are the second segmentation results.

[0080] In one embodiment of the present disclosure, the transformation module 42 is specifically used to: randomly select one of flipping, blurring, cropping, and color transformation to process the first sub-image; and randomly select one of flipping, blurring, cropping, and color transformation to process the second sub-image.

[0081] In one embodiment of the present disclosure, the training module 44 is specifically used to: determine whether any first pixel point in the first segmentation result and any second pixel point in the second segmentation result belong to the same segmentation category; if the first pixel point and the second pixel point belong to the same segmentation category, calculate the contrast loss based on the feature distance between the first pixel point and the second pixel point and the preset first sample pair label value; if the first pixel point and the second pixel point belong to different segmentation categories, calculate the contrast loss based on the feature distance between the first pixel point and the second pixel point and the preset second sample pair label value; train the segmentation network with the goal of minimizing the contrast loss value.

[0082] In one embodiment of the present disclosure, the labeling module 45 is specifically used to: input the image data to be labeled into the segmentation network to generate a third segmentation result; determine the edge area and the center area of ​​each mask in the third segmentation result; based on the third segmentation result, re-train the segmentation network, wherein the loss calculated at the edge area is ignored during the training process; input the image data to be labeled into the re-trained segmentation network to generate a data labeling result.

[0083] In one embodiment of the present disclosure, the device further includes: a labeling module, configured to input the image data to be labeled into the segmentation network to generate a data labeling result.

[0084] The model training device provided in the embodiment of the present disclosure can execute any model training method provided in the embodiment of the present disclosure, and has the corresponding functional modules and beneficial effects of the execution method. The contents not described in detail in the embodiment of the device of the present disclosure can refer to the description in any method embodiment of the present disclosure.

[0085] An embodiment of the present disclosure also provides an electronic device, which includes one or more processors and a memory.

[0086] The processor may be a central processing unit (CPU) or other forms of processing units having data processing capabilities and / or instruction execution capabilities, and may control other components in the electronic device to perform desired functions.

[0087] The memory may include one or more computer program products, and the computer program product may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. Volatile memory may include, for example, random access memory (RAM) and / or cache memory (cache), etc. Non-volatile memory may include, for example, read-only memory (ROM), hard disk, flash memory, etc. One or more computer program instructions may be stored on a computer-readable storage medium, and a processor may run the program instructions to implement the method of the embodiments of the present disclosure above and / or other desired functions. Various contents such as input signals, signal components, noise components, etc. may also be stored in the computer-readable storage medium.

[0088] In one example, the electronic device may further include: an input device and an output device, and these components are interconnected through a bus system and / or other forms of connection mechanisms. In addition, the input device may also include, for example, a keyboard, a mouse, etc. The output device may output various information to the outside, including determined distance information, direction information, etc. The output device may include, for example, a display, a speaker, a printer, a communication network and a remote output device connected thereto, etc.

[0089] Of course, for simplicity, only some of the components in the electronic device related to the present disclosure are shown, and components such as buses, input / output interfaces, etc. are omitted. In addition, the electronic device may further include any other appropriate components according to specific application scenarios.

[0090] In addition to the above methods and devices, the embodiments of the present disclosure may also be a computer program product, which includes computer program instructions. When the computer program instructions are executed by a processor, the processor executes any method provided by the embodiments of the present disclosure.

[0091] The computer program product may be written in any combination of one or more programming languages ​​to write program code for performing the operations of the disclosed embodiments, including object-oriented programming languages ​​such as Java, C++, etc., and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0092] In addition, the embodiments of the present disclosure may also be a computer-readable storage medium on which computer program instructions are stored. When the computer program instructions are executed by a processor, the processor executes any method provided by the embodiments of the present disclosure.

[0093] Computer readable storage media can adopt any combination of one or more readable media. The readable medium can be a readable signal medium or a readable storage medium. The readable storage medium can include, for example, but is not limited to, a system, device or device of electricity, magnetism, light, electromagnetic, infrared, or semiconductor, or any combination of the above. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0094] It should be noted that, in this article, relational terms such as "first" and "second" are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0095] The above description is only a specific embodiment of the present disclosure, so that those skilled in the art can understand or implement the present disclosure. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present disclosure. Therefore, the present disclosure will not be limited to the embodiments described herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A model training method, characterized in that: include: Determining a first sub-image and a second sub-image from the driving environment image, wherein the first sub-image and the second sub-image contain the object to be segmented and have an overlapping area; Performing affine transformation on the first sub-image and the second sub-image respectively; Using a segmentation network to perform image segmentation processing on the first sub-image after affine transformation and the second sub-image after affine transformation, respectively, to generate a first segmentation result and a second segmentation result; The segmentation network is trained according to the first segmentation result and the second segmentation result, wherein a training goal is that in the first segmentation result and the second segmentation result, a feature distance between two positions belonging to the same segmentation category is smaller than a feature distance between two positions belonging to different segmentation categories.

2. The method according to claim 1, characterized in that The determining of the first sub-image and the second sub-image from the driving environment image includes: Performing target detection on the driving environment image to obtain a target image containing the object to be segmented; The first sub-image and the second sub-image having overlapping areas are randomly selected from the target image.

3. The method according to claim 2, characterized in that The adopting of the segmentation network to perform image segmentation processing on the first sub-image after affine transformation and the second sub-image after affine transformation to generate a first segmentation result and a second segmentation result includes: Using the segmentation network to perform image segmentation on the target image, obtaining masks corresponding to K categories and feature maps corresponding to the masks; Clustering the features at each position in the feature map with K as an initial value to generate updated masks corresponding to K categories; Performing the same affine transformation as the first sub-image and the second sub-image on the updated masks corresponding to the K categories, respectively, to generate a first mask corresponding to the first sub-image and a second mask corresponding to the second sub-image; The segmentation network is used to perform image segmentation on the first sub-image after affine transformation and the second sub-image after affine transformation, respectively, to obtain a first feature map corresponding to the first sub-image and a second feature map corresponding to the second sub-image; wherein the first mask and the first feature map are the first segmentation result, and the second mask and the second feature map are the second segmentation result.

4. The method according to claim 1, characterized in that The performing affine transformation on the first sub-image and the second sub-image respectively comprises: Randomly select one of flipping, blurring, cropping, and color transformation to process the first sub-image; and A method is randomly selected from flipping, blurring, cropping, and color transformation to process the second sub-image.

5. The method according to claim 1, characterized in that The step of training the segmentation network according to the first segmentation result and the second segmentation result includes: Determine whether any first pixel in the first segmentation result and any second pixel in the second segmentation result belong to the same segmentation category; If the first pixel point and the second pixel point belong to the same segmentation category, calculating the contrast loss according to the feature distance between the first pixel point and the second pixel point and a preset first sample pair label value; If the first pixel point and the second pixel point belong to different segmentation categories, calculating the contrast loss according to the feature distance between the first pixel point and the second pixel point and a preset second sample pair label value; The segmentation network is trained with the goal of minimizing the contrast loss value.

6. The method according to claim 1, characterized in that The step of inputting the image data to be annotated into the segmentation network to generate data annotation results includes: Inputting the image data to be labeled into the segmentation network to generate a third segmentation result; Determine an edge area and a center area of ​​each mask in the third segmentation result; Based on the third segmentation result, training the segmentation network again, wherein the loss calculated at the edge area is ignored during the training process; The image data to be annotated is input into the retrained segmentation network to generate data annotation results.

7. A data annotation method, comprising: Image data to be annotated; Input the image data to be annotated into the segmentation network to generate data annotation results; The segmentation network is a segmentation network trained by the model training method according to any one of claims 1 to 6.

8. A model training device, characterized in that: include: A determination module, configured to determine a first sub-image and a second sub-image from the driving environment image, wherein the first sub-image and the second sub-image contain the object to be segmented and have an overlapping area; A transformation module, used for performing affine transformation on the first sub-image and the second sub-image respectively; A segmentation module, used to perform image segmentation processing on the first sub-image after affine transformation and the second sub-image after affine transformation by using a segmentation network, to generate a first segmentation result and a second segmentation result; A training module is used to train the segmentation network according to the first segmentation result and the second segmentation result, wherein the training goal is: in the first segmentation result and the second segmentation result, the feature distance between two positions belonging to the same segmentation category is smaller than the feature distance between two positions belonging to different segmentation categories.

9. An electronic device, characterized in that: include: processor; a memory for storing instructions executable by the processor; The processor is used to read the executable instructions from the memory and execute the instructions to implement the model training method described in any one of claims 1 to 6 or the data labeling method in claim 7.

10. A computer-readable storage medium, characterized in that: The storage medium stores a computer program, and when the computer program is executed by the processor, it implements the model training method described in any one of claims 1 to 6 or the data labeling method described in claim 7.

11. A vehicle, characterized in that: include: The device as claimed in claim 8, or the electronic device as claimed in claim 9.

Images