Label fusion method and apparatus, computer-readable storage medium, terminal

Abstract

A label fusion method and device, a computer readable storage medium and a terminal, the method comprising: determining one or more frames of labeled frames, and determining a first predicted frame according to optical flow data between the labeled frames and adjacent unlabeled frames, and determining a first optical flow predicted label of each pixel point in the unlabeled frames; calculating a first Euclidean distance between each pixel point in the unlabeled frames and a pixel point at the same position in the first predicted frame, and calculating a second Euclidean distance between the first semantic predicted label of the pixel point and the first optical flow predicted label; determining a first fusion Euclidean distance according to the first Euclidean distance and the second Euclidean distance based on a first preset weight ratio; and determining a first fusion label according to a comparison result of the first fusion Euclidean distance and a first preset threshold. The above scheme can fuse optical flow information and semantic information, and automatically obtain accurate and high-quality fusion labels.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer vision, and in particular to a label fusion method and device, a computer readable storage medium, and a terminal. BACKGROUND

[0002] In the field of computer vision, especially image segmentation, when deep machine learning is performed, a large number of labeled sample data are often needed for model training and optimization. However, in the past, artificial labeling methods were often used to obtain labeled data, which is high in cost, low in efficiency, and unable to obtain a sufficient number of labeled sample data, thereby severely limiting the performance of the model. To solve this problem, people have begun to explore how to replace humans with machines to achieve the automation of label determination and annotation.

[0003] In the prior art, there are generally the following implementation methods for automatically labeling images or video frames:

[0004] (1) Label propagation technology based on optical flow, that is, the label of a labeled frame is automatically propagated to adjacent unlabeled frames in the forward or backward direction by using the motion correlation / optical flow information between adjacent video frames in a video stream to obtain the label of the adjacent unlabeled frame. However, the robustness of this technology is weak because long-time motion estimation is still a very difficult task. In the process of label propagation based on optical flow information / motion information, the accuracy of the estimated optical flow information and the label propagation decreases rapidly with the increase in the number of propagation frames or the length of time, and cumulative bias and noise are easily generated, thereby failing to guarantee the quality of the propagated label and the accuracy of the annotation.

[0005] (2) Automatic labeling technology based on semantics, that is, an unlabeled frame is input into a trained semantic prediction model, and the semantic prediction model directly outputs label information. However, since there is still a lack of a perfect semantic prediction model, it is also difficult to obtain accurate and high-quality labels by simply using the existing semantic prediction model to predict labels.

[0006] (3) Label fusion technology combining optical flow and semantic information: the existing fusion scheme uses a supervised trained optical flow model to predict optical flow labels and fuse them with labels predicted by another model. However, the supervised method for training the optical flow model requires a large number of labeled sample data, and the training cost is very high. Moreover, the structure of the optical flow model used in the prior art is very complex, which may also lead to an increase in the computational overhead. In addition, the existing fusion strategy is often too simple, and the robustness and universality are low. Furthermore, the prior art does not solve the noise problem generated in the label propagation and fusion process, and therefore the fused label is still not accurate enough. SUMMARY

[0007] One of the purposes of the embodiment of the present application is to provide a label fusion method, the present application can fuse optical flow information and semantic information, and automatically obtain accurate and high-quality fusion labels.

[0008] To achieve the above-mentioned purpose, the embodiment of the present application provides a label fusion method, comprising the following steps: determining one or more labeled frames, and determining a first prediction frame at the time of an unlabeled frame according to the optical flow data between the labeled frame and the adjacent unlabeled frame, and determining the first optical flow prediction label of each pixel point in the unlabeled frame; for each pixel point in the unlabeled frame, calculating the first Euclidean distance between the pixel point and the pixel point at the same position in the first prediction frame, and calculating the second Euclidean distance between the first semantic prediction label of the pixel point and the first optical flow prediction label; based on a first preset weight ratio, determining a first fusion Euclidean distance according to the first Euclidean distance and the second Euclidean distance; comparing the first fusion Euclidean distance with a first preset threshold, and determining the first optical flow prediction label or the first semantic prediction label as the first fusion label of the pixel point according to the comparison result.

[0009] Optionally, the determination of one or more labeled frames comprises: determining the labels of one or more images and labeling to obtain the labeled frames.

[0010] Optionally, the optical flow data is the output result of the optical flow prediction model after the labeled frame and the adjacent unlabeled frame are input into the optical flow prediction model.

[0011] Optionally, before determining the first prediction frame at the time of the unlabeled frame according to the optical flow data between the labeled frame and the adjacent unlabeled frame, and determining the first optical flow prediction label of each pixel point in the unlabeled frame, the method further comprises: inputting a plurality of pairs of adjacent unlabeled frames as sample data into a first preset model; for each pair of adjacent unlabeled frames, predicting the optical flow data between the pair of adjacent unlabeled frames and the second prediction frame of any one of the frames, and determining the third Euclidean distance cumulative value of each pair of pixel points belonging to the same position in the second prediction frame and the unlabeled frame to which the second prediction frame belongs; summing the third Euclidean distance cumulative values of each pair of adjacent unlabeled frames as the function value of the first loss function; minimizing the function value of the first loss function to train the first preset model until the first loss function converges, stop iteration and obtain the optical flow prediction model.

[0012] Optionally, the first preset model is selected from any one of the following: traffic flow ARFlow model, recursive full affine transformation RAFT model.

[0013] Optionally, the first semantic prediction label of each pixel point in the unlabeled frame is the output result of the semantic prediction model after the unlabeled frame is input into the semantic prediction model.

[0014] Optionally, before calculating the second Euclidean distance between the first semantic prediction label and the first optical flow prediction label of each pixel in the unmarked frame, the method further comprises: inputting a plurality of marked frames as sample data into a second preset model; predicting the second semantic prediction label of each pixel in each marked frame, and determining the cross entropy between the second semantic prediction label and the actual label; summing the cross entropy between the second semantic prediction label and the actual label of each pixel in each marked frame as the function value of the second loss function; minimizing the function value of the second loss function to train the second preset model until the second loss function converges, stopping iteration and obtaining the semantic prediction model.

[0015] Optionally, the second preset model is selected from any one of the following: a high-resolution network (HRNet) model, a cascade network (CascadePSP) model, and a deep convolutional neural network (DeepLab) model.

[0016] Optionally, the weight value of the first Euclidean distance is greater than the weight value of the second Euclidean distance; comparing the first fusion Euclidean distance with a first preset threshold, and determining the first fusion label of the pixel according to the comparison result includes: if the first fusion Euclidean distance is less than or equal to the first preset threshold, determining the first optical flow prediction label as the first fusion label of the pixel; if the first fusion Euclidean distance is greater than the first preset threshold, determining the first semantic prediction label as the first fusion label of the pixel.

[0017] Optionally, after obtaining the first fusion label of each pixel in the unmarked frame, the method further comprises: using a denoising model to perform denoising processing on the first fusion label of each pixel in the unmarked frame.

[0018] Optionally, before denoising the first fusion label of each pixel point in the unmarked frame by using the denoising model, the method further comprises: determining a second fusion label of each pixel point in the marked frame based on optical flow data between the marked frame and an adjacent unmarked frame, the first fusion label of each pixel point in the unmarked frame, and a third semantic prediction label of each pixel point in the marked frame; inputting the second fusion label of each pixel point in the marked frame as sample data into a third preset model; predicting a denoised second fusion label of each pixel point in each marked frame, and determining a cross entropy between the denoised second fusion label and an actual label of each pixel point in the marked frame; summing the cross entropy between the denoised second fusion label of each pixel point in each marked frame and the actual label of each pixel point in the marked frame as a function value of a third loss function; minimizing the function value of the third loss function to train the third preset model until the third loss function converges, the iteration is stopped, and the denoising model is obtained.

[0019] Optionally, determining the second fusion label of each pixel point in the marked frame based on the optical flow data between the marked frame and an adjacent unmarked frame, the first fusion label of each pixel point in the unmarked frame, and the third semantic prediction label of each pixel point in the marked frame comprises: determining a third prediction frame at the time of the marked frame according to the optical flow data between the marked frame and the adjacent unmarked frame, and determining a second optical flow prediction label of each pixel point in the marked frame according to the optical flow data and the first fusion label of each pixel point in the unmarked frame; for each pixel point in the marked frame, calculating a fourth Euclidean distance between the pixel point and a pixel point at the same position in the third prediction frame, and calculating a fifth Euclidean distance between the third semantic prediction label and the second optical flow prediction label of the pixel point; determining a second fusion Euclidean distance according to the fourth Euclidean distance and the fifth Euclidean distance based on a second preset weight ratio; comparing the second fusion Euclidean distance with a second preset threshold, and determining the second optical flow prediction label or the third semantic prediction label as the second fusion label of the pixel point according to the comparison result.

[0020] Optionally, the third preset model is selected from any one of the following: a U-shaped network U-Net model, and a pixel point prediction pix2pixel model.

[0021] The embodiment of the present application further provides a label fusion device, comprising: a light flow prediction module, determining one or more frames of labeled frames, and determining a first prediction frame at a time of an unlabeled frame according to light flow data between the labeled frames and the adjacent unlabeled frame, and determining a first light flow prediction label of each pixel point in the unlabeled frame; a difference calculation module, configured to calculate a first Euclidean distance between each pixel point in the unlabeled frame and a pixel point at the same position in the first prediction frame, and calculate a second Euclidean distance between a first semantic prediction label of the pixel point and the first light flow prediction label; a fusion difference calculation module, configured to determine a first fusion Euclidean distance according to the first Euclidean distance and the second Euclidean distance based on a first preset weight ratio; and a label fusion module, configured to compare the first fusion Euclidean distance with a first preset threshold, and determine the first light flow prediction label or the first semantic prediction label as a first fusion label of the pixel point according to a comparison result.

[0022] The embodiment of the present application further provides a computer readable storage medium, which stores a computer program, and the computer program is run on a processor to execute the steps of the label fusion method.

[0023] The embodiment of the present application further provides a terminal, comprising a memory and a processor, wherein the memory stores a computer program capable of running on the processor, and the processor runs the computer program to execute the steps of the label fusion method.

[0024] Compared with the prior art, the technical scheme of the embodiment of the present application has the following beneficial effects:

[0025] In the embodiment of the present application, it is determined that one or more frames are labeled frames, and according to the optical flow data between the labeled frames and adjacent unlabeled frames, a first predicted frame at the time of the unlabeled frame is determined, and a first optical flow prediction label of each pixel point in the unlabeled frame is determined; for each pixel point in the unlabeled frame, a first Euclidean distance between the pixel point and a pixel point at the same position in the first predicted frame is calculated, and a second Euclidean distance between the first semantic prediction label and the first optical flow prediction label of the pixel point is calculated; based on a first preset weight ratio, a first fusion Euclidean distance is determined according to the first Euclidean distance and the second Euclidean distance; and then according to a comparison result of the first fusion Euclidean distance and a first preset threshold, a first fusion label of the pixel point is determined. Compared with the existing label propagation technology based on optical flow, with the increase of the number or length of propagation frames, the accuracy of predicted optical flow information and labels decreases quickly; or compared with the existing automatic labeling technology based on semantics, due to the lack of a relatively perfect semantic prediction model, only relying on the existing semantic prediction model to predict labels also has insufficient accuracy; in the embodiment of the present application, the optical flow information and semantic information are fused in the process of determining the label, the optical flow data between the labeled frames and adjacent unlabeled frames is first applied to the labeled frames to obtain a predicted frame and an optical flow prediction label, then the unlabeled frame and the predicted frame are compared to calculate the difference between the two, and at the same time, the optical flow prediction label and the semantic prediction label of the unlabeled frame are compared to calculate the difference between the two, then a weighted strategy is introduced to calculate the fusion difference result, and according to the comparison result of the fusion difference result and the first preset threshold, a relatively accurate and high-quality fusion label of the adjacent unlabeled frame can be obtained, and the fusion scheme can be iterated for each pair of adjacent frames to realize automatic and efficient determination of the fusion label of each unlabeled frame.

[0026] Further, the optical flow data is a result output by the optical flow prediction model after the labeled frames and adjacent unlabeled frames are input into the optical flow prediction model. In addition, the optical flow prediction model can be obtained by inputting a plurality of pairs of adjacent unlabeled frames as sample data into a first preset model for training. Compared with the existing optical flow and semantic label fusion technology, the optical flow model with complex structure obtained by using a supervised training method (which requires a large amount of labeled sample data) has the problems of difficulty in obtaining labeled sample data for model training, high training cost, and large computational overhead; in the embodiment of the present application, the optical flow prediction model used for predicting the optical flow data is obtained by using a self-supervised method (sample data does not need to be labeled), which is easy to collect a sufficient number of unlabeled sample data for model training, and can obtain an optimized optical flow prediction model, thereby helping to improve the accuracy of optical flow prediction and label propagation.

[0027] Further, after obtaining the first fusion label of each pixel point in the unmarked frame, the method further comprises: adopting a denoising model to perform denoising processing on the first fusion label of each pixel point in the unmarked frame. The denoising model can be obtained by inputting the second fusion label of each pixel point in the marked frame obtained by performing reverse fusion processing on the obtained first fusion label of each pixel point in the unmarked frame into a third preset model for training. In the embodiment of the application, the first optical flow prediction label or the first semantic prediction label itself may have noise, and in addition, the fusion process is performed on a pixel-by-pixel basis to select the first optical flow prediction label / first optical flow prediction label, which may also introduce noise (for example, two pixel points of the same object, one of which has a fusion label of class 1 and the other of which has a fusion label of class 2, and such fusion is erroneous). By adopting the denoising model obtained by training to perform denoising processing, the details of the fusion label can be corrected to further improve the accuracy. BRIEF DESCRIPTION OF DRAWINGS

[0028] Figure 1 is a flowchart of a label fusion method in an embodiment of the application;

[0029] Figure 2 is a flowchart of another label fusion method in an embodiment of the application;

[0030] Figure 3 is a structural schematic diagram of a label fusion device in an embodiment of the application. DETAILED DESCRIPTION

[0031] As described above, in order to obtain a sufficient number of labeled sample data, people began to explore how to replace manual labeling with machine automation.

[0032] In the prior art, automatic labeling of images or video frames generally has the following implementation methods: (1) label propagation technology based on optical flow, that is, using the motion correlation / optical flow information between adjacent video frames in a video stream, the label of a marked frame is automatically propagated to adjacent unmarked frames to obtain the label of the adjacent unmarked frame. (2) automatic labeling technology based on semantics, that is, by inputting an unmarked frame into a trained semantic prediction model, the semantic prediction model directly outputs label information. (3) label fusion technology combining optical flow and semantic information: the existing fusion scheme adopts a supervised trained optical flow model to predict an optical flow label, and combines the label predicted by another model.

[0033] The inventor of the present application has found through research that in the process of label propagation according to optical flow information / motion information, the accuracy of the estimated optical flow information and label propagation decreases rapidly with the increase of the number of propagation frames or the length of time, and cumulative bias and noise are easily generated; in the automatic labeling technology based on semantics, a perfect semantic prediction model is still lacking; and in the existing optical flow and semantic label fusion technology, a large amount of sample data with labels is needed to train the optical flow model by using a supervised method, the training cost is high, the calculation overhead is large, and the noise problem generated in the label propagation and fusion process is not solved, so the above existing technologies may not be able to obtain accurate and high-quality fusion labels.

[0034] In the embodiment of the present application, one or more labeled frames are determined, and a first prediction frame at the time of an unlabeled frame is determined according to the optical flow data between the labeled frame and the adjacent unlabeled frame, and a first optical flow prediction label of each pixel point in the unlabeled frame is determined; for each pixel point in the unlabeled frame, a first Euclidean distance between the pixel point and a pixel point at the same position in the first prediction frame is calculated, and a second Euclidean distance between the first semantic prediction label of the pixel point and the first optical flow prediction label is calculated; a first fusion Euclidean distance is determined according to the first Euclidean distance and the second Euclidean distance based on a first preset weight ratio; and a first fusion label of the pixel point is determined according to a comparison result of the first fusion Euclidean distance and a first preset threshold. Compared with the existing label propagation technology based on optical flow, the accuracy of the predicted optical flow information and label decreases rapidly with the increase of the number of propagation frames or the length of time; or compared with the existing automatic labeling technology based on semantics, since a perfect semantic prediction model is lacking, only relying on the existing semantic prediction model to predict the label also has insufficient accuracy; in the embodiment of the present application, the optical flow information and semantic information are fused in the process of determining the label, the prediction frame and the optical flow prediction label are obtained by acting the optical flow data between the labeled frame and the adjacent unlabeled frame on the labeled frame, then the difference between the unlabeled frame and the prediction frame is calculated by comparison, and the difference between the optical flow prediction label of the unlabeled frame and the semantic prediction label is calculated by comparison, then a weighted strategy is introduced to calculate the fusion difference result, and a more accurate and high-quality fusion label can be obtained according to the comparison result of the fusion difference result and the first preset threshold.

[0035] In order to make the above-mentioned purposes, features and benefits of the present application more obvious and easy to understand, specific embodiments of the present application will be described in detail below with reference to the accompanying drawings.

[0036] Reference Figure 1 , Figure 1 is a flowchart of a label fusion method in the embodiment of the present application. The method can include steps S11 to S14:

[0037] Step S11: determining one or more frames of labeled frames, and determining a first predicted frame at a time of an unlabeled frame according to optical flow data between the labeled frames and the adjacent unlabeled frame, and determining a first optical flow prediction label of each pixel point in the unlabeled frame;

[0038] Step S12: for each pixel point in the unlabeled frame, calculating a first Euclidean distance between the pixel point and a pixel point at a same position in the first predicted frame, and calculating a second Euclidean distance between a first semantic prediction label of the pixel point and the first optical flow prediction label;

[0039] Step S13: determining a first fused Euclidean distance according to the first Euclidean distance and the second Euclidean distance based on a first preset weight ratio;

[0040] Step S14: comparing the first fused Euclidean distance with a first preset threshold, and determining the first optical flow prediction label or the first semantic prediction label as a first fused label of the pixel point according to a comparison result.

[0041] In the implementation of step S11, the labeled frame can be obtained by determining and labeling a label of one or more frames of images. Specifically, one or more frames of scene images can be selected from a plurality of frames of scene images continuously captured in a certain time for the same scene (for example, a traffic scene containing a driving vehicle, an obstacle, and a road), and a label is determined and labeled. Alternatively, a plurality of video frames can be obtained by frame extraction processing of a video stream every preset time length (for example, every 0.5 seconds or 1 second), and then a label of a video frame is determined and labeled every preset frame number (every 5 frames) from the plurality of video frames. Other methods can also be used to obtain the labeled frame, and the embodiments of the present application do not limit this.

[0042] The method of labeling the image / video frame can be manually determining a label for labeling, or extracting a verified effective label from a constructed map for labeling, or other methods can be used to determine a label, and the embodiments of the present application do not limit this.

[0043] The adjacent unlabeled frame can be a previous frame or a next frame of the labeled frame in the above-mentioned plurality of frames of continuously captured scene images. Alternatively, the adjacent unlabeled frame can be a previous frame or a next frame of the labeled frame in the above-mentioned plurality of video frames obtained by frame extraction processing of a video stream.

[0044] The optical flow can refer to the mode motion speed in the time-varying image, because when the object is in motion, the brightness mode of the corresponding point on the image is also in motion, so the optical flow can express the change of the image, including the motion information of the target, and thus can be used by the observer to determine the motion condition of the target. Specifically, the optical flow data can be the offset of each pixel point in the labeled frame and the adjacent unlabeled frame in the time interval of the two adjacent frames, so the optical flow data can be a two-dimensional matrix form, and each value in the two-dimensional matrix is used to indicate the front and back offset of the same pixel point.

[0045] Specifically, according to the optical flow data between the labeled frame and the adjacent unlabeled frame, a first prediction frame at the time of the unlabeled frame is determined, that is, according to the offset of each pixel point in the labeled frame from the time of the labeled frame to the time of the unlabeled frame, the position of each pixel point in the labeled frame at the time of the unlabeled frame can be determined, thereby obtaining the first prediction frame.

[0046] Specifically, according to the optical flow data between the labeled frame and the adjacent unlabeled frame, a first optical flow prediction label of each pixel point in the unlabeled frame is determined. That is, according to the offset of each pixel point in the labeled frame from the time of the labeled frame to the time of the unlabeled frame, and the actual label of each pixel point in the labeled frame, the prediction label of each pixel point in the first prediction frame can be obtained, and the prediction label of each pixel point in the first prediction frame is used as the first optical flow prediction label of each pixel point in the unlabeled frame one by one.

[0047] Further, the optical flow data can be the result output by the optical flow prediction model after the labeled frame and the adjacent unlabeled frame are input into the optical flow prediction model.

[0048] Further, before determining the first prediction frame at the time of the unlabeled frame according to the optical flow data between the labeled frame and the adjacent unlabeled frame, and determining the first optical flow prediction label of each pixel point in the unlabeled frame, the method further comprises: inputting a plurality of pairs of adjacent unlabeled frames as sample data into a first preset model; for each pair of adjacent unlabeled frames, predicting the optical flow data between the pair of adjacent unlabeled frames and the second prediction frame of any one frame, and determining the third Euclidean distance cumulative value of each pair of pixel points belonging to the same position in the second prediction frame and the unlabeled frame to which the second prediction frame belongs; summing the third Euclidean distance cumulative values of each pair of adjacent unlabeled frames as the function value of the first loss function; minimizing the function value of the first loss function to train the first preset model until the first loss function converges, stop iteration and obtain the optical flow prediction model.

[0049] The multiple pairs of adjacent unlabeled frames can be multiple pairs of adjacent video frames continuously extracted or extracted every preset frame number from one or more videos, can be multiple pairs of adjacent scene images of a scene continuously photographed, or can be multiple pairs of adjacent scene images selected from multiple frames of scene images continuously photographed for one or more scenes, or can be multiple pairs of adjacent unlabeled frames determined in other conventional manners, and the embodiments of the present application do not limit this.

[0050] For each pair of adjacent unlabeled frames, the optical flow data between the pair of adjacent unlabeled frames is predicted, and a second prediction frame of any one of the frames is predicted, wherein the second prediction frame of any one of the frames refers to a second prediction frame of the frame at another frame time, which is further predicted according to the predicted optical flow data (offset of each pixel point).

[0051] It should be noted that the multiple pairs of adjacent unlabeled frames determined above are sample data without the need for determining labels and labeling, and can be directly input into the first preset model and trained in a self-supervised manner.

[0052] As some non-limiting examples, the first preset model can be selected from any one of the following: a traffic flow ARFlow model, a recursive full pair field transformation RAFT model, or can be various existing appropriate network models with learning ability, and the embodiments of the present application do not limit this.

[0053] As some non-limiting examples, the first loss function can be an optical flow mapping loss function or a smoothing loss function using a mean square error, or can be other types of loss functions that can achieve the same or similar functions, and the embodiments of the present application do not limit this.

[0054] It can be understood that by training the first preset model using the multiple pairs of adjacent unlabeled frames as sample data, the first preset model can learn the mapping relationship between the adjacent two frames of images (or video frames) and the optical flow data between the adjacent two frames of images (or video frames), and the trained optical flow prediction model can predict more accurate optical flow data between the adjacent two frames of images (or video frames) according to the input adjacent two frames of images (or video frames).

[0055] In the embodiments of the present application, compared with the existing optical flow and semantic label fusion technology, a complex optical flow model obtained by a supervised training method (which requires a large amount of labeled sample data) is used for model training, and it is difficult to obtain labeled sample data for model training, the training cost is high, and the operation cost is large; the optical flow prediction model obtained by the self-supervised method (the sample data does not need to be labeled) is used to predict the optical flow data in the embodiments of the present application, and it is easy to collect a large number of unlabeled sample data for model training, and a more optimized optical flow prediction model is obtained, thereby helping to improve the accuracy of optical flow prediction and label propagation.

[0056] In the implementation of step S12, the first semantic prediction label of each pixel point in the unlabeled frame can be an output result of the semantic prediction model after the unlabeled frame is input into the semantic prediction model.

[0057] Further, before the second Euclidean distance between the first semantic prediction label and the first optical flow prediction label of each pixel point in the unlabeled frame is calculated, the method further comprises: inputting a plurality of labeled frames as sample data into a second preset model; predicting a second semantic prediction label of each pixel point in each labeled frame, and determining a cross-entropy between the second semantic prediction label and an actual label; summing the cross-entropy between the second semantic prediction label and the actual label of each pixel point in each labeled frame as a function value of a second loss function; minimizing the function value of the second loss function to train the second preset model until the second loss function converges, stop iteration and obtain the semantic prediction model.

[0058] The plurality of labeled frames can be the labeled frames determined in step S11; can be determined by randomly or continuously extracting a plurality of video frames from one or more videos, labeling and marking to obtain, or can be determined by randomly or continuously shooting a plurality of scene images for one or more scenes, labeling and marking to obtain; or can be determined by other conventional methods, and the embodiments of the present application do not limit this.

[0059] It should be noted that when the second preset model is trained, the sample data and the training method for training the first preset model are different: the sample data for training the second preset model is a plurality of labeled frames (i.e. all labeled images or video frames); the training method adopts a supervised training method.

[0060] As some non-limiting embodiments, the second preset model can be selected from any one of the following: a high-resolution network (HRNet) model, a cascade network (CascadePSP) model, and a deep convolutional neural network (DeepLab) model; or can be any existing network model with learning ability, and the embodiments of the present application do not limit this.

[0061] As some non-limiting embodiments, the second loss function can be a cross-entropy loss function or a binary cross-entropy loss function, or can be any other type of loss function that can achieve the same or similar function, and the embodiments of the present application do not limit this.

[0062] It can be understood that, by training the second preset model by using the multiple frames of labeled frames as sample data, the second preset model can learn the mapping relationship between each frame of image (or video frame) and the semantic label thereof, and the semantic prediction model obtained through the training can predict the semantic prediction label of the unlabeled frame according to the input unlabeled frame.

[0063] In the implementation of step S13, the first fused Euclidean distance is determined according to the first Euclidean distance and the second Euclidean distance based on the first preset weight ratio.

[0064] In one specific implementation of the embodiment of the present application, the step of determining the first fused Euclidean distance according to the first Euclidean distance and the second Euclidean distance can include: taking the sum value of the product of the first Euclidean distance and the first weight and the product of the second Euclidean distance and the second weight as the first fused Euclidean distance.

[0065] In another specific implementation of the embodiment of the present application, the step of determining the first fused Euclidean distance according to the first Euclidean distance and the second Euclidean distance can include: calculating the average value of the product of the first Euclidean distance and the first weight and the product of the second Euclidean distance and the second weight as the first fused Euclidean distance.

[0066] The first preset weight ratio is the ratio of the first weight to the second weight.

[0067] The first Euclidean distance is used to indicate the difference between the unlabeled frame and the first predicted frame, the second Euclidean distance is used to indicate the difference between the first optical flow prediction label and the first semantic prediction label of the unlabeled frame, and the first fused Euclidean distance is used to indicate the fused difference calculated after introducing the weighting strategy.

[0068] In one specific embodiment, the weight value of the first Euclidean distance is greater than the weight value of the second Euclidean distance, for example, the first preset weight ratio is 7:3, that is, a relatively larger weight is assigned to the difference between the unlabeled frame and the first predicted frame, and a relatively smaller weight is assigned to the first semantic prediction label and the first optical flow prediction label of the unlabeled frame, which helps to improve the accuracy of the fused difference (first fused label) calculated subsequently.

[0069] In the implementation of step S14, the first fused Euclidean distance is compared with the first preset threshold, and according to the comparison result, it is determined whether the first optical flow prediction label or the first semantic prediction label is taken as the first fused label of the pixel point.

[0070] Further, the weight value of the first Euclidean distance is greater than the weight value of the second Euclidean distance; the comparing the first fused Euclidean distance with a first preset threshold and determining the first optical flow prediction label or the first semantic prediction label as the first fused label of the pixel point according to a comparison result includes: if the first fused Euclidean distance is less than or equal to the first preset threshold, determining the first optical flow prediction label as the first fused label of the pixel point; if the first fused Euclidean distance is greater than the first preset threshold, determining the first semantic prediction label as the first fused label of the pixel point.

[0071] It should be noted that the value of the first preset threshold should not be too large, otherwise the first fused Euclidean distance will be less than or equal to the first preset threshold with a high probability, and the first optical flow prediction label will be determined as the first fused label of each pixel point with a high probability, thereby possibly reducing the accuracy of the determined first fused label of each pixel point. The value of the first preset threshold should not be too small, otherwise the first fused Euclidean distance will be greater than the first preset threshold with a high probability, and the first semantic prediction label will be determined as the first fused label of each pixel point with a high probability, which also possibly reduces the accuracy of the determined first fused label of each pixel point.

[0072] In specific implementations, the first preset threshold is between the minimum value and the maximum value of the first Euclidean distance, for example: the minimum value of the first Euclidean distance is 0 and the maximum value is 10, and the first preset threshold can be 8. In some non-limiting embodiments, the first preset threshold can be 80% to 95% of the first Euclidean distance, and 35% to 40% of the second Euclidean distance.

[0073] Specifically, determining the first fused label of each pixel point in the unlabeled frame, that is, determining the category of each pixel point, can specifically include: road, vehicle, obstacle, etc. categories, and can also include drivable area, obstacle, background area, etc. categories, but is not limited thereto. In specific implementations, the image or video frame and its corresponding first fused label can constitute a set of labeled data, and these labeled data can be used as sample data to train semantic models and various models to optimize the performance of the models. The trained optimized model is deployed on an autonomous vehicle to realize automatic identification of various scenes in the driving environment.

[0074] Compared with the existing label propagation technology based on optical flow, the accuracy of the predicted optical flow information and the label decreases quickly with the increase of the propagation frame number or time length; or compared with the existing automatic labeling technology based on semantics, due to the lack of a relatively perfect semantic prediction model, only relying on the existing semantic prediction model to predict the label is also insufficient in accuracy; the embodiment of the present application fuses optical flow information and semantic information in the process of determining the label, first applies the optical flow data between the labeled frame and the adjacent unlabeled frame to the labeled frame to obtain a predicted frame and an optical flow predicted label, then compares the unlabeled frame and the predicted frame to calculate the difference between the two, and at the same time, compares the optical flow predicted label and the semantic predicted label of the unlabeled frame to calculate the difference between the two, then introduces a weighting strategy to calculate the fusion difference result, and according to the comparison result of the fusion difference result and the first preset threshold, a more accurate and high-quality fusion label of the adjacent unlabeled frame can be obtained, and the above fusion scheme can be iterated for each pair of adjacent frames to realize automatic and efficient determination of the fusion label of each unlabeled frame.

[0075] Referring to Figure 2 , Figure 2 is a flowchart of another label fusion method in the embodiment of the present application. The another label fusion method can include steps S11 to S14, and can also include step S21, wherein step S21 can be executed after step S14, and the different content in step S21 will be described below. Figure 1

[0076] In step S21, a denoising model is used to perform denoising processing on the first fusion label of each pixel point in the unlabeled frame.

[0077] Further, before the denoising model is used to perform denoising processing on the first fusion label of each pixel point in the unlabeled frame, the method further includes: determining a second fusion label of each pixel point in the labeled frame based on the optical flow data between the labeled frame and the adjacent unlabeled frame, the first fusion label of each pixel point in the unlabeled frame, and the third semantic prediction label of each pixel point in the labeled frame; inputting the second fusion label of each pixel point in multiple frames of the labeled frame as sample data into a third preset model; predicting the denoised second fusion label of each pixel point in each frame of the labeled frame, and determining the cross entropy between the denoised second fusion label and the actual label; summing the cross entropy between the denoised second fusion label of each pixel point in each frame of the labeled frame and the actual label as a function value of a third loss function; minimizing the function value of the third loss function to train the third preset model until the third loss function converges, stop iteration and obtain the denoising model.

[0078] ​As some non-limiting examples, the third preset model can be selected from any one of the following: a U-shaped network U-Net model, a pixel point prediction pix2pixel model; or can be other various appropriate network models with learning ability, and embodiments of the present application do not limit this.

[0079] As some non-limiting examples, the second loss function can be a cross-entropy loss function, a mean square error loss function, an absolute error loss function, or other types of loss functions that can achieve the same or similar functions, and embodiments of the present application do not limit this.

[0080] It can be understood that by using the second fusion label of each pixel point in the multiple frames of the labeled frames as sample data to train the third preset model, the third preset model can learn the mapping relationship between the fusion label and the de-noised fusion label, and the de-noising model trained can predict the de-noised fusion label as accurate and high quality as possible according to the input fusion label.

[0081] In specific implementation, the process of determining the second fusion label of each pixel point in the labeled frame based on the optical flow data between the labeled frame and the adjacent unlabeled frame, the first fusion label of each pixel point in the unlabeled frame, and the third semantic prediction label of each pixel point in the labeled frame can refer to the process of determining the first fusion label of each pixel point in the unlabeled frame in steps S11 to S14 in FIG. 1, and specific steps are as follows: Figure 1

[0082] According to the optical flow data between the labeled frame and the adjacent unlabeled frame, a third prediction frame at the time of the labeled frame is determined, and according to the optical flow data and the first fusion label of each pixel point in the unlabeled frame, a second optical flow prediction label of each pixel point in the labeled frame is determined; for each pixel point in the labeled frame, a fourth Euclidean distance between the pixel point and a pixel point at the same position in the third prediction frame is calculated, and a fifth Euclidean distance between the third semantic prediction label of the pixel point and the second optical flow prediction label is calculated; based on a second preset weight ratio, the second fusion Euclidean distance is determined according to the fourth Euclidean distance and the fifth Euclidean distance; the second fusion Euclidean distance is compared with a second preset threshold, and according to the comparison result, the second optical flow prediction label or the third semantic prediction label is determined as the second fusion label of the pixel point.

[0083] ​Specifically, according to the optical flow data between the labeled frame and the adjacent unlabeled frame, a third prediction frame at the time of the labeled frame is determined, that is, according to the offset of each pixel point in the unlabeled frame from the time of the unlabeled frame to the time of the labeled frame (which is the negative of the offset from the time of the labeled frame to the time of the unlabeled frame), the position of each pixel point in the unlabeled frame at the time of the labeled frame can be determined, thereby obtaining the third prediction frame.

[0084] Specifically, according to the optical flow data and the first fusion label of each pixel point in the unlabeled frame, a second optical flow prediction label of each pixel point in the labeled frame is determined, that is, according to the offset of each pixel point in the unlabeled frame from the time of the unlabeled frame to the time of the labeled frame, and the first fusion label of each pixel point in the unlabeled frame, the prediction label of each pixel point in the third prediction frame can be obtained, and the prediction label of each pixel point in the third prediction frame is one-to-one as the second optical flow prediction label of each pixel point in the labeled frame.

[0085] The determination method of the third semantic prediction label of each pixel point in the labeled frame can refer to the determination method of the first semantic prediction label in step S12, that is, the output result of the semantic prediction model after the labeled frame is input into the semantic prediction model.

[0086] In the embodiment of the application, the first optical flow prediction label and the first semantic prediction label obtained by prediction may not be accurate enough (with noise), and in addition, the fusion process is performed on a pixel-by-pixel basis to select the first optical flow prediction label / first optical flow prediction label, which may also introduce noise (for example, two pixel points of the same object, one of which has a fusion label of class 1 and the other of which has a fusion label of class 2, such fusion is erroneous), and by using the trained denoising model for denoising processing, the accuracy of the fusion label can be further improved.

[0087] Referring to Figure 3 , Figure 3 is a structural schematic diagram of a label fusion device in an embodiment of the application. The label fusion device can include:

[0088] The optical flow prediction module 31 determines one or more labeled frames, and according to the optical flow data between the labeled frame and the adjacent unlabeled frame, a first prediction frame at the time of the unlabeled frame is determined, and a first optical flow prediction label of each pixel point in the unlabeled frame is determined;

[0089] The difference calculation module 32 is configured to calculate a first Euclidean distance between each pixel in the unmarked frame and a pixel at a same position in the first prediction frame, and calculate a second Euclidean distance between a first semantic prediction label of the pixel and a first optical flow prediction label.

[0090] The fusion difference calculation module 33 is configured to determine a first fusion Euclidean distance according to the first Euclidean distance and the second Euclidean distance based on a first preset weight ratio.

[0091] The label fusion module 34 is configured to compare the first fusion Euclidean distance with a first preset threshold, and determine the first optical flow prediction label or the first semantic prediction label as a first fusion label of the pixel according to a comparison result.

[0092] For the principle, specific implementation and beneficial effects of the label fusion device, please refer to the foregoing and the related description of the label fusion method shown in the Figures 1 to 2 The related description of the label fusion method, which will not be repeated here.

[0093] The embodiment of the application further provides a computer readable storage medium, which stores a computer program, and the computer program is run by a processor to execute the steps of the label fusion method. The computer readable storage medium can include a non-volatile memory or a non-transitory memory, and can also include an optical disc, a mechanical hard disk, a solid state disk and the like.

[0094] Specifically, in the embodiment of the application, the processor can be a central processing unit (CPU), and the processor can also be other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components and the like. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor and the like.

[0095] It should also be understood that the memory in the embodiments of the present application can be a volatile memory or a non-volatile memory, or can include both volatile and non-volatile memories. Among them, the non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM) or flash memory. The volatile memory can be random access memory (RAM) used as an external cache. By way of example, but not by way of limitation, many forms of random access memory (RAM) are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchlink dynamic random access memory (SLDRAM) and direct rambus random access memory (DR RAM).

[0096] The embodiments of the present application also provide a terminal, including a memory and a processor, the memory stores a computer program capable of running on the processor, and the processor executes the steps of the label fusion method when running the computer program. The terminal can include but is not limited to mobile phones, computers, tablet computers and other terminal devices, and can also be servers, cloud platforms and the like.

[0097] It should be understood that the term "and / or" herein only describes the association relationship of the associated objects, which means that there can be three relationships, for example, A and / or B, which can represent three cases of A alone, A and B together, and B alone. In addition, the character " / " herein represents that the front and rear associated objects are in an "or" relationship.

[0098] The "multiple" appearing in the embodiments of the present application means two or more.

[0099] The first, second and the like appearing in the embodiments of the present application are only for indicating and distinguishing the described objects, and do not have sequence, and do not represent the special limitation of the number of devices in the embodiments of the present application, and cannot constitute any limitation on the embodiments of the present application.

[0100] It should be noted that the serial numbers of the steps in the embodiments do not represent the limitation of the execution sequence of the steps.

[0101] Although the present application is disclosed as above, the present application is not limited to this. Any person skilled in the art can make various changes and modifications without departing from the spirit and scope of the present application, and therefore the protection scope of the present application should be subject to the range defined by the claims.

Claims

1. A label fusion method characterized by, The method comprises the following steps: determining one or more labeled frames, and determining a first predicted frame at a time of an unlabeled frame adjacent to the labeled frame according to optical flow data between the labeled frame and the unlabeled frame, and determining a first optical flow prediction label of each pixel point in the unlabeled frame; for each pixel point in the unlabeled frame, calculating a first Euclidean distance between the pixel point and a pixel point at a same position in the first predicted frame, and calculating a second Euclidean distance between a first semantic prediction label of the pixel point and the first optical flow prediction label; determining a first fusion Euclidean distance according to the first Euclidean distance and the second Euclidean distance based on a first preset weight ratio; comparing the first fusion Euclidean distance with a first preset threshold, and determining the first optical flow prediction label or the first semantic prediction label as a first fusion label of the pixel point according to a comparison result.

2. The method of claim 1, wherein, The method further comprises the following steps before determining one or more labeled frames: determining labels of one or more images and labeling to obtain the labeled frames.

3. The method of claim 1, wherein, The optical flow data is a result output by an optical flow prediction model after inputting the labeled frame and the adjacent unlabeled frame into the optical flow prediction model.

4. The method of claim 3, wherein, The method further comprises the following steps before determining a first predicted frame at a time of an unlabeled frame adjacent to the labeled frame according to optical flow data between the labeled frame and the unlabeled frame, and determining a first optical flow prediction label of each pixel point in the unlabeled frame: inputting a plurality of pairs of adjacent unlabeled frames as sample data into a first preset model; for each pair of adjacent unlabeled frames, predicting optical flow data between the pair of adjacent unlabeled frames and a second predicted frame of any one of the pair of adjacent unlabeled frames, and determining a third Euclidean distance cumulative value of each pair of pixel points belonging to a same position in the second predicted frame and the unlabeled frame to which the second predicted frame belongs; summing the third Euclidean distance cumulative values of the pairs of adjacent unlabeled frames as a function value of a first loss function; minimizing the function value of the first loss function to train the first preset model until the first loss function converges, stopping iteration and obtaining the optical flow prediction model.

5. The method of claim 4, wherein, The first preset model is selected from any one of the following: a traffic flow ARFlow model and a recursive full affine transformation RAFT model.

6. The method of claim 1, wherein, The first semantic prediction label of each pixel point in the unlabeled frame is an output result of a semantic prediction model after inputting the unlabeled frame into the semantic prediction model.

7. The method of claim 6, wherein, The method further comprises the following steps before calculating a second Euclidean distance between a first semantic prediction label of each pixel point in the unlabeled frame and the first optical flow prediction label: inputting a plurality of labeled frames as sample data into a second preset model; predicting a second semantic prediction label of each pixel point in each labeled frame, and determining a cross entropy between the second semantic prediction label and an actual label; summing the cross entropies between the second semantic prediction labels and the actual labels of each pixel point in each labeled frame as a function value of a second loss function; minimizing the function value of the second loss function to train the second preset model until the second loss function converges, stopping iteration and obtaining the semantic prediction model.

8. The method of claim 7, wherein, The second preset model is selected from any one of the following: A high-resolution network (HRNet) model, a cascade network (CascadePSP) model, and a deep convolutional neural network (DeepLab) model.

9. The method of claim 1, wherein, The weight value of the first Euclidean distance is greater than the weight value of the second Euclidean distance. The first fusion Euclidean distance is compared with a first preset threshold, and according to a comparison result, it is determined whether the first optical flow prediction label or the first semantic prediction label is taken as the first fusion label of the pixel point. If the first fusion Euclidean distance is less than or equal to the first preset threshold, the first optical flow prediction label is determined as the first fusion label of the pixel point. If the first fusion Euclidean distance is greater than the first preset threshold, the first semantic prediction label is determined as the first fusion label of the pixel point.

10. The method of claim 1, wherein, After obtaining the first fusion label of each pixel point in the unmarked frame, the method further comprises: A denoising model is used to perform denoising processing on the first fusion label of each pixel point in the unmarked frame.

11. The method of claim 10, wherein, Before the denoising model is used to perform denoising processing on the first fusion label of each pixel point in the unmarked frame, the method further comprises: Based on the optical flow data between the marked frame and the adjacent unmarked frame, the first fusion label of each pixel point in the unmarked frame, and the third semantic prediction label of each pixel point in the marked frame, a second fusion label of each pixel point in the marked frame is determined; A plurality of second fusion labels of each pixel point in the marked frame are used as sample data to input a third preset model; The denoised second fusion label of each pixel point in each frame of the marked frame is predicted, and the cross entropy between the denoised second fusion label and the actual labeled label is determined; The cross entropy between the denoised second fusion label of each pixel point in each frame of the marked frame and the actual labeled label is summed as a function value of a third loss function; The function value of the third loss function is minimized to train the third preset model until the third loss function converges, the iteration is stopped, and the denoising model is obtained.

12. The method of claim 11, wherein, Based on the optical flow data between the marked frame and the adjacent unmarked frame, the first fusion label of each pixel point in the unmarked frame, and the third semantic prediction label of each pixel point in the marked frame, a second fusion label of each pixel point in the marked frame is determined; According to the optical flow data between the marked frame and the adjacent unmarked frame, a third prediction frame at the time of the marked frame is determined, and according to the optical flow data and the first fusion label of each pixel point in the unmarked frame, a second optical flow prediction label of each pixel point in the marked frame is determined; For each pixel point in the marked frame, a fourth Euclidean distance between the pixel point and a pixel point at the same position in the third prediction frame is calculated, and a fifth Euclidean distance between the third semantic prediction label of the pixel point and the second optical flow prediction label is calculated; Based on a second preset weight ratio, a second fusion Euclidean distance is determined according to the fourth Euclidean distance and the fifth Euclidean distance; The second fusion Euclidean distance is compared with a second preset threshold, and according to a comparison result, it is determined whether the second optical flow prediction label or the third semantic prediction label is taken as the second fusion label of the pixel point.

13. The method of claim 11, wherein, The third preset model is selected from any one of the following: a U-shaped network U-Net model, and a pixel point prediction pix2pixel model.

14. A label fusion apparatus characterized by comprising: The method comprises the following steps: an optical flow prediction module, which determines one or more labeled frames, and determines a first predicted frame at a time of an unlabeled frame according to optical flow data between the labeled frame and the adjacent unlabeled frame, and determines a first optical flow prediction label of each pixel point in the unlabeled frame; a difference calculation module, which calculates a first Euclidean distance between the pixel point and a pixel point at the same position in the first predicted frame, and calculates a second Euclidean distance between the first semantic prediction label and the first optical flow prediction label of the pixel point for each pixel point in the unlabeled frame; a fusion difference calculation module, which determines a first fusion Euclidean distance according to the first Euclidean distance and the second Euclidean distance based on a first preset weight ratio; a label fusion module, which compares the first fusion Euclidean distance with a first preset threshold, and determines the first optical flow prediction label or the first semantic prediction label as a first fusion label of the pixel point according to a comparison result.

15. A computer readable storage medium having stored thereon a computer program, characterized in that, The computer program is run by the processor to execute the steps of the label fusion method in any one of claims 1 to 13.

16. A terminal comprising a memory and a processor, said memory having stored thereon a computer program capable of running on said processor, characterized in that, The processor runs the computer program to execute the steps of the label fusion method in any one of claims 1 to 13.

Images