Semi-supervised semantic segmentation network training method and device

Abstract

The invention provides a semi-supervised semantic segmentation network training method and device, and the method comprises the steps: training an initial detection model provided with semantic segmentation branches based on an annotation data set, and obtaining a first target detection model; performing reasoning analysis on the residual sample images in the original data set based on the semantic segmentation branch of the first target detection model to obtain a semantic segmentation result; carrying out evaluation analysis on a correctable segmentation result in the semantic segmentation result, and determining an available segmentation result; and merging a target segmentation result and an available segmentation result in the semantic segmentation result into an annotation data set, training a first target detection model by using the obtained new annotation data set, and obtaining a target detection model under the condition that a preset model iteration termination condition is satisfied. According to the semi-supervised semantic segmentation network training method provided by the invention, the labeling workload in the model training process can be effectively reduced, so that the model training efficiency is improved, and the labor cost is reduced.

Description

Technical Field

[0001] The present invention relates to the field of image detection technology, and in particular to a semi-supervised semantic segmentation network training method and device, and also to an electronic device and a non-transitory computer-readable storage medium. Background Art

[0002] In recent years, with the continuous development of deep learning target detection technology, related intelligent products have been more and more widely used, such as face detection, automatic driving, and automatic recognition of various targets in the security field. Deep learning target detection technology is based on massive labeled images. The richness of sample images and the accuracy of annotations directly determine the training effect of the detection model. However, it is difficult to obtain a sufficient batch of sample images in actual application scenarios. Even if enough sample images are collected, fine semantic segmentation and annotation are difficult and the workload is very large. Therefore, how to achieve semi-automatic annotation to improve the feature extraction efficiency of current small sample images has become a key issue for the implementation of intelligent products. Summary of the invention

[0003] The present invention provides a semi-supervised semantic segmentation network training method to solve the defects in the prior art that fine semantic segmentation annotation is difficult and labor-intensive, resulting in poor model training efficiency.

[0004] The present invention provides a semi-supervised semantic segmentation network training method, comprising: Obtaining a labeled data set; wherein the labeled data set is obtained by obtaining a target sample image from an original data set and performing polygonal semantic segmentation and annotation; Based on the labeled data set, an initial detection model provided with a semantic segmentation branch is trained to obtain a first target detection model; based on the semantic segmentation branch of the first target detection model, an inference analysis is performed on the unlabeled sample images in the original data set to obtain a semantic segmentation result; Evaluate and analyze the modifiable segmentation results in the semantic segmentation results to determine the usable segmentation results in the modifiable segmentation results; merge the target segmentation results in the semantic segmentation results and the usable segmentation results in the modifiable segmentation results into the annotated data set to obtain a new annotated data set; The first target detection model is iteratively trained based on the new labeled data set, and a target detection model is obtained when a preset model iteration termination condition is met.

[0005] According to a semi-supervised semantic segmentation network training method provided by the present invention, the evaluating and analyzing the modifiable segmentation results in the semantic segmentation results to determine the available segmentation results in the modifiable segmentation results specifically includes: Obtaining a revisable segmentation result from the semantic segmentation result; wherein the semantic segmentation result comprises a plurality of target segmentation images; and the revisable segmentation result comprises at least one target segmentation image from the plurality of target segmentation images; Based on the semantic segmentation result and the obtained target detection result, determining the inference result evaluation score of each target segmentation image in the modifiable segmentation result; wherein the target detection result is obtained by performing target detection on the remaining sample images in the original data set except the target sample image based on the backbone network branch of the first target detection model; Based on the inference result evaluation score of each target segmented image in the revisable segmentation result, an available segmentation result is determined from the revisable segmentation result.

[0006] According to a semi-supervised semantic segmentation network training method provided by the present invention, determining the inference result evaluation score of each target segmentation image in the modifiable segmentation result based on the semantic segmentation result and the obtained target detection result specifically includes: Determining a target detection coefficient for each target segmentation image in the correctable segmentation result based on the obtained target detection result; Determine a target segmentation score for each target segmentation image in the modifiable segmentation result based on the semantic segmentation result; wherein the semantic segmentation result includes the number of pixels in the target area delineated by the target segmentation; Based on the target detection coefficient and the target segmentation score, an inference result evaluation score of each target segmentation image in the modifiable segmentation result is determined.

[0007] According to a semi-supervised semantic segmentation network training method provided by the present invention, the step of obtaining a modifiable segmentation result in the semantic segmentation result specifically includes: Quantitatively sorting the semantic segmentation results according to the segmentation effect to obtain a quantitative sorting result; wherein the quantitative sorting result is obtained by sorting the multiple target segmentation images in sequence according to the quality of the segmentation effect; The semantic segmentation result within the target ratio range in the quantized sorting result is used as the modifiable segmentation result.

[0008] According to a semi-supervised semantic segmentation network training method provided by the present invention, the first target detection model is iteratively trained based on the new labeled data set, and the target detection model is obtained when a preset model iteration termination condition is met, specifically including: Training the first target detection model based on the new labeled data set to obtain a loss function value after each round of training of the first target detection model; When the difference between the loss function values ​​after two adjacent rounds of training is less than or equal to a preset target threshold, it is determined that the preset model iteration termination condition is met and a target detection model is obtained; wherein the loss function value is a measure of the error between the predicted value and the actual value of the first target detection model.

[0009] According to a semi-supervised semantic segmentation network training method provided by the present invention, the inference result evaluation score of each target segmentation image in the revisable segmentation result is used to determine the usable segmentation result from the revisable segmentation result, specifically including: The inference result evaluation score of each target segmentation image in the modifiable segmentation result is compared and analyzed with a preset score threshold. When the inference result evaluation score is greater than or equal to the score threshold, the target segmentation image corresponding to the inference result evaluation score is used as an available segmentation result.

[0010] The present invention also provides a semi-supervised semantic segmentation network training device, comprising: The annotated data acquisition unit is used to obtain an annotated data set; wherein the annotated data set is obtained by obtaining a target sample image from an original data set and performing polygonal semantic segmentation and annotation; A model training and reasoning analysis unit is used to train an initial detection model with a semantic segmentation branch based on the labeled data set to obtain a first target detection model; based on the semantic segmentation branch of the first target detection model, perform reasoning analysis on the unlabeled sample images in the original data set to obtain a semantic segmentation result; A segmentation result evaluation and screening unit, used to evaluate and analyze the modifiable segmentation results in the semantic segmentation results, determine the available segmentation results in the modifiable segmentation results; merge the target segmentation results in the semantic segmentation results and the available segmentation results in the modifiable segmentation results into the annotated data set to obtain a new annotated data set; The model iteration training analysis unit is used to iteratively train the first target detection model based on the new labeled data set, and obtain the target detection model when the preset model iteration termination condition is met.

[0011] According to a semi-supervised semantic segmentation network training device provided by the present invention, the segmentation result evaluation and screening unit is specifically used for: Obtaining a revisable segmentation result from the semantic segmentation result; wherein the semantic segmentation result comprises a plurality of target segmentation images; and the revisable segmentation result comprises at least one target segmentation image from the plurality of target segmentation images; Based on the semantic segmentation result and the obtained target detection result, determining the inference result evaluation score of each target segmentation image in the modifiable segmentation result; wherein the target detection result is obtained by performing target detection on the remaining sample images in the original data set except the target sample image based on the backbone network branch of the first target detection model; Based on the inference result evaluation score of each target segmented image in the revisable segmentation result, an available segmentation result is determined from the revisable segmentation result.

[0012] According to a semi-supervised semantic segmentation network training device provided by the present invention, the determining of the inference result evaluation score of each target segmentation image in the modifiable segmentation result based on the semantic segmentation result and the obtained target detection result specifically includes: Determining a target detection coefficient for each target segmentation image in the correctable segmentation result based on the obtained target detection result; Determine a target segmentation score for each target segmentation image in the modifiable segmentation result based on the semantic segmentation result; wherein the semantic segmentation result includes the number of pixels in the target area delineated by the target segmentation; Based on the target detection coefficient and the target segmentation score, an inference result evaluation score of each target segmentation image in the modifiable segmentation result is determined.

[0013] According to a semi-supervised semantic segmentation network training device provided by the present invention, the step of obtaining a modifiable segmentation result in the semantic segmentation result specifically includes: Quantitatively sorting the semantic segmentation results according to the segmentation effect to obtain a quantitative sorting result; wherein the quantitative sorting result is obtained by sorting the multiple target segmentation images in sequence according to the quality of the segmentation effect; The semantic segmentation result within the target ratio range in the quantized sorting result is used as the modifiable segmentation result.

[0014] According to a semi-supervised semantic segmentation network training device provided by the present invention, the model iterative training analysis unit is specifically used for: Training the first target detection model based on the new labeled data set to obtain a loss function value after each round of training of the first target detection model; When the difference between the loss function values ​​after two adjacent rounds of training is less than or equal to a preset target threshold, it is determined that the preset model iteration termination condition is met and a target detection model is obtained; wherein the loss function value is a measure of the error between the predicted value and the actual value of the first target detection model.

[0015] According to a semi-supervised semantic segmentation network training device provided by the present invention, the method of determining the usable segmentation result from the modifiable segmentation result based on the inference result evaluation score of each target segmentation image in the modifiable segmentation result specifically includes: The inference result evaluation score of each target segmentation image in the modifiable segmentation result is compared and analyzed with a preset score threshold. When the inference result evaluation score is greater than or equal to the score threshold, the target segmentation image corresponding to the inference result evaluation score is used as an available segmentation result.

[0016] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the steps of any of the above-described semi-supervised semantic segmentation network training methods are implemented.

[0017] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of any of the above-described semi-supervised semantic segmentation network training methods.

[0018] The present invention also provides a computer program product, comprising a computer program, which, when executed by a processor, implements the semi-supervised semantic segmentation network training method as described in any one of the above items.

[0019] The semi-supervised semantic segmentation network training method provided by the present invention trains an initial detection model provided with a semantic segmentation branch through a labeled data set to obtain a first target detection model; performs reasoning analysis on the remaining sample images in the original data set based on the semantic segmentation branch of the first target detection model to obtain a semantic segmentation result; evaluates and analyzes the modifiable segmentation results in the semantic segmentation results to determine the available segmentation results; incorporates the target segmentation results and the available segmentation results in the semantic segmentation results into the labeled data set, and uses the obtained new labeled data set to train the first target detection model, and obtains the target detection model when a preset model iteration termination condition is met, which can effectively reduce the labeling workload in the model training process, thereby improving the efficiency of model training and reducing labor costs. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0021] Figure 1It is a flow chart of the semi-supervised semantic segmentation network training method provided by the present invention.

[0022] Figure 2 It is a complete flow chart of the semi-supervised semantic segmentation network training method provided by the present invention.

[0023] Figure 3 It is a model schematic diagram of the semi-supervised semantic segmentation network training method provided by the present invention.

[0024] Figure 4 It is a structural schematic diagram of the semi-supervised semantic segmentation network training device provided by the present invention.

[0025] Figure 5 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0026] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0027] Combine the following Figure 1-Figure 5 The semi-supervised semantic segmentation network training method and device of the present invention are described, and its embodiments are described in detail.

[0028] The following is a detailed description of an embodiment of the semi-supervised semantic segmentation network training method according to the present invention. Figure 1 As shown, it is a flow chart of the semi-supervised semantic segmentation network training method provided by the present invention, and the specific implementation process includes the following steps: Step 101, obtaining a labeled data set; wherein the labeled data set is obtained by obtaining a target sample image from an original data set and performing polygonal semantic segmentation and annotation.

[0029] In the embodiment of the present invention, it is first necessary to construct an original data set containing a large number of sample images. In order to improve the annotation efficiency, some target sample images can be extracted from the original data set, and polygon semantic segmentation annotation can be performed on the target sample images to construct an annotated data set. Specifically, the original data set is first grouped into a target image set and a background image set, and each target of the target sample image in the target image set is annotated with a polygon and roughly annotated along the target boundary.

[0030] Step 102: Based on the labeled data set, an initial detection model with a semantic segmentation branch is trained to obtain a first target detection model; based on the semantic segmentation branch of the first target detection model, an inference analysis is performed on the unlabeled sample images in the original data set to obtain a semantic segmentation result.

[0031] In an embodiment of the present invention, the initial detection model may be a yoloV8 deep learning model to be trained. The initial detection model provided with a semantic segmentation branch can be obtained by adding a semantic segmentation branch to the yoloV8 deep learning model, and the initial detection model is trained by a labeled data set formed by the current labeled image, and two tasks of semantic segmentation and target detection are trained simultaneously. Specifically, the semantic segmentation branch of the initial detection model needs to be adjusted to the structure of the initial detection model. That is, the basic model is a yoloV8 deep learning model, and an additional semantic segmentation branch is added to realize the semantic segmentation function. That is, the semantic segmentation branch is a network branch for semantic segmentation of image data. The semantic segmentation branch connects the multi-level feature map of the backbone network branch of the initial detection model, and multi-level features are fused by upsampling step by step plus convolution and jump connection to form a semantic segmentation branch similar to the U-net form (wherein the left branch of U-net is the backbone network branch of the initial detection model, and the right branch is an independent semantic segmentation branch), so that the generated semantic segmentation probability map is 1 / 4 of the original image size. The semantic segmentation probability map and the semantic segmentation result (i.e., semantic segmentation annotation) are used as the input of the cross entropy loss function (CEL), thereby completing the training of the semantic segmentation branch.

[0032] Step 103: Evaluate and analyze the modifiable segmentation results in the semantic segmentation results to determine the available segmentation results in the modifiable segmentation results; merge the target segmentation results in the semantic segmentation results and the available segmentation results in the modifiable segmentation results into the annotated data set to obtain a new annotated data set.

[0033] Among them, the semantic segmentation results can be divided into target segmentation results (i.e., high-quality segmentation results, such as the corresponding target ratio range is the top 10%), correctable segmentation results (i.e., correctable segmentation results, such as the corresponding target ratio range is the top 10%-top 50%) and poor segmentation results (i.e., segmentation results with poor segmentation effect accuracy, such as the corresponding target ratio range is the bottom 50%) according to the accuracy of the segmentation effect.

[0034] In the process of evaluating and analyzing the revisable segmentation results in the semantic segmentation results and determining the usable segmentation results in the revisable segmentation results, the revisable segmentation results in the semantic segmentation results are first obtained; wherein the semantic segmentation results include multiple target segmentation images; and the revisable segmentation results include at least one target segmentation image among the multiple target segmentation images. Furthermore, based on the semantic segmentation results and the obtained target detection results, the inference result evaluation score of each target segmentation image in the revisable segmentation results is determined; wherein the target detection result is obtained by performing target detection on the remaining sample images in the original data set except the target sample image based on the backbone network branch of the first target detection model. Finally, based on the inference result evaluation score of each target segmentation image in the revisable segmentation results, the usable segmentation results are determined from the revisable segmentation results.

[0035] In the process of obtaining the correctable segmentation results in the semantic segmentation results, the semantic segmentation results can first be quantitatively sorted according to the segmentation effect to obtain a quantitative sorting result; the quantitative sorting result is obtained by sorting the multiple target segmentation images in order according to the quality of the segmentation effect. The semantic segmentation results within the target ratio range (such as the top 10%-top 50%) in the quantitative sorting result are used as correctable segmentation results.

[0036] In the process of determining the inference result evaluation score of each target segmentation image in the correctable segmentation result based on the semantic segmentation result and the obtained target detection result, firstly, the target detection coefficient of each target segmentation image in the correctable segmentation result is determined based on the obtained target detection result; then, the target segmentation score of each target segmentation image in the correctable segmentation result is determined based on the semantic segmentation result; wherein the semantic segmentation result includes the number of pixels of the target area delineated by the target segmentation. Finally, the inference result evaluation score of each target segmentation image in the correctable segmentation result is determined based on the target detection coefficient and the target segmentation score.

[0037] It should be noted that the inference result evaluation score of each image (i.e., the inference result evaluation score of each target segmentation image) is composed of two parts: the target detection coefficient and the target segmentation score. Represents the inference result score of the i-th target segmentation image:

[0038] in, The target detection coefficient indicates whether the target detection result of the i-th target segmentation image is correct. If it is correct, it takes 1, and if it is wrong, it takes 0; DiceCoefficient is the target segmentation score, where , represents the number of pixels in the target area delineated by target segmentation, The number of pixels representing the area where the true target is located.

[0039] In addition, in the process of determining the available segmentation result from the correctable segmentation result based on the inference result evaluation score of each target segmentation image in the correctable segmentation result, it is necessary to compare and analyze the inference result evaluation score of each target segmentation image in the correctable segmentation result with a preset score threshold. When the inference result evaluation score is greater than or equal to the score threshold, the target segmentation image corresponding to the inference result evaluation score is used as the available segmentation result.

[0040] Step 104: iteratively train the first target detection model based on the new labeled data set, and obtain a target detection model when a preset model iteration termination condition is met.

[0041] Specifically, the first target detection model may be trained based on the new annotated data set to obtain the loss function value of the first target detection model after each round of training. When the difference between the loss function values ​​after two adjacent rounds of training is less than or equal to a preset target threshold, it is determined that the preset model iteration termination condition is met, and the target detection model is obtained. The loss function value is a measure of the error between the predicted value and the actual value of the first target detection model.

[0042] Furthermore, the present invention also provides a semi-supervised semantic segmentation network target detection method, which is applied to the target detection model obtained, including: obtaining a target image to be detected, inputting the target image to be detected into the target detection model for semantic segmentation reasoning, and obtaining a semantic segmentation result output by the target detection model; wherein the target detection model is obtained by training a semi-supervised semantic segmentation network based on a sample image and a semantic segmentation result label corresponding to the sample image.

[0043] In an embodiment of the present invention, in order to coordinate the relationship between data volume, annotation accuracy and feature extraction effect, the present invention proposes a semi-supervised semantic segmentation network training and target detection method. First, semantic segmentation annotation is performed on some target sample images of the original data set, and then a semantic segmentation branch is added to the YoloV8 deep learning model, and two tasks of semantic segmentation and target detection are trained at the same time; then the first target detection model obtained after training is used to perform reasoning analysis on the unlabeled sample images in the original data set, generate annotation data and grade the annotation data according to availability; then the modifiable annotations (i.e., the corrected segmentation results) are reviewed by a variety of post-processing methods; finally, the available annotation data (i.e., the available segmentation results) are added to the annotation data set to retrain the first target detection model; the above training, reasoning, grading, and review processes are continuously repeated to enrich the semantic segmentation annotation data set until the model reaches the expected result. The semi-supervised semantic segmentation target detection model of the present invention enables the deep learning target detection model to be implemented with only a few thousand image samples to complete the preliminary training, and then it can be supplemented while testing at runtime. At the same time, the present invention does not require manual annotation of each image when expanding the data set of deep learning products, and the workload and professional level are greatly reduced, which is conducive to the promotion and application of new products.

[0044] like Figure 2 As shown, the specific steps include the following steps S1-S5: Step S1: construct an original data set P, extract a part of the sample image to obtain a target sample image, perform polygonal semantic segmentation and annotation on the target sample image, and obtain an annotated data set P0; that is, the original data set P includes the annotated data set P0 and an unlabeled data set.

[0045] For example, in the specific implementation process: sample images are collected for the train running process in different time periods, totaling more than 12,000 original data sets P with trains, and 5,000 target sample images in the original data set P are selected for rough polygon annotation to generate the labeled data set P0, with the label "train", and the target detection model training is completed using the labeled data set P0. Step S1 includes constructing the original data set P and the labeled data set P0. After sorting out the images collected by more than 40 cameras on the railway line for 7 days, 5-10 sample images are extracted at the same time interval when each train passes, and each camera collects 3-5 scene sample images per hour, totaling more than 25,000 sample images to form the original data set P, among which 5,000 target sample images are randomly selected from more than 12,000 sample images with trains passing and rough polygon annotation is performed to generate the description data set P0. The present invention replaces the annotation work of the newly expanded data set with the traditional image algorithm (i.e., the preset evaluation criteria) of parameter quantization and threshold screening, so that the image sample annotation standard is unified and controllable, which is more conducive to the upgrade and operation of the model.

[0046] Step S2: Add a semantic segmentation branch to the yoloV8 deep learning model to obtain a semi-supervised semantic segmentation detection model (i.e., the initial detection model). Simultaneously train the semantic segmentation and target detection tasks, i.e., train the networks of the semantic segmentation branch and the detection branch. Train the new model (i.e., the first target detection model) using the currently labeled target sample images. Figure 3 As shown, step S2 specifically includes the following processes: Establishing the semantic segmentation branch of the initial detection model requires adjusting the structure of the initial detection model. The basic model is the yoloV8 deep learning model, and an additional semantic segmentation branch is added to realize the semantic segmentation function. Connect the multi-level feature map of the backbone network branch (i.e., the backbone network) of the initial detection model, and fuse the multi-level features through step-by-step upsampling, convolution, and jump connections to form a semantic segmentation branch similar to the U-net form (where the left branch of U-net is the backbone network branch of the initial detection model, and the right branch is an independent semantic segmentation branch), so that the generated semantic segmentation probability map is 1 / 4 of the size of the original image; use the semantic segmentation probability map and the semantic segmentation annotation as the input of the BCE loss function to complete the training of the semantic segmentation branch and obtain the semantic segmentation prediction. Among them, P1-P5 are several different convolutional layers contained in the backbone network. T2-T5 are several different convolutional layers contained in the semantic segmentation branch. The yoloV8 deep learning model mainly consists of three parts: Backbone (i.e., the backbone network branch), Neck, and Head. The network model is shown below Figure 3 As shown in the figure, Backbone is mainly responsible for feature extraction of input images. Neck is responsible for multi-scale feature fusion of feature maps and passing these features to the prediction layer. Head performs the final regression prediction.

[0047] For example, in the specific implementation process: the basic model is the yoloV8 model, and an additional semantic segmentation branch is added to realize the semantic segmentation function. The multi-level feature graphs of the backbone network of the connection detection model are connected, and the multi-level features are integrated through step-by-step upsampling, convolution and jump connection to form a semantic segmentation branch similar to U-net (where the left branch of U-net is the backbone network of the detection model, and the right branch is an independent semantic segmentation branch). The initial detection model is trained together through target detection annotation and semantic segmentation annotation. Specifically, the selected annotation dataset P0 is used as the training set to train the new model (the first target detection model). First, the network parameters are initialized, and the training learning rate is set to , momentum 0.9, training batch size set to 16, training iteration number is 200. Then, network forward propagation is performed: the training set images are randomly cropped into 512×512 size images after data augmentation (rotation, scaling, brightness adjustment), normalized and input into network forward propagation, the detection branch outputs the detection box prediction value, the semantic segmentation branch outputs the semantic segmentation prediction value, and the two prediction values ​​are respectively calculated with the corresponding label values. Error back propagation, update network parameters, after each round of training, the model calculates the overall loss function on the validation set, training iteration 200 rounds, take the target detection model with the smallest loss function on the validation set as the optimal neural network (i.e., target detection model).

[0048] Step S3: Use the first target detection model to perform reasoning analysis on the images in the unlabeled data set, generate labeled data, and filter the data. The labeled data is divided into three categories: ① high-quality labeled data (i.e., data set ①); ② correctable labeled data (i.e., data set ②); ③ ​​poorly labeled data (i.e., data set ③). Among them, step S3 specifically includes the following process: reasoning the unlabeled sample images in the current original image set to obtain semantic segmentation results, and manually evaluating the semantic segmentation effect, and dividing all semantic segmentation results into the following three categories: ① high-quality segmentation results (i.e., target segmentation results, top 10%); ② correctable segmentation results (top 10% - top 50%); ③ poor segmentation results (bottom 50%).

[0049] For example, in the specific implementation process: first, the 20,000 unlabeled sample images in the current original image set are inferred; then, according to the inference results, the target detection label and the target segmentation label are output respectively, and the target segmentation probability map is output at the same time. The inference results are quantified by combining target detection with image target segmentation, and all detection and segmentation results are divided into the following three categories: ① High-quality segmentation results (top 10%) ② Correctable segmentation results (top 10% - top 50%) ③ Poor segmentation results (bottom 50%). The inference result evaluation score of each image is composed of two parts: the target detection coefficient and the target segmentation score. Pi represents the inference result score of the i-th image:

[0050] in The target detection coefficient indicates whether the target detection result of the i-th target segmentation image is correct. If it is correct, it takes 1, and if it is wrong, it takes 0; DiceCoefficient is the target segmentation score, where , represents the number of pixels in the target area delineated by target segmentation, The number of pixels representing the area where the true target is located.

[0051] Step S4: Batch process the ② type of annotated data (i.e., corrected annotated data) through a variety of post-processing methods (such as morphological methods, GrabCut, etc.), and divide it into data set ④ usable annotations and data set ⑤ unusable annotations. Among them, step S4 specifically includes the following process: Batch process ② correctable segmentation results through a variety of post-processing methods (morphological methods, GrabCut, etc.) to obtain new segmentation results. Through manual evaluation of the semantic segmentation effect, it is divided into ④ usable segmentation results and ⑤ unusable segmentation results.

[0052] For example, in the specific implementation process: batch processing can correct the segmentation data set ②, and merge them into data sets ④ and ⑤ respectively. Through a variety of post-processing methods (such as morphological methods, GrabCut, etc.), batch processing ② can correct the segmentation results and obtain new segmentation results. Through the judgment criteria and segmentation results (image p1 contains a train target. If the train is recognized correctly, the value of the intersection area of ​​the segmented area and the train area is calculated as the target segmentation score. The first 50% of the segmentation results of the target segmentation score are merged into ④, and the last 50% of the segmentation results are merged into ⑤. If the recognition is wrong, it is directly merged into data set ⑤; image p1 does not contain a train target. If the inference model does not recognize the train, the image is directly merged into ④. If the train is recognized, then the recognition is wrong and it is directly merged into data set ⑤), the processing of the correctable segmentation data set ② is completed. The judgment logic is shown in the attachment. Figure 2 .

[0053] Step S5: Use ①+④ datasets and labeled target sample images as new labeled datasets, repeat steps S2-S5, as semantic annotations are improved, the target detection effect will also increase, if the target detection effect reaches the expected result, this step will not be repeated, and the target detection model is obtained. Wherein, step S5 specifically includes the following process: use ① high-quality segmentation result (i.e., target segmentation result) obtained in S3 and ④ available segmentation result obtained in S4 as annotations to add to the labeled dataset, repeat steps S2-S5, as semantic annotation data is improved, the target detection effect will also increase, if the target detection effect reaches the expected result, this step will not be repeated, and the target detection model is obtained, if the target detection effect does not reach the expected result, then continue to repeat steps S2-S5 to further label the dataset.

[0054] For example, in the specific implementation process: use the ① high-quality segmentation result obtained in step S3 and the ④ available segmentation result obtained in step S4 as annotations to add to the annotated dataset to obtain a new annotated dataset, repeat steps S2-S5, and as the semantic annotation is improved, the target detection effect will also increase. If the target detection effect meets expectations, this step will not be repeated to obtain a target detection model. If the target detection effect does not meet expectations, continue to repeat steps S2-S5 to further annotate the dataset. After several rounds of repetitions, a training set of more than 23,500 images was finally formed, including more than 11,000 train images and more than 12,500 non-train images. After completing 200 rounds of model training, the model's Loss no longer decreased significantly, and the difference between the two rounds of Loss was less than , as the iteration termination condition. The training termination condition of the present invention, the training error, the training number and the expected relationship can be flexibly adjusted according to the needs to adapt to different accuracy requirements.

[0055] The network loss of the model of the present invention has three parts: classification loss, positioning loss, and center-ness loss. The corresponding algorithm can be shown as follows:

[0056] in, Represents the score of each category predicted at the feature map (x, y) point; Indicates the true category label corresponding to the feature map (x, y) point; When the feature map (x, y) point matches a positive sample, it is 1, otherwise it is 0; Represents the predicted target bounding box information at the feature map (x, y) point; Represents the real target bounding box information corresponding to the feature map (x, y) point; Indicates the center-ness of the prediction at the feature map (x, y) point; Indicates the true center-ness corresponding to the point (x, y) in the feature map; is the number of positive samples; It is in the form of focal loss, i.e. classification loss function; is the IOU los of UnitBox, i.e. the positioning loss function; is the corresponding center-ness loss function; The calculation formula , given the regression target=( l *, t *, r *, b *),in l *, t *,r *, b * is the regression parameter.

[0057]

[0058] As shown in the table above, in one embodiment, after 10 days of operation testing of 40 network cameras on site and 2,400 field test cases, the average detection accuracy of the semi-supervised semantic segmentation target detection model was 97.4%, and the average detection accuracy of the current background modeling method was 90.3%. The model's missed detection and false detection are greatly improved compared to the current background modeling method.

[0059] The semi-supervised semantic segmentation network training method provided by the present invention trains an initial detection model provided with a semantic segmentation branch through a labeled data set to obtain a first target detection model; performs reasoning analysis on the remaining sample images in the original data set based on the semantic segmentation branch of the first target detection model to obtain a semantic segmentation result; evaluates and analyzes the modifiable segmentation results in the semantic segmentation results to determine the available segmentation results; incorporates the target segmentation results and the available segmentation results in the semantic segmentation results into the labeled data set, and uses the obtained new labeled data set to train the first target detection model, and obtains the target detection model when a preset model iteration termination condition is met, which can effectively reduce the labeling workload in the model training process, thereby improving the efficiency of model training and reducing labor costs.

[0060] The following is a description of the semi-supervised semantic segmentation network training device provided by the present invention. The semi-supervised semantic segmentation network training device described below and the semi-supervised semantic segmentation network training method described above can be referred to each other. Figure 4 As shown, it is a schematic diagram of the structure of the semi-supervised semantic segmentation network training device provided by the present invention. The semi-supervised semantic segmentation network training device of the present invention specifically includes the following parts: The annotated data obtaining unit 401 is used to obtain an annotated data set; wherein the annotated data set is obtained by obtaining a target sample image from an original data set and performing polygonal semantic segmentation and annotation.

[0061] The model training and reasoning analysis unit 402 is used to train an initial detection model with a semantic segmentation branch based on the labeled data set to obtain a first target detection model; based on the semantic segmentation branch of the first target detection model, reasoning analysis is performed on the unlabeled sample images in the original data set to obtain a semantic segmentation result.

[0062] The segmentation result evaluation and screening unit 403 is used to evaluate and analyze the modifiable segmentation results in the semantic segmentation results, determine the available segmentation results in the modifiable segmentation results; merge the target segmentation results in the semantic segmentation results and the available segmentation results in the modifiable segmentation results into the annotated data set to obtain a new annotated data set.

[0063] The model iteration training analysis unit 404 is used to iteratively train the first target detection model based on the new labeled data set, and obtain the target detection model when the preset model iteration termination condition is met.

[0064] According to a semi-supervised semantic segmentation network training device provided by the present invention, the segmentation result evaluation and screening unit is specifically used for: Obtaining a revisable segmentation result from the semantic segmentation result; wherein the semantic segmentation result comprises a plurality of target segmentation images; and the revisable segmentation result comprises at least one target segmentation image from the plurality of target segmentation images; Based on the semantic segmentation result and the obtained target detection result, determining the inference result evaluation score of each target segmentation image in the modifiable segmentation result; wherein the target detection result is obtained by performing target detection on the remaining sample images in the original data set except the target sample image based on the backbone network branch of the first target detection model; Based on the inference result evaluation score of each target segmented image in the revisable segmentation result, an available segmentation result is determined from the revisable segmentation result.

[0065] According to a semi-supervised semantic segmentation network training device provided by the present invention, the determining of the inference result evaluation score of each target segmentation image in the modifiable segmentation result based on the semantic segmentation result and the obtained target detection result specifically includes: Determining a target detection coefficient for each target segmentation image in the correctable segmentation result based on the obtained target detection result; Determine a target segmentation score for each target segmentation image in the modifiable segmentation result based on the semantic segmentation result; wherein the semantic segmentation result includes the number of pixels in the target area delineated by the target segmentation; Based on the target detection coefficient and the target segmentation score, an inference result evaluation score of each target segmentation image in the modifiable segmentation result is determined.

[0066] According to a semi-supervised semantic segmentation network training device provided by the present invention, the step of obtaining a modifiable segmentation result in the semantic segmentation result specifically includes: Quantitatively sorting the semantic segmentation results according to the segmentation effect to obtain a quantitative sorting result; wherein the quantitative sorting result is obtained by sorting the multiple target segmentation images in sequence according to the quality of the segmentation effect; The semantic segmentation result within the target ratio range in the quantized sorting result is used as the modifiable segmentation result.

[0067] According to a semi-supervised semantic segmentation network training device provided by the present invention, the model iterative training analysis unit is specifically used for: Training the first target detection model based on the new labeled data set to obtain a loss function value after each round of training of the first target detection model; When the difference between the loss function values ​​after two adjacent rounds of training is less than or equal to a preset target threshold, it is determined that the preset model iteration termination condition is met and a target detection model is obtained; wherein the loss function value is a measure of the error between the predicted value and the actual value of the first target detection model.

[0068] According to a semi-supervised semantic segmentation network training device provided by the present invention, the method of determining the usable segmentation result from the modifiable segmentation result based on the inference result evaluation score of each target segmentation image in the modifiable segmentation result specifically includes: The inference result evaluation score of each target segmentation image in the modifiable segmentation result is compared and analyzed with a preset score threshold. When the inference result evaluation score is greater than or equal to the score threshold, the target segmentation image corresponding to the inference result evaluation score is used as an available segmentation result.

[0069] The semi-supervised semantic segmentation network training device provided by the present invention trains an initial detection model provided with a semantic segmentation branch through a labeled data set to obtain a first target detection model; performs reasoning analysis on the remaining sample images in the original data set based on the semantic segmentation branch of the first target detection model to obtain a semantic segmentation result; evaluates and analyzes the modifiable segmentation results in the semantic segmentation results to determine the available segmentation results; incorporates the target segmentation results and the available segmentation results in the semantic segmentation results into the labeled data set, and trains the first target detection model using the obtained new labeled data set, and obtains the target detection model when a preset model iteration termination condition is met, which can effectively reduce the labeling workload in the model training process, thereby improving the efficiency of model training and reducing labor costs.

[0070] Figure 5 An example of a physical structure diagram of an electronic device is shown in FIG. Figure 5As shown, the electronic device (ie, electronic device) may include: a processor (processor) 501, a communication interface (Communications Interface) 504, a memory (memory) 502 and a communication bus 503, wherein the processor 501, the communication interface 504, and the memory 502 communicate with each other via the communication bus 503. The processor 501 can call the logic instructions in the memory 502 to execute a semi-supervised semantic segmentation network training method, which includes: obtaining a labeled data set; wherein the labeled data set is obtained by obtaining a target sample image from an original data set and performing polygonal semantic segmentation annotation; based on the labeled data set, an initial detection model with a semantic segmentation branch is trained to obtain a first target detection model; based on the semantic segmentation branch of the first target detection model, an inference analysis is performed on the unlabeled sample images in the original data set to obtain a semantic segmentation result; an evaluation and analysis is performed on the correctable segmentation results in the semantic segmentation results to determine the available segmentation results in the correctable segmentation results; the target segmentation results in the semantic segmentation results and the available segmentation results in the correctable segmentation results are merged into the labeled data set to obtain a new labeled data set; based on the new labeled data set, the first target detection model is iteratively trained to obtain a target detection model when a preset model iteration termination condition is met.

[0071] In addition, the logic instructions in the above-mentioned memory 502 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on this understanding, the technical solution of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0072] On the other hand, the present application also provides a computer program product, which includes a computer program, which can be stored on a computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the semi-supervised semantic segmentation network training method provided by the above methods, which method includes: obtaining a labeled data set; wherein the labeled data set is obtained by obtaining a target sample image from an original data set for polygonal semantic segmentation annotation; based on the labeled data set, an initial detection model with a semantic segmentation branch is trained to obtain a first target detection model; based on the semantic segmentation branch of the first target detection model, an inference analysis is performed on the unlabeled sample images in the original data set to obtain a semantic segmentation result; an evaluation and analysis is performed on the correctable segmentation result in the semantic segmentation result to determine the available segmentation result in the correctable segmentation result; the target segmentation result in the semantic segmentation result and the available segmentation result in the correctable segmentation result are merged into the labeled data set to obtain a new labeled data set; based on the new labeled data set, the first target detection model is iteratively trained to obtain a target detection model when a preset model iteration termination condition is met.

[0073] On the other hand, the present application also provides a computer-readable storage medium, which includes a stored program, wherein when the program is run, the semi-supervised semantic segmentation network training method provided by the above-mentioned methods is executed, and the method includes: obtaining a labeled data set; wherein the labeled data set is obtained by obtaining a target sample image from an original data set and performing polygonal semantic segmentation annotation; based on the labeled data set, an initial detection model with a semantic segmentation branch is trained to obtain a first target detection model; based on the semantic segmentation branch of the first target detection model, an inference analysis is performed on the unlabeled sample images in the original data set to obtain a semantic segmentation result; an evaluation and analysis is performed on the correctable segmentation results in the semantic segmentation results to determine the available segmentation results in the correctable segmentation results; the target segmentation results in the semantic segmentation results and the available segmentation results in the correctable segmentation results are merged into the labeled data set to obtain a new labeled data set; based on the new labeled data set, the first target detection model is iteratively trained to obtain a target detection model when a preset model iteration termination condition is met.

[0074] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0075] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0076] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A semi-supervised semantic segmentation network training method, characterized in that: include: Obtaining a labeled data set; wherein the labeled data set is obtained by obtaining a target sample image from an original data set and performing polygonal semantic segmentation and annotation; Based on the labeled data set, an initial detection model provided with a semantic segmentation branch is trained to obtain a first target detection model; based on the semantic segmentation branch of the first target detection model, an inference analysis is performed on the unlabeled sample images in the original data set to obtain a semantic segmentation result; Evaluate and analyze the modifiable segmentation results in the semantic segmentation results to determine the usable segmentation results in the modifiable segmentation results; merge the target segmentation results in the semantic segmentation results and the usable segmentation results in the modifiable segmentation results into the annotated data set to obtain a new annotated data set; The first target detection model is iteratively trained based on the new labeled data set, and a target detection model is obtained when a preset model iteration termination condition is met.

2. The semi-supervised semantic segmentation network training method according to claim 1, characterized in that: The step of evaluating and analyzing the modifiable segmentation results in the semantic segmentation results to determine the available segmentation results in the modifiable segmentation results specifically includes: Obtaining a revisable segmentation result from the semantic segmentation result; wherein the semantic segmentation result comprises a plurality of target segmentation images; and the revisable segmentation result comprises at least one target segmentation image from the plurality of target segmentation images; Based on the semantic segmentation result and the obtained target detection result, determining the inference result evaluation score of each target segmentation image in the modifiable segmentation result; wherein the target detection result is obtained by performing target detection on the remaining sample images in the original data set except the target sample image based on the backbone network branch of the first target detection model; Based on the inference result evaluation score of each target segmented image in the revisable segmentation result, an available segmentation result is determined from the revisable segmentation result.

3. The semi-supervised semantic segmentation network training method according to claim 2, characterized in that: The step of determining the inference result evaluation score of each target segmentation image in the modifiable segmentation result based on the semantic segmentation result and the obtained target detection result specifically includes: Determining a target detection coefficient for each target segmentation image in the correctable segmentation result based on the obtained target detection result; Determine a target segmentation score for each target segmentation image in the modifiable segmentation result based on the semantic segmentation result; wherein the semantic segmentation result includes the number of pixels in the target area delineated by the target segmentation; Based on the target detection coefficient and the target segmentation score, an inference result evaluation score of each target segmentation image in the modifiable segmentation result is determined.

4. The semi-supervised semantic segmentation network training method according to claim 2, characterized in that: The obtaining of the modifiable segmentation result in the semantic segmentation result specifically includes: Quantitatively sorting the semantic segmentation results according to the segmentation effect to obtain a quantitative sorting result; wherein the quantitative sorting result is obtained by sorting the multiple target segmentation images in sequence according to the quality of the segmentation effect; The semantic segmentation result within the target ratio range in the quantized sorting result is used as the modifiable segmentation result.

5. The semi-supervised semantic segmentation network training method according to claim 1, characterized in that: The iterative training of the first target detection model based on the new labeled data set, and obtaining the target detection model when a preset model iteration termination condition is met, specifically includes: Training the first target detection model based on the new labeled data set to obtain a loss function value after each round of training of the first target detection model; When the difference between the loss function values ​​after two adjacent rounds of training is less than or equal to a preset target threshold, it is determined that the preset model iteration termination condition is met and a target detection model is obtained; wherein the loss function value is a measure of the error between the predicted value and the actual value of the first target detection model.

6. The semi-supervised semantic segmentation network training method according to claim 2, characterized in that: The step of determining the usable segmentation result from the modifiable segmentation result based on the inference result evaluation score of each target segmentation image in the modifiable segmentation result specifically includes: The inference result evaluation score of each target segmentation image in the modifiable segmentation result is compared and analyzed with a preset score threshold. When the inference result evaluation score is greater than or equal to the score threshold, the target segmentation image corresponding to the inference result evaluation score is used as an available segmentation result.

7. A semi-supervised semantic segmentation network training device, characterized in that: include: The annotated data acquisition unit is used to obtain an annotated data set; wherein the annotated data set is obtained by obtaining a target sample image from an original data set and performing polygonal semantic segmentation and annotation; A model training and reasoning analysis unit is used to train an initial detection model with a semantic segmentation branch based on the labeled data set to obtain a first target detection model; based on the semantic segmentation branch of the first target detection model, perform reasoning analysis on the unlabeled sample images in the original data set to obtain a semantic segmentation result; A segmentation result evaluation and screening unit, used to evaluate and analyze the modifiable segmentation results in the semantic segmentation results, determine the available segmentation results in the modifiable segmentation results; merge the target segmentation results in the semantic segmentation results and the available segmentation results in the modifiable segmentation results into the annotated data set to obtain a new annotated data set; The model iteration training analysis unit is used to iteratively train the first target detection model based on the new labeled data set, and obtain the target detection model when the preset model iteration termination condition is met.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the steps of the semi-supervised semantic segmentation network training method as described in any one of claims 1 to 6 are implemented.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the semi-supervised semantic segmentation network training method as described in any one of claims 1 to 6 are implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the semi-supervised semantic segmentation network training method as described in any one of claims 1 to 6 is implemented.

Images