Semantic segmentation model training method and image detection method based on comparative learning

Abstract

The invention discloses a semantic segmentation model training method and an image detection method based on comparative learning, which are applied to the field of image detection, and comprise the following steps: extracting sample features of a training sample by using a feature extraction module; inputting the sample features into a semantic segmentation module and a comparative learning module to obtain a mask pattern and a feature pattern group; according to the label graph and the mask graph corresponding to the training sample, semantic segmentation loss is obtained through calculation; calculating comparison loss according to the feature data in the feature graph group and the feature queue; updating parameters of the semantic segmentation model according to the semantic segmentation loss and the comparison loss, and updating feature data in the feature queue according to the feature graph group and the feature data storage moment; obtaining a trained semantic segmentation model until a preset condition is met; the semantic segmentation model comprises a feature extraction module and a semantic segmentation module. According to the method, the occupation of memory resources can be reduced, the contrast loss of each iteration is effectively and accurately calculated, and the performance of the model is improved.

Description

Technical Field

[0001] The present invention relates to the field of image detection, and in particular to a semantic segmentation model training method and an image detection method based on contrastive learning. Background Art

[0002] Semantic segmentation is a key technology in the field of computer vision. It recognizes and segments objects in an image by classifying each pixel in the image. Contrastive learning is an unsupervised learning method that learns features by learning the similarities and differences between samples, thereby improving the generalization ability of the model without the need for a large amount of manually labeled data. In the semantic segmentation task, contrastive learning can enable the model to learn more discriminative feature representations, thereby improving the performance of semantic segmentation. However, in the training of semantic segmentation models based on contrastive learning, a large number of sample features are stored for calculating contrast loss.

[0003] Therefore, how to improve the performance of the semantic segmentation model and reduce memory resource usage is an urgent problem that needs to be solved. Summary of the invention

[0004] In view of this, the purpose of the present invention is to provide a semantic segmentation model training method and an image detection method based on contrastive learning, which solves the problems of poor performance and large occupation of memory resources of the semantic segmentation model in the prior art.

[0005] In order to solve the above technical problems, the present invention provides a semantic segmentation model training method based on contrastive learning, comprising:

[0006] Extracting sample features of training samples using a feature extraction module;

[0007] Inputting the sample features into a semantic segmentation module and a contrastive learning module to obtain a mask map and a feature map group respectively;

[0008] Calculating semantic segmentation loss according to the label image and the mask image corresponding to the training sample;

[0009] Calculating the contrast loss based on the feature map group and the feature data in the feature queue;

[0010] The parameters of the semantic segmentation model are dynamically updated according to the semantic segmentation loss and the contrast loss, and the feature data in the feature queue are dynamically updated according to the feature map group and the feature data storage time; until the preset conditions are met, the training ends and a trained semantic segmentation model is obtained; the semantic segmentation model includes the feature extraction module and the semantic segmentation module.

[0011] Optionally, the sample features are input into a semantic segmentation module to obtain a mask map, including:

[0012] Inputting the sample features into an upsampling layer to obtain a first output result;

[0013] The first output result is input into a softmax classification layer to obtain the mask image; the pixel values ​​in the mask image represent the pixel categories.

[0014] Optionally, the sample features are input into a contrastive learning module to obtain a feature map group, including:

[0015] Inputting the sample features into a convolutional layer to obtain a second output result;

[0016] The second output result is input into the upsampling layer and the normalization layer to obtain the feature map group; the number of channels of the feature map group is the number of features.

[0017] Optionally, dynamically updating the feature data in the feature queue according to the feature graph group and the feature data storage time includes:

[0018] When the feature queue is full, the feature data of the feature graph group calculated at the current iteration number is stored in the feature queue, and the feature data at the earliest storage time in the feature queue is deleted.

[0019] Optionally, dynamically updating the feature data in the feature queue according to the feature graph group and the feature data storage time includes:

[0020] When there are multiple objects of the same category in the feature map group, each region of the same category is taken out along the channel direction; each region has the same number of channels;

[0021] Based on each region, average is calculated in the height and width directions to obtain each feature data; the width and height of each feature data are 1 respectively, and the number of channels remains unchanged;

[0022] Based on the number of each feature data, part of the data of each feature data is respectively taken for splicing to obtain spliced ​​feature data; the width and height of the spliced ​​feature data are 1 respectively, and the number of channels remains unchanged;

[0023] The feature data in the feature queue is dynamically updated according to the spliced ​​feature data and the feature data storage time.

[0024] The present invention also provides an image detection method, comprising:

[0025] Using the above-mentioned semantic segmentation model training method based on contrastive learning to obtain a trained semantic segmentation model;

[0026] The trained semantic segmentation model is used to detect the image to be detected to obtain a detection result.

[0027] The present invention also provides a semantic segmentation model training device based on contrastive learning, comprising:

[0028] A sample feature acquisition module is used to extract sample features of training samples using a feature extraction module;

[0029] A processing module, used for inputting the sample features into a semantic segmentation module and a contrastive learning module to obtain a mask map and a feature map group respectively;

[0030] A first calculation module, used to calculate the semantic segmentation loss according to the label map and the mask map corresponding to the training sample;

[0031] A second calculation module, used for calculating the contrast loss according to the feature map group and the feature data in the feature queue;

[0032] A training update module is used to dynamically update the parameters of the semantic segmentation model according to the semantic segmentation loss and the contrast loss, and dynamically update the feature data in the feature queue according to the feature map group and the feature data storage time; until the preset conditions are met, the training ends and a trained semantic segmentation model is obtained; the semantic segmentation model includes the feature extraction module and the semantic segmentation module.

[0033] The present invention also provides an image detection device, comprising

[0034] A model acquisition module, used to obtain a trained semantic segmentation model using the above-mentioned semantic segmentation model training method based on contrastive learning;

[0035] The detection module is used to detect the image to be detected using the trained semantic segmentation model to obtain a detection result.

[0036] The present invention also provides an electronic device, comprising:

[0037] Memory for storing computer programs;

[0038] A processor is used to implement the above method when executing the computer program.

[0039] The present invention also provides a computer-readable storage medium, in which computer-executable instructions are stored. When the computer-executable instructions are loaded and executed by a processor, the above method is implemented.

[0040] It can be seen that the present invention uses a feature extraction module to extract sample features of training samples; the sample features are input into a semantic segmentation module and a contrastive learning module to obtain a mask map and a feature map group respectively; the semantic segmentation loss is calculated according to the label map corresponding to the training sample and the mask map; the contrast loss is calculated according to the feature map group and the feature data in the feature queue; the parameters of the semantic segmentation model are dynamically updated according to the semantic segmentation loss and the contrast loss, and the feature data in the feature queue is dynamically updated according to the feature map group and the feature data storage time; until the preset conditions are met, the training ends and a trained semantic segmentation model is obtained; the semantic segmentation model includes a feature extraction module and a semantic segmentation module. When the semantic segmentation model is trained, the present invention uses contrastive learning to assist in its training, and during the training process, the feature data in the feature queue is dynamically updated using the method of replacing the old with the new, avoiding the problem of reading all feature data at one time, or the problem of poor model training effect caused by insufficient feature data; while reducing the memory resources occupied, the contrast loss of each iteration can be effectively and accurately calculated through the feature queue replaced by the old and the new, thereby improving the performance of the model.

[0041] In addition, the present invention also provides an image detection method, a semantic segmentation model training device based on contrastive learning, an image detection device, an electronic device and a computer-readable storage medium, which also have the above-mentioned beneficial effects. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0043] Figure 1 A flowchart of a semantic segmentation model training method based on contrastive learning provided by an embodiment of the present invention;

[0044] Figure 2 An example diagram of a semantic segmentation model training architecture based on contrastive learning provided in an embodiment of the present invention;

[0045] Figure 3 An example diagram of a feature data splicing method provided by an embodiment of the present invention;

[0046] Figure 4 A flowchart of an image detection method provided by an embodiment of the present invention;

[0047] Figure 5 A schematic diagram of the structure of a semantic segmentation model training device based on contrastive learning provided by an embodiment of the present invention;

[0048] Figure 6 A schematic diagram of the structure of an image detection device provided by an embodiment of the present invention;

[0049] Figure 7 A schematic structural diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0050] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0051] Semantic segmentation is a key technology in the field of computer vision. It recognizes and segments objects in an image by classifying each pixel in the image. Contrastive learning, as an unsupervised learning technology, indirectly learns the relationship between samples by comparing the similarities or differences between different samples, thereby improving the generalization ability of the model without the need for a large amount of manually labeled data. Therefore, contrastive learning can be used for auxiliary training of semantic segmentation models. For example, cross-image pixel contrast technology uses the semantic similarity of pixels between different images to strengthen the embedding similarity between pixels of the same semantic category, while increasing the embedding distance between pixels of different categories. The introduction of this global context information provides a richer feature representation for semantic segmentation tasks. This technology can effectively improve the performance of semantic segmentation, especially in the recognition of small objects and complex structures.

[0052] In unsupervised contrastive learning, "Memory Bank" is a key mechanism used to store feature representations of samples encountered during training for use in calculating contrastive loss. It can also be called a feature queue. Currently, all image features in the training dataset are generally stored in memory for calculation. This method has very high hardware requirements and requires a large enough memory space to store data. At the same time, a large amount of data is used for comparison during model training, which will seriously slow down the training speed of the model. Alternatively, only the data of the most recent few batches (mini-batches) can be retained. Since the data of a batch is limited under normal circumstances, this will result in insufficient sample features and will not effectively improve the performance of the segmentation model.

[0053] In view of the problems existing in the above methods, the present invention proposes a semantic segmentation model training method based on contrastive learning, which avoids the problem of reading all image feature data at one time, or the problem of poor model training effect caused by insufficient feature data. Figure 1 , Figure 1 A flowchart of a semantic segmentation model training method based on contrastive learning provided by an embodiment of the present invention. The method may include:

[0054] S101: Extracting sample features of training samples using a feature extraction module.

[0055] The execution subject of this embodiment is a terminal. This embodiment does not limit the type of terminal, as long as it can complete the operation of the semantic segmentation model training method based on contrastive learning. This embodiment does not limit the feature extraction module. As long as it meets the deep learning feature extraction method. Among them, the deep learning feature extraction can be a VGGNet (Visual Geometry Group Network) network, a ResNet (Residual Network) network, etc.

[0056] S102: Input the sample features into the semantic segmentation module and the contrastive learning module to obtain a mask map and a feature map group respectively.

[0057] In this step, the sample features extracted in step S101 are respectively input into the semantic segmentation module and the contrastive learning module to obtain a mask map and a feature map group respectively. This embodiment does not limit the network structure in the semantic segmentation module. This embodiment does not limit the network structure in the contrastive learning module. For example, reference can be made to Figure 2 , Figure 2 An example diagram of a semantic segmentation model training architecture based on contrastive learning provided in an embodiment of the present invention. Sample features are input into the segmentation branch and the contrast branch respectively. The network structure of the semantic segmentation module in the segmentation branch may include an upsampling layer and a softmax classification layer; the network structure of the contrastive learning module in the contrast branch may include a convolution layer, an upsampling layer, and normalization.

[0058] Furthermore, the above-mentioned inputting of sample features into the semantic segmentation module to obtain a mask map may specifically include the following steps:

[0059] Step 21: Input the sample features to the upsampling layer to obtain the first output result;

[0060] Step 22: Input the first output result to the softmax classification layer to obtain a mask image; the pixel values ​​in the mask image represent the pixel categories.

[0061] Specifically, the feature extraction module will reduce the image resolution during the feature extraction process, so upsampling is required to improve the resolution. This embodiment does not limit the upsampling method. For example, the upsampling method can be bilinear interpolation, or it can be nearest neighbor interpolation. The softmax classification layer is used to classify pixels in the channel dimension, and finally outputs a single-channel mask image. The pixel value of the mask image represents the category of the pixel.

[0062] Furthermore, the above-mentioned inputting the sample features into the contrastive learning module to obtain the feature graph group may include the following steps:

[0063] Step 31: Input the sample features into the convolution layer to obtain a second output result;

[0064] Step 32: Input the second output result to the upsampling layer and the normalization layer to obtain a feature map group; the number of channels of the feature map group is the number of features.

[0065] Specifically, the convolution layer in this embodiment can be a 1x1 convolution, which is used to expand or compress the output channel dimension of the previous step to a fixed value D (D is generally 128 or 256), and then pass through an upsampling layer and a normalization layer. Among them, the normalization layer can use L1 or L2 normalization, which is used to reduce the computational disturbance caused by excessive pixel value variance. The final output result is a set of high-dimensional feature maps containing rich feature information. If the input is a 3-channel color image, the height of the input image is H, the width is W, and the number of channels is 3. At this time, the height of the feature map in the output feature map group is H, the width is W, and the number of channels is D. The number of channels represents the number of features.

[0066] S103: Calculate the semantic segmentation loss based on the label image and mask image corresponding to the training sample.

[0067] During the training phase, the mask image output by the semantic segmentation module will be compared with the corresponding manually annotated label image to calculate the semantic segmentation loss.

[0068] S104: Calculate the contrast loss based on the feature map group and the feature data in the feature queue.

[0069] The feature map group output by the contrastive learning module will be compared with the feature data in the feature queue to calculate the contrast loss. Specifically, during the training phase, the contrast branch will randomly select a certain number of pixels from the feature map group, that is, randomly select a certain amount of feature data with a size of 1x1 and a channel dimension of D, and compare it with the feature data in the queue. The obtained contrast loss is used to update the parameters of the semantic segmentation model. It should be noted that the feature queue is used to cyclically store feature data during the model training process. The feature queue stores multiple feature data with a size of 1x1 and a channel dimension of D (called a queue data).

[0070] S105: Dynamically update the parameters of the semantic segmentation model according to the semantic segmentation loss and the contrast loss, and dynamically update the feature data in the feature queue according to the feature map group and the feature data storage time; until the preset conditions are met, the training ends and a trained semantic segmentation model is obtained; the semantic segmentation model includes a feature extraction module and a semantic segmentation module.

[0071] It should be noted that the feature queue capacity in the present embodiment is set to 1 / N of the total number of training samples (the value of N is optimally 2 or 3). During the training process, the feature data of the feature queue is stored dynamically, that is, the feature data in the feature queue is dynamically updated according to the first-in-first-out or new-old update replacement method, and the parameters of the semantic segmentation model are also dynamically updated according to the semantic segmentation loss and contrast loss calculated in real time. This embodiment does not limit the preset conditions. For example, the number of training times is a preset number of times; or the loss value is less than a preset loss threshold. After the training is completed, the contrast branch is automatically removed, leaving only the segmentation branch, that is, the trained semantic segmentation model includes a trained feature extraction module and a semantic segmentation module. The present invention can be embedded in a semantic segmentation framework of any architecture without adding burden to the actual reasoning.

[0072] Furthermore, the above-mentioned dynamically updating the feature data in the feature queue according to the feature graph group and the feature data storage time may include the following steps:

[0073] When the feature queue is not full, the feature data of the feature graph group calculated in each iteration can be stored in sequence; when the feature queue is full, the feature data of the feature graph group calculated at the current iteration number is stored in the feature queue, and the feature data at the earliest storage time in the feature queue is deleted.

[0074] Specifically, the dynamic storage rule in this embodiment is: when the feature queue is full, the queue will automatically remove the old feature data and store new feature data. For example, the feature queue capacity is 100, when the 101th feature data comes in, the earliest 1st feature data will be cleared. In this way, the capacity of the feature queue can be controlled, and the demand for actual memory is small. At the same time, the feature data in the feature queue is constantly flowing and is not limited to fixed mini-batch data, so that the semantic segmentation model can be exposed to more data outside the current mini-batch, which is conducive to improving sample richness.

[0075] Furthermore, the above-mentioned dynamically updating the feature data in the feature queue according to the feature graph group and the feature data storage time may include the following steps:

[0076] When there are multiple objects of the same category in the feature map group, the regions of the same category are taken out along the channel direction; the number of channels in each region is the same;

[0077] Based on each area, the average is calculated in the height and width directions to obtain each feature data; the width and height of each feature data are 1 respectively, and the number of channels remains unchanged;

[0078] Based on the number of each feature data, part of the data of each feature data is taken and spliced ​​to obtain spliced ​​feature data; the width and height of the spliced ​​feature data are 1 respectively, and the number of channels remains unchanged;

[0079] The feature data in the feature queue is dynamically updated according to the spliced ​​feature data and the feature data storage time.

[0080] Specifically, a queue data represents the features of a category in an image. If there are multiple objects of the same category in an image, for example, there are multiple dogs or multiple cats in an image, then the following operations are performed. You can refer to Figure 3 For example, if there are two objects of category C in a feature map of size HxW and D channels, the two areas will be deducted along the channel direction, and then averaged in the width and height directions. After processing, the size is 1x1 and the number of channels is D. Then half of the channels are randomly selected in each channel direction and spliced ​​into a new queue data of size 1x1 and dimension D. This queue data represents the features of category C in the current image. Then the obtained feature data (the spliced ​​feature data) is put into the feature queue to become a queue data.

[0081] The semantic segmentation model training method based on contrastive learning provided by the embodiment of the present invention is applied, and the sample features of the training samples are extracted by using a feature extraction module; the sample features are input into the semantic segmentation module and the contrastive learning module to obtain a mask image and a feature image group respectively; the semantic segmentation loss is calculated according to the label image corresponding to the training sample and the mask image; the contrast loss is calculated according to the feature image group and the feature data in the feature queue; the parameters of the semantic segmentation model are dynamically updated according to the semantic segmentation loss and the contrast loss, and the feature data in the feature queue is dynamically updated according to the feature image group and the feature data storage time; until the preset conditions are met, the training is terminated, and a trained semantic segmentation model is obtained; the semantic segmentation model includes a feature extraction module and a semantic segmentation module. The present invention uses contrastive learning to assist in the training of the semantic segmentation model, and contrastive learning is used for contrast calculation in the pixel dimension to improve the segmentation accuracy of the entire semantic segmentation model; and in the training process, the feature data in the feature queue is dynamically updated by using the method of replacing the old with the new, avoiding the problem of reading all feature data at one time, or the problem of poor model training effect caused by insufficient feature data; while reducing the occupation of memory resources, the contrast loss of each iteration can be effectively and accurately calculated through the feature queue replaced by the old and the new, thereby improving the performance of the model. In addition, the content demand required by the feature queue is controllable, and the queue capacity can be set according to the actual content size; the feature queue is used to store the flowing feature data, avoiding the problem of memory explosion and slow training speed caused by storing all image feature data; at the same time, it also overcomes the shortcoming of limited mini-batch data, and the data output by the model can be compared with the data of the entire data set during model training, which can effectively improve the accuracy of the segmentation model without bringing too much memory and calculation burden.

[0082] Please refer to Figure 4 , Figure 4 A flowchart of an image detection method provided by an embodiment of the present invention. The method may include:

[0083] S201: using the above-mentioned semantic segmentation model training method based on contrastive learning to obtain a trained semantic segmentation model;

[0084] S202: Use the trained semantic segmentation model to detect the image to be detected and obtain a detection result.

[0085] In this embodiment, the semantic segmentation model training method based on contrastive learning is used to obtain a trained semantic segmentation model, which can be used for image detection. For example, industrial defect detection can be performed, such as being able to identify various scratches in an image.

[0086] The image detection method provided by the embodiment of the present invention is applied to obtain a trained semantic segmentation model by using the semantic segmentation model training method based on contrastive learning; the trained semantic segmentation model is used to detect the image to be detected to obtain a detection result. The present invention can improve the accuracy of image detection, and training the model based on the training method can reduce memory space.

[0087] The following is an introduction to a semantic segmentation model training device based on contrastive learning provided in an embodiment of the present invention. The semantic segmentation model training device based on contrastive learning described below and the semantic segmentation model training method based on contrastive learning described above can refer to each other.

[0088] Please refer to Figure 5 , Figure 5 A structural diagram of a semantic segmentation model training device based on contrastive learning provided by an embodiment of the present invention may include:

[0089] The sample feature acquisition module 100 is used to extract the sample features of the training sample using the feature extraction module;

[0090] A processing module 200 is used to input the sample features into a semantic segmentation module and a contrastive learning module to obtain a mask map and a feature map group respectively;

[0091] A first calculation module 300 is used to calculate the semantic segmentation loss according to the label image and the mask image corresponding to the training sample;

[0092] A second calculation module 400 is used to calculate the contrast loss according to the feature map group and the feature data in the feature queue;

[0093] The training update module 500 is used to dynamically update the parameters of the semantic segmentation model according to the semantic segmentation loss and the contrast loss, and dynamically update the feature data in the feature queue according to the feature map group and the feature data storage time; until the preset conditions are met, the training ends and a trained semantic segmentation model is obtained; the semantic segmentation model includes the feature extraction module and the semantic segmentation module.

[0094] Based on the above embodiment, the processing module 200 may include:

[0095] A first input unit, used to input the sample feature into the upsampling layer to obtain a first output result;

[0096] The second input unit is used to input the first output result into the softmax classification layer to obtain the mask image; the pixel values ​​in the mask image represent the pixel categories.

[0097] Based on the above embodiment, the processing module 200 may include:

[0098] A third input unit is used to input the sample feature into the convolution layer to obtain a second output result;

[0099] The fourth input unit is used to input the second output result to the upsampling layer and the normalization layer to obtain the feature map group; the number of channels of the feature map group is the number of features.

[0100] Based on the above embodiment, the training update module 500 may include:

[0101] The first updating unit is used to store the feature data of the feature graph group calculated by the current iteration number into the feature queue when the feature queue is full, and delete the feature data at the earliest storage time in the feature queue.

[0102] Based on the above embodiment, the training update module 500 may include:

[0103] A region extraction unit, used for extracting regions of the same category along the channel direction when there are multiple objects of the same category in the feature map group; the number of channels of each region is the same;

[0104] A calculation unit, used for averaging the height and width directions of the respective regions to obtain respective feature data; the width and height of the respective feature data are 1 respectively, and the number of channels remains unchanged;

[0105] A splicing unit, used for taking part of each feature data and splicing them based on the number of each feature data to obtain spliced ​​feature data; the width and height of the spliced ​​feature data are 1 respectively, and the number of channels remains unchanged;

[0106] The second updating unit is used to dynamically update the feature data in the feature queue according to the spliced ​​feature data and the feature data storage time.

[0107] It should be noted that the order of the modules and units in the above-mentioned semantic segmentation model training device based on contrastive learning can be changed without affecting the logic.

[0108] The semantic segmentation model training device based on contrastive learning provided by the embodiment of the present invention is applied, through the sample feature acquisition module 100, which is used to use the feature extraction module to extract the sample features of the training samples; the processing module 200 is used to input the sample features into the semantic segmentation module and the contrastive learning module to obtain the mask map and the feature map group respectively; the first calculation module 300 is used to calculate the semantic segmentation loss according to the label map corresponding to the training sample and the mask map; the second calculation module 400 is used to calculate the contrast loss according to the feature map group and the feature data in the feature queue; the training update module 500 is used to dynamically update the parameters of the semantic segmentation model according to the semantic segmentation loss and the contrast loss, and dynamically update the feature data in the feature queue according to the feature map group and the feature data storage time; until the preset conditions are met, the training ends and the trained semantic segmentation model is obtained; the semantic segmentation model includes the feature extraction module and the semantic segmentation module. The present invention uses contrastive learning to assist in the training of the semantic segmentation model, and contrastive learning is used for contrast calculation in the pixel dimension to improve the segmentation accuracy of the entire semantic segmentation model; and in the training process, the feature data in the feature queue is dynamically updated by using the method of replacing the old with the new, avoiding the problem of reading all feature data at one time, or the problem of poor model training effect caused by insufficient feature data; while reducing the occupation of memory resources, the contrast loss of each iteration can be effectively and accurately calculated through the feature queue replaced by the old and the new, thereby improving the performance of the model. In addition, the content demand required by the feature queue is controllable, and the queue capacity can be set according to the actual content size; the feature queue is used to store the flowing feature data, avoiding the problem of memory explosion and slow training speed caused by storing all image feature data; at the same time, it also overcomes the shortcoming of limited mini-batch data, and the data output by the model can be compared with the data of the entire data set during model training, which can effectively improve the accuracy of the segmentation model without bringing too much memory and calculation burden.

[0109] The following is an introduction to a semantic segmentation model training device based on contrastive learning provided in an embodiment of the present invention. The semantic segmentation model training device based on contrastive learning described below and the semantic segmentation model training method based on contrastive learning described above can refer to each other.

[0110] Please refer to Figure 6 , Figure 6 A schematic diagram of the structure of an image detection device provided by an embodiment of the present invention may include:

[0111] A model acquisition module 600 is used to obtain a trained semantic segmentation model using the above-mentioned semantic segmentation model training method based on contrastive learning;

[0112] The detection module 700 is used to detect the image to be detected using the trained semantic segmentation model to obtain a detection result.

[0113] It should be noted that the order of the modules and units in the above-mentioned image detection device can be changed without affecting the logic.

[0114] The image detection device provided by the embodiment of the present invention is applied, through the model acquisition module 600, used to obtain a trained semantic segmentation model using the above-mentioned semantic segmentation model training method based on contrastive learning; the detection module 700 is used to detect the image to be detected using the trained semantic segmentation model to obtain a detection result. The present invention can improve the accuracy of image detection, and training the model based on the above-mentioned training method can reduce memory space.

[0115] The electronic device provided by an embodiment of the present invention is introduced below. The electronic device described below and the semantic segmentation model training method and / or image detection method based on contrastive learning described above can be referenced to each other.

[0116] Please refer to Figure 7 , Figure 7 A schematic diagram of the structure of an electronic device provided by an embodiment of the present invention may include:

[0117] A memory 10, used for storing computer programs;

[0118] The processor 20 is used to execute a computer program to implement the above-mentioned semantic segmentation model training method and / or image detection method based on contrastive learning.

[0119] The memory 10 , the processor 20 , and the communication interface 31 all communicate with each other via the communication bus 32 .

[0120] In the embodiment of the present invention, the memory 10 is used to store one or more programs, and the program may include program code, and the program code includes computer operation instructions. In the embodiment of the present invention, the memory 10 may store programs for implementing the following functions:

[0121] Extracting sample features of training samples using a feature extraction module;

[0122] Input the sample features into the semantic segmentation module and the contrastive learning module to obtain the mask map and feature map group respectively;

[0123] The semantic segmentation loss is calculated based on the label image corresponding to the training sample and the mask image;

[0124] The contrast loss is calculated based on the feature data in the feature map group and the feature queue;

[0125] Dynamically update the parameters of the semantic segmentation model according to the semantic segmentation loss and the contrast loss, and dynamically update the feature data in the feature queue according to the feature map group and the feature data storage time; until the preset conditions are met, the training ends and a trained semantic segmentation model is obtained; the semantic segmentation model includes a feature extraction module and a semantic segmentation module;

[0126] or,

[0127] Using the above-mentioned semantic segmentation model training method based on contrastive learning to obtain a trained semantic segmentation model;

[0128] Use the trained semantic segmentation model to detect the image to be detected and obtain the detection result.

[0129] In a possible implementation, the memory 10 may include a program storage area and a data storage area, wherein the program storage area may store an operating system and an application required for at least one function, etc.; the data storage area may store data created during use.

[0130] In addition, the memory 10 may include a read-only memory and a random access memory, and provide instructions and data to the processor. A portion of the memory may also include an NVRAM. The memory stores an operating system and operating instructions, executable modules or data structures, or a subset thereof, or an extended set thereof, wherein the operating instructions may include various operating instructions for implementing various operations. The operating system may include various system programs for implementing various basic tasks and processing hardware-based tasks.

[0131] The processor 20 may be a central processing unit (CPU), an application specific integrated circuit, a digital signal processor, a field programmable gate array or other programmable logic device, a microprocessor or any conventional processor, etc. The processor 20 may call a program stored in the memory 10 .

[0132] The communication interface 31 may be an interface of a communication module, and is used to connect to other devices or systems.

[0133] Of course, it should be noted that Figure 7 The structure shown does not constitute a limitation on the electronic device in the embodiment of the present invention. In actual applications, the electronic device may include Figure 7 More or fewer components than shown, or combinations of certain components.

[0134] The computer-readable storage medium provided in an embodiment of the present invention is introduced below. The computer-readable storage medium described below and the semantic segmentation model training method and / or image detection method based on contrastive learning described above can be referenced to each other.

[0135] The present invention also provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the above-mentioned semantic segmentation model training method and / or image detection method based on contrastive learning are implemented.

[0136] The computer-readable storage medium may include: a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and other media that can store program codes.

[0137] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.

[0138] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0139] Finally, it should be noted that, in this article, relationships such as first and second, etc. are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprises" or any other variations are intended to cover non-exclusive inclusion, so that a process, method, article or device that includes a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device.

[0140] The above is a detailed introduction to the semantic segmentation model training method, image detection method, device, equipment and computer-readable storage medium based on contrastive learning provided by the present invention. Specific examples are used in this article to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea; at the same time, for those skilled in the art, according to the ideas of the present invention, there will be changes in the specific implementation methods and application scopes. In summary, the content of this specification should not be understood as a limitation on the present invention.

Claims

1. A semantic segmentation model training method based on contrastive learning, characterized in that: include: Extracting sample features of training samples using a feature extraction module; Inputting the sample features into a semantic segmentation module and a contrastive learning module to obtain a mask map and a feature map group respectively; Calculating semantic segmentation loss according to the label image and the mask image corresponding to the training sample; Calculating the contrast loss based on the feature map group and the feature data in the feature queue; The parameters of the semantic segmentation model are dynamically updated according to the semantic segmentation loss and the contrast loss, and the feature data in the feature queue are dynamically updated according to the feature map group and the feature data storage time; until the preset conditions are met, the training ends and a trained semantic segmentation model is obtained; the semantic segmentation model includes the feature extraction module and the semantic segmentation module.

2. The semantic segmentation model training method based on contrastive learning according to claim 1, characterized in that: The sample features are input into the semantic segmentation module to obtain a mask map, including: Inputting the sample features into an upsampling layer to obtain a first output result; The first output result is input into a softmax classification layer to obtain the mask image; the pixel values ​​in the mask image represent the pixel categories.

3. The semantic segmentation model training method based on contrastive learning according to claim 1, characterized in that: The sample features are input into the contrastive learning module to obtain a feature map group, including: Inputting the sample features into a convolutional layer to obtain a second output result; The second output result is input into the upsampling layer and the normalization layer to obtain the feature map group; the number of channels of the feature map group is the number of features.

4. The semantic segmentation model training method based on contrastive learning according to claim 1, characterized in that: Dynamically updating the feature data in the feature queue according to the feature graph group and the feature data storage time includes: When the feature queue is full, the feature data of the feature graph group calculated at the current iteration number is stored in the feature queue, and the feature data at the earliest storage time in the feature queue is deleted.

5. The semantic segmentation model training method based on contrastive learning according to any one of claims 1 to 4, characterized in that: Dynamically updating the feature data in the feature queue according to the feature graph group and the feature data storage time includes: When there are multiple objects of the same category in the feature map group, each region of the same category is taken out along the channel direction; each region has the same number of channels; Based on each region, averaging is performed in the height and width directions to obtain each feature data; the width and height of each feature data are 1 respectively, and the number of channels remains unchanged; Based on the number of each feature data, respectively take part of the data of each feature data and splice them to obtain spliced ​​feature data; the width and height of the spliced ​​feature data are 1 respectively, and the number of channels remains unchanged; The feature data in the feature queue is dynamically updated according to the spliced ​​feature data and the feature data storage time.

6. An image detection method, characterized in that: include: Using the semantic segmentation model training method based on contrastive learning as described in any one of claims 1 to 5 to obtain a trained semantic segmentation model; The trained semantic segmentation model is used to detect the image to be detected to obtain a detection result.

7. A semantic segmentation model training device based on contrastive learning, characterized in that: include: A sample feature acquisition module is used to extract sample features of training samples using a feature extraction module; A processing module, used for inputting the sample features into a semantic segmentation module and a contrastive learning module to obtain a mask map and a feature map group respectively; A first calculation module, used to calculate the semantic segmentation loss according to the label map and the mask map corresponding to the training sample; A second calculation module, used for calculating the contrast loss according to the feature map group and the feature data in the feature queue; A training update module is used to dynamically update the parameters of the semantic segmentation model according to the semantic segmentation loss and the contrast loss, and dynamically update the feature data in the feature queue according to the feature map group and the feature data storage time; until the preset conditions are met, the training ends and a trained semantic segmentation model is obtained; the semantic segmentation model includes the feature extraction module and the semantic segmentation module.

8. An image detection device, characterized in that: include A model acquisition module, used to obtain a trained semantic segmentation model using the semantic segmentation model training method based on contrastive learning as described in any one of claims 1 to 5; The detection module is used to detect the image to be detected using the trained semantic segmentation model to obtain a detection result.

9. An electronic device, characterized in that: include: Memory for storing computer programs; A processor, configured to implement the method according to any one of claims 1 to 6 when executing the computer program.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, and when the computer-executable instructions are loaded and executed by a processor, the method according to any one of claims 1 to 6 is implemented.

Images