An adversarial training method for semantic segmentation in computer vision

Abstract

The present invention relates to the technical field of adversarial training methods, and specifically to a semantic segmentation adversarial training method for computer vision, including the following steps: S1: pixel-level pre-training; S2: establishing a training model based on the pixel-level pre-training process; S3: model extension interactive training. In the present invention, in terms of image cropping training, the training process is to further optimize the existing model, and the unit time consumed by the training process is less, which can effectively improve the operation efficiency. And when migrating to downstream segmentation tasks, the generalization is ensured based on the query encoder and the key encoder, and there is no need to set the weight decay again. During the training operation process, the similarity between each pixel and other pixels is calculated as a weight term, and the transformed X variable is weighted and summed to obtain the Y variable. A consistency constraint between the generated Y variable and the X<'gt;' variable is generated, and on this basis, model extension interactive training is carried out to further improve the model generalization and training effect.

Description

Technical Field

[0001] The present invention relates to the technical field of adversarial training methods, and particularly to a semantic segmentation adversarial training method for computer vision. Background Art

[0002] The adversarial training method is a method of selecting corresponding samples for adversarial pre-training to sandwich the following task training. In the existing adversarial training methods, the proxy task is usually a classification task. Directly migrating the backbone to a dense task such as segmentation will lead to a task gap, and the training often uses the decoder of the segmentation model. The decoder uses a random initialization method, which will lead to the occurrence of an architectural gap and needs to be improved. Summary of the Invention

[0003] The purpose of the present invention is to solve the disadvantages existing in the prior art, and to propose a semantic segmentation adversarial training method for computer vision.

[0004] To achieve the above purpose, the present invention adopts the following technical solutions: A semantic segmentation adversarial training method for computer vision, including the following steps:

[0005] S1: Pixel-level pre-training;

[0006] S2: Based on the pixel-level pre-training process, establish a training model;

[0007] S3: Model expansion interactive training.

[0008] As a further solution of the present invention, in S1, the steps of the pixel-level pre-training are specifically:

[0009] S110: Image cropping training;

[0010] S120: Compare the pixel dense contrast loss and the instance contrast loss, and import the samples into the shielding pool.

[0011] As a further solution of the present invention, in S110, the steps of the image cropping training are specifically:

[0012] S111: Crop and intercept two arbitrary positions of a single training image to obtain a first screenshot and a second screenshot;

[0013] S112: Select two arbitrary images as background images;

[0014] S113: Paste the first screenshot and the second screenshot cropped and intercepted in S111 as foreground images onto two groups of background images respectively.

[0015] As a further solution of the present invention, in S120, the steps of comparing the pixel density contrast loss and the instance contrast loss and importing the samples into the shielding pool are specifically as follows:

[0016] S121: Introduce a query encoder and a key encoder, and update the key encoder through the query encoder;

[0017] S122: Use the pasting result of the first screenshot in S113 as sample one, and the pasting result of the second screenshot as sample two;

[0018] S123: Query and compare the pixel density contrast loss of sample one and sample two;

[0019] S124: Import sample one and sample two into the shielding pool;

[0020] S125: Query and compare the instance contrast loss of sample one and sample two in the shielding pool.

[0021] As a further solution of the present invention, in S123, the comparison formula for the pixel density contrast loss is as follows:

[0022]

[0023] Where E is the recording result of the pixel density contrast loss, E1 is the total energy consumption per unit time, and W1 is the total forming mass of sample one and sample two per unit time.

[0024] As a further solution of the present invention, in S125, the instance contrast loss uses a contrast loss function. Specifically, during the execution of S111, a random data augmentation module is used to take the execution result as the augmented image, and based on the query encoder and the key encoder, sample one and sample two corresponding to the augmented image are extracted. The projection head uses a set of small auxiliary network representations for further transformation, and instances are generated from the projection frames based on the instance mapping function. The contrast loss function makes the positive sample representation results as close as possible and ensures the difference from the negative samples.

[0025] As a further solution of the present invention, in S2, the steps of establishing the training model are specifically as follows:

[0026] S210: Establish a model library file and establish a training model in the model library file;

[0027] S220: Automatically record the overall step process of S1 based on the training model;

[0028] S230: Package the recording items of S220 and store the training model file.

[0029] As a further solution of the present invention, in S220, the step of automatically recording the overall step flow of S1 is specifically as follows:

[0030] S221: In the input stage, extract and enhance Sample One and Sample Two;

[0031] S222: Obtain the skeleton and projection data of Sample One, and generate the X variable;

[0032] S223: Obtain the momentum skeleton and projection data of Sample Two, and generate the X ′ variable;

[0033] S224: Record the pixel density contrast loss in S123 and the instance contrast loss in S125;

[0034] S225: Transmit through the image transmission module, calculate the similarity between each pixel and other pixels, use it as a weight term, perform weighted summation on the transformed X variable, obtain the Y variable, and generate the consistency constraint between the Y variable and the X ′ variable.

[0035] As a further solution of the present invention, in S3, the steps of model expansion interactive training are specifically as follows:

[0036] S310: Based on the training model file obtained in S2, introduce another set of training pictures;

[0037] S320: Use the same pixel of the current training picture from another perspective as the positive sample;

[0038] S330: Arbitrarily select a pixel from another set of training pictures as the negative sample;

[0039] S340: Call the S120 step to compare the pixel density contrast loss and the instance contrast loss;

[0040] S350: Call the S220 step, automatically record the process of S340, and generate a new training model based on the called training model file.

[0041] As a further solution of the present invention, in S3, the steps of model expansion interactive training further include establishing an association between the new training model and the called training model file based on the comparison of the obtained Y variable values, that is, arranging them in descending order as a reference based on the Y variable value thickness, and generating an association index.

[0042] Compared with the prior art, the advantages and positive effects of the present invention are as follows:

[0043] In the present invention, in the method of image cropping training, the training process is to further optimize an existing model. The training process consumes less unit time, can effectively improve the operation efficiency, and when migrating to downstream segmentation tasks, the generalization is ensured based on the query encoder and the key encoder, without the need to set the weight decay again. During the training operation process, the similarity between each pixel and other pixels is calculated as a weight term, and the transformed X variable is weighted and summed to obtain the Y variable, and a consistency constraint between the generated Y variable and the X variable is generated. On this basis, model expansion interactive training is carried out to further improve the model generalization and training effect. ′ A consistency constraint between the Y variable and the X variable is generated, and on this basis, model expansion interactive training is carried out to further improve the model generalization and training effect. Description of the Drawings

[0044] Figure 1 It is a schematic diagram of the main steps of a method for adversarial training of semantic segmentation in computer vision proposed by the present invention;

[0045] Figure 2 It is a schematic diagram of the operation process of a method for adversarial training of semantic segmentation in computer vision proposed by the present invention;

[0046] Figure 3 It is a schematic diagram of the refinement of step 1 of a method for adversarial training of semantic segmentation in computer vision proposed by the present invention;

[0047] Figure 4 It is a schematic diagram of the refinement of step 110 of a method for adversarial training of semantic segmentation in computer vision proposed by the present invention;

[0048] Figure 5 It is a schematic diagram of the refinement of step 120 of a method for adversarial training of semantic segmentation in computer vision proposed by the present invention;

[0049] Figure 6 It is a schematic diagram of the refinement of step 2 of a method for adversarial training of semantic segmentation in computer vision proposed by the present invention;

[0050] Figure 7 It is a schematic diagram of the refinement of step 220 of a method for adversarial training of semantic segmentation in computer vision proposed by the present invention;

[0051] Figure 8 It is a schematic diagram of the refinement of step 3 of a method for adversarial training of semantic segmentation in computer vision proposed by the present invention. Detailed Embodiment

[0052] In order to make the objectives, technical solutions and advantages of the present invention clearer, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0053] In the description of the present invention, it should be understood that the orientation or positional relationship indicated by the terms "length", "width", "upper", "lower", "front", "rear", "left", "right", "vertical", "horizontal", "top", "bottom", "inner", "outer", etc. is based on the orientation or positional relationship shown in the drawings. It is only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore should not be construed as a limitation to the present invention. In addition, in the description of the present invention, the meaning of "a plurality" is two or more, unless otherwise specifically defined.

[0054] Embodiment 1

[0055] Please refer to Figures 1 to 8 , the present invention provides a technical solution: a semantic segmentation adversarial training method for computer vision, including the following steps:

[0056] S1: pixel-level pre-training;

[0057] S2: Based on the pixel-level pre-training process, establish a training model;

[0058] S3: Model extension interactive training.

[0059] Specifically, in the way of image cropping training, its training process is to further optimize the existing model. The unit time consumed in the training process is less, which can effectively improve the operation efficiency. And when migrating to the downstream segmentation task, based on the query encoder and the key encoder, its generalization is ensured, and there is no need to set the weight decay again. During the training operation process, calculate the similarity between each pixel and other pixels, use it as a weight term, and perform a weighted sum on the transformed X variable to obtain the Y variable, generate the consistency constraint between the generated Y variable and the X ′ variable, and on this basis, perform model extension interactive training to further improve the model generalization and training effect.

[0060] Please refer to Figure 3 , in S1, the steps of pixel-level pre-training are specifically as follows:

[0061] S110: Image cropping training;

[0062] S120: Compare the pixel dense contrast loss and the instance contrast loss, and import the samples into the shielding pool.

[0063] Specifically, based on steps S110 to S120, a pixel-level pre-training effect can be achieved. In the image cropping training method, the training process is to further optimize an existing model, with less unit time consumption in the training process, which can effectively improve the operation efficiency. And when migrating to downstream segmentation tasks, the generalization is ensured based on the query encoder and the key encoder, without the need to set weight decay again.

[0064] Please refer to Figure 4 , in S110, the steps of image cropping training are specifically as follows:

[0065] S111: Crop two arbitrary positions of a single training image to obtain a first screenshot and a second screenshot;

[0066] S112: Select two arbitrary images as background images;

[0067] S113: Use the first screenshot and the second screenshot cropped in S111 as foreground images, and paste them onto the two groups of background images respectively.

[0068] Specifically, paste the cropped images as foregrounds onto two different background images, and make the graphic features and average features at the foreground positions similar.

[0069] Please refer to Figure 5 , in S120, the steps of comparing the pixel dense contrast loss and the instance contrast loss and importing the samples into the masking pool are specifically as follows:

[0070] S121: Introduce a query encoder and a key encoder, and update the key encoder through the query encoder;

[0071] S122: Use the pasting result of the first screenshot in S113 as sample one, and the pasting result of the second screenshot as sample two;

[0072] S123: Query and compare the pixel dense contrast loss of sample one and sample two;

[0073] S124: Import sample one and sample two into the masking pool;

[0074] S125: Query and compare the instance contrast loss of sample one and sample two in the masking pool.

[0075] Specifically, for different crops of the same image, constrain the consistency of the features at their original positions.

[0076] In S123, the comparison formula for the pixel dense contrast loss is as follows:

[0077]

[0078] Among them, E is the recording result of pixel density contrast loss, E1 is the total energy consumption per unit time, and W1 is the total forming mass of Sample 1 and Sample 2 per unit time.

[0079] In S125, the instance contrast loss adopts a contrast loss function. Specifically, during the execution of S111 by a random data augmentation module, the execution result is used as the enhanced image, and based on the query encoder and the key encoder, Sample 1 and Sample 2 corresponding to the enhanced image are extracted. The projection head uses a set of small auxiliary network representations for further transformation, and instances are generated from the projection frame based on the instance mapping function. The contrast loss function makes the positive sample representation result as close as possible and ensures the difference from the negative sample.

[0080] Please refer to Figure 6 , in S2, the steps to establish the training model are specifically as follows:

[0081] S210: Establish a model library file and establish a training model in the model library file;

[0082] S220: Based on the training model, automatically record the overall step process of S1;

[0083] S230: Package the recording items of S220 and store the training model file.

[0084] Specifically, this step automatically generates and stores the generated training model file by establishing a model library file, facilitating the subsequent steps of calling, refinement, and indexing.

[0085] Please refer to Figure 7 , in S220, the steps to automatically record the overall step process of S1 are specifically as follows:

[0086] S221: In the input stage, extract and enhance Sample 1 and Sample 2;

[0087] S222: Obtain the skeleton and projection data of Sample 1 and generate the X variable;

[0088] S223: Obtain the momentum skeleton and projection data of Sample 2 and generate the X ′ variable;

[0089] S224: Record the pixel density contrast loss in S123 and the instance contrast loss in S125;

[0090] S225: Transmit through the image transmission module and calculate the similarity between each pixel and other pixels as the weight term, and perform a weighted sum on the transformed X variable to obtain the Y variable, generating the consistency constraint between the Y variable and the X ′ variable.

[0091] Specifically, in this step, the weight values are obtained by calculating the similarity between each pixel and other pixels as the weight terms, and the weighted sum is calculated to obtain the Y variable and generate constraints.

[0092] Please refer to Figure 8 , in S3, the steps of model extended interactive training are specifically as follows:

[0093] S310: Based on the training model file obtained in S2, introduce another set of training pictures.

[0094] S320: Use the same pixels of the current training picture from another perspective as the positive samples.

[0095] S330: Randomly select pixels from another set of training pictures as the negative samples.

[0096] S340: Call the comparison of pixel density contrast loss and instance contrast loss in step S120.

[0097] S350: Call step S220 to automatically record the process of step S340 and generate a new training model based on the called training model file.

[0098] In S3, the steps of model extended interactive training also include establishing the association between the new training model and the called training model file based on the comparison of the obtained Y variable values, that is, arranging them in descending order with the thickness of the Y variable value as the reference and generating an association index.

[0099] Specifically, in step S3, model extended interactive training is performed on steps S1 to S2 to further improve the model generalization and training effect.

[0100] Working principle: pixel-level pre-training [image cropping training (crop and intercept two arbitrary positions of a single training image to obtain the first screenshot and the second screenshot, select two arbitrary images as background images, crop and intercept the first screenshot and the second screenshot of S111 as foreground images, and paste them onto the two groups of background images respectively), compare the pixel density contrast loss and the instance contrast loss, and import the samples into the shielding pool (introduce the query encoder and the key encoder, and update the key encoder through the query encoder. In S113, the pasting result of the first screenshot is used as sample one, and the pasting result of the second screenshot is used as sample two. Query and compare the pixel density contrast loss of sample one and sample two, introduce sample one and sample two into the shielding pool, and query and compare the instance contrast loss of sample one and sample two in the shielding pool)]; Based on the pixel-level pre-training process, establish a training model [establish a model library file, and establish a training model in the model library file. Based on the training model, automatically record the overall S1 step process (in the input stage, extract and enhance sample one and sample two, obtain the skeleton and projection data of sample one, generate the X variable, obtain the momentum skeleton and projection data of sample two, generate the X ′ variable, record the pixel density contrast loss in S123 and the instance contrast loss in S125, transmit through the image transmission module, and calculate the similarity between each pixel and other pixels as the weight term, and perform a weighted sum on the transformed X variable to obtain the Y variable, and generate the consistency constraint between the Y variable and the X ′ variable), encapsulate the S220 record item, and store the training model file]; Model extension interactive training (based on the training model file obtained in S2, introduce another group of training images, use the same pixel of the current training image from another perspective as the positive sample, and use any pixel selected from another group of training images as the negative sample. Call the S120 step to compare the pixel density contrast loss and the instance contrast loss, call the S220 step, automatically record the S340 step process, and generate a new training model based on the called training model file) Based on the comparison of the obtained Y variable values, establish the association between the new training model and the called training model file, sort in descending order based on the thickness of the Y variable value as the reference, and generate the association index.

[0101] The above is only the preferred embodiment of the present invention, and it does not limit the present invention in other forms. Any person skilled in the art may use the disclosed technical content to make changes or modifications into equivalent embodiments with equivalent changes and apply them to other fields. However, as long as it does not depart from the technical content of the technical solution of the present invention, any simple modification, equivalent change and modification made to the above embodiments based on the technical essence of the present invention still fall within the protection scope of the technical solution of the present invention.

Claims

1. A semantic segmentation adversarial training method for computer vision, characterized in that, It includes the following steps: S1: Pixel-level pre-training. The specific steps of S1 are as follows: S110: Obtain the first screenshot and the second screenshot through image cropping and paste them; S120: Compare the pixel density contrast loss and the instance contrast loss, and import the samples into the masking pool. Specifically, it includes: S121: Introduce a query encoder and a key encoder, and update the key encoder through the query encoder; S122: Take the paste result of the first screenshot as sample one, and the paste result of the second screenshot as sample two; S123: Query the pixel density comparison loss between sample one and sample two. The calculation formula for the pixel density comparison loss is as follows: , where E is the recorded result of the pixel density comparison loss, E1 is the total energy consumption per unit time, and W1 is the total molding mass of sample one and sample two per unit time. S124: Import sample one and sample two into the masking pool; S125: Query and compare the instance contrast loss of sample one and sample two in the masking pool. The instance contrast loss uses a contrast loss function. Specifically, during the execution of image cropping through a random data augmentation module, the execution result is used as the enhanced image, and based on the query encoder and the key encoder, sample one and sample two corresponding to the enhanced image are extracted. The projection head uses a set of small auxiliary network representations for further transformation. Instances are generated from the projection frames based on the instance mapping function. The contrast loss function makes the positive sample representation results as close as possible and ensures the difference from the negative samples; S2: Based on the pixel-level pre-training process, establish a training model. The specific steps of S2 are as follows: S220: Automatically record the overall steps of S1 based on the training model. Specifically, it includes: S221: At the input stage, extract and enhance sample one and sample two; S222: Obtain the skeleton and projection data of sample one and generate the X variable; S223: Obtain the momentum skeleton and projection data of sample two and generate the X' variable; S224: Record the pixel density contrast loss in S123 and the instance contrast loss in S125; S225: Transmit through the image transmission module and calculate the similarity between each pixel and other pixels as a weight term, and perform a weighted sum on the transformed X variable to obtain the Y variable, generating a consistency constraint between the Y variable and the X' variable; S230: Package the S220 recording items and store the training model file; S3: Model extension interactive training. The steps of the model extension interactive training include: generating a new training model based on the called training model file, establishing an association between the new training model and the called training model file based on the comparison of the obtained Y variable values, that is, arranging in descending order based on the thickness of the Y variable values as a reference and generating an association index.

2. The semantic segmentation adversarial training method for computer vision according to claim 1, wherein In S110, the specific steps of the image cropping training are as follows: S111: Crop and intercept two arbitrary positions of a single training picture to obtain the first screenshot and the second screenshot; S112: Select two arbitrary pictures as background pictures; S113: Use the first screenshot and the second screenshot cropped and intercepted in S111 as foreground pictures and paste them onto the two groups of background pictures respectively.

3. The semantic segmentation adversarial training method for computer vision according to claim 1, characterized in that, In S3, the specific steps of the model extension interactive training are as follows: S310: Based on the training model file obtained in S2, introduce another group of training pictures; S320: Use the same pixel of the current training picture from another perspective as the positive sample; S330: Select any pixel from another set of training images as a negative sample; S340: Call the steps in S120 to compare the pixel density contrast loss and the instance contrast loss; S350: Call the steps in S220, automatically record the process of S340, and generate a new training model based on the called training model file.

Images