Sample generation method, sample generation device and computer storage medium

By acquiring the original image and using the detection model and segmentation model to generate sample images, combining the diffusion generation model and object replacement model, the problem of low sample generation efficiency in image difference detection is solved, and automated high-quality sample generation is realized, suitable for change detection Model training.

CN118658025BActive Publication Date: 2024-12-06ZHEJIANG DAHUA TECH CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411143542.4
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-08-20
Publication Date
2024-12-06
Estimated Expiration
2044-08-20

Smart Images

  • Figure CN118658025B_ABST
    Figure CN118658025B_ABST
Patent Text Reader

Abstract

The present application proposes a sample generation method, a sample generation device, and a computer storage medium. The sample generation method comprises: obtaining an original image; inputting the preset target prompt information and the original image into a detection model to obtain target detection frame information in the original image; inputting the target detection frame information and the original image into a segmentation model to obtain a target mask map; and generating a sample image based on the change information of the target mask map at the target position of the original image. Through the above sample generation method, unlabeled images are processed to automatically generate samples and annotations.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a sample generation method, a sample generation device and a computer storage medium. Background Art

[0002] In recent years, with the maturity of deep learning-based image target detection technology, image difference detection technology based on target detection framework has been widely used in many fields such as video surveillance, satellite image analysis, and manufacturing quality control. Image difference analysis can be used to identify changes in the same scene at different times, and then the results of image changes can be applied to specific practical applications.

[0003] The object of image difference detection can be any entity that can be defined and whose color and state may change. Since the semantic differences of the detection objects vary greatly, the deep learning method is used to train the change detection network, which requires a large number of samples to meet the usage requirements.

[0004] With the rapid development of artificial intelligence, large model technology is also evolving. Large model technology refers to a neural network model with a large number of parameters trained using deep learning methods. Usually, a large model corresponds to a large data set. After learning a large amount of data, the network can fit a variety of objects. Therefore, the semantic expression ability of the large model is relatively good. Automatic labeling of unlabeled samples helps to speed up the data labeling process, reduce the burden of manual labeling, and improve the accuracy and consistency of labeling. However, these automated labeling methods still require human participation and supervision to ensure the quality and accuracy of the labeling results, resulting in low efficiency in generating labeled samples. Summary of the invention

[0005] In order to solve the above technical problems, the present application proposes a sample generation method, a sample generation device and a computer storage medium.

[0006] In order to solve the above technical problems, the present application proposes a sample generation method, which includes:

[0007] Get the original image;

[0008] Inputting the preset target prompt information and the original image into the detection model to obtain the target detection frame information in the original image;

[0009] Input the target detection frame information and the original image into a segmentation model to obtain a target mask map;

[0010] A sample image is generated based on change information of the target mask image at the target position of the original image.

[0011] The step of generating a sample image based on change information of the target mask image at the target position of the original image includes:

[0012] Determine a changed region image of a target position in the original image based on the target mask image;

[0013] Generate Gaussian white noise corresponding to the prompt words in the prompt library based on the diffusion generation model;

[0014] Processing the changed region image by using Gaussian white noise of the diffusion generation model to obtain the change information;

[0015] The sample image is generated based on the change information and the original image.

[0016] Wherein, the sample generation method includes:

[0017] Inputting the first training image into the diffusion generation model to be trained, iterating multiple times according to the forward diffusion process, and generating a forward diffusion loss based on the predicted white noise and the preset white noise generated on the training image in each iteration;

[0018] Inputting the second training image into the diffusion generation model to be trained, iterating multiple times according to the reverse diffusion process, and generating reverse diffusion loss based on the restored image and the target image obtained in each iteration;

[0019] The diffusion generation model to be trained is trained based on the forward diffusion loss and the reverse diffusion loss.

[0020] The step of generating a sample image based on change information of the target mask image at the target position of the original image includes:

[0021] Acquire a sample library, wherein the sample library includes a plurality of example images;

[0022] Any instance image in the sample library, the target mask image, and the original image are input into the object substitution model to obtain a sample image output by the object substitution model.

[0023] The step of generating a sample image based on change information of the target mask image at the target position of the original image includes:

[0024] Get a sample erasure model of the prediction process of the pre-trained erasure network;

[0025] The target mask image and the original image are input into the sample erasing model, so that the sample erasing model fills the target position in the original image with the image background according to the prediction process of the erasing network, and outputs the sample image

[0026] Wherein, after generating the sample image based on the change information of the target position of the original image in the target mask image, the sample generation method further includes:

[0027] Inputting the sample image and the original image into a change detection model to obtain a detection mask image output by the change detection model;

[0028] Inputting the detection mask image and the target mask image of the original image into a comparison control module to obtain a comparison result;

[0029] In response to the comparison result that the sample image meets the quality requirement, the sample image is output as a target sample image.

[0030] The step of inputting the detection mask image and the target mask image of the original image into a comparison control module to obtain a comparison result includes:

[0031] Inputting the detection mask image and the target mask image of the original image into a comparison control module, and calculating the number of pixels having the same pixel value in the detection mask image and the target mask image;

[0032] Based on the number of pixels, obtaining a mask score of the sample image;

[0033] When the mask score is greater than or equal to a preset threshold, a comparison result indicating that the sample image meets the quality requirement is output.

[0034] Wherein, the sample generation method further includes:

[0035] Acquire a training image and a generated image generated based on the training image, and a real annotated image corresponding to the training image;

[0036] Based on the generated images and the real annotated images, forming several training sets;

[0037] Inputting the generated image in each training set and the training image into the change detection model to be trained to obtain a predicted output mask image;

[0038] Calculating a training loss value based on the output mask image and the true annotated image;

[0039] The change detection model to be trained is trained based on the training loss value.

[0040] The number of the generated images is greater than the number of the real annotated images, the generated images and the real annotated images form the training set according to a preset mixing ratio, and each real annotated image is trained more times than each generated image.

[0041] To solve the above technical problems, the present application also proposes a sample generation device, which includes a memory and a processor coupled to the memory; wherein the memory is used to store program data, and the processor is used to execute the program data to implement the sample generation method as described above.

[0042] In order to solve the above technical problem, the present application also proposes a computer storage medium, which is used to store program data. When the program data is executed by a computer, it is used to implement the above sample generation method.

[0043] Compared with the prior art, the beneficial effects of the present application are as follows: the sample generation device obtains the original image; inputs the preset target prompt information and the original image into the detection model to obtain the target detection frame information in the original image; inputs the target detection frame information and the original image into the segmentation model to obtain the target mask map; and generates the sample image based on the change information of the target mask map in the target position of the original image. Through the above sample generation method, the unlabeled image is processed to automatically generate samples and annotations. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work. Among them:

[0045] Figure 1 It is a flow chart of an embodiment of a sample generation method provided by the present application;

[0046] Figure 2 It is a schematic diagram of the overall process of the change detection sample generation method provided by the present application;

[0047] Figure 3 yes Figure 1 The schematic diagram of the flow chart of the first implementation mode of step S14 of the sample generating method shown;

[0048] Figure 4 yes Figure 1 The schematic diagram of the flow chart of the second implementation mode of step S14 of the sample generating method shown;

[0049] Figure 5 It is a flowchart of the object replacement sample generation method provided by this application;

[0050] Figure 6 yes Figure 1 The schematic diagram of the flow chart of the third implementation mode of step S14 of the sample generating method shown;

[0051] Figure 7 is a flow chart of another embodiment of the sample generation method provided by the present application;

[0052] Figure 8 It is a process framework diagram of the sample quality screening method provided in this application;

[0053] Fig. 9 It is a schematic diagram of the framework of an embodiment of a change detection algorithm structure provided by the present application;

[0054] Fig.10 yes Fig. 9 A schematic diagram of the framework of the basic structure of feature extraction in the change detection algorithm structure shown;

[0055] Fig.11 is a structural schematic diagram of an embodiment of a sample generation device provided by the present application;

[0056] Fig.12 It is a structural diagram of an embodiment of a computer storage medium provided by the present application. DETAILED DESCRIPTION

[0057] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0058] The terms "first", "second", "third", "fourth", etc. (if any) in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present application described herein can, for example, be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0059] The change detection sample generation method based on a large model provided in this application can generate data that can be directly used for training change segmentation models and change detection models without manual annotation. The sample generation method of this application can not only simulate object changes, but also simulate the disappearance and appearance of objects, so that the lightweight change detection model has semantic information that can be comparable to that of a large model, thereby improving the accuracy and recall rate of the model. At the same time, a sample quality control method that does not require manual participation is designed for the generated data to eliminate defective samples and make sample generation more accurate.

[0060] Specifically, this application proposes a new method to solve the problem of change detection sample generation. It uses multiple trained large models to process unlabeled images and automatically generate samples and annotations. Therefore, no training data is required, and samples can be generated by providing only the original images. At the same time, a sample quality screening method is designed to automatically remove low-quality data samples so that the generated data can be directly used for network training.

[0061] Therefore, this application involves two parts, the first part is the sample generation method, and the second part is the automated quality screening of the generated data. The following is a detailed description of the sample generation method:

[0062] Please refer to Figure 1 and Figure 2 , Figure 1 is a flow chart of an embodiment of a sample generation method provided by the present application. Figure 2 It is a schematic diagram of the overall process of the change detection sample generation method provided by this application.

[0063] The sample generation method of the present application is applied to a sample generation device, wherein the sample generation device of the present application can be a server, a terminal device, or a system in which a server and a terminal device cooperate with each other. Accordingly, the various parts included in the sample generation device, such as various units, sub-units, modules, and sub-modules, can all be set in the server, can all be set in the terminal device, or can be set in the server and the terminal device respectively.

[0064] Furthermore, the above-mentioned server can be hardware or software. When the server is hardware, it can be implemented as a distributed server cluster consisting of multiple servers, or it can be implemented as a single server. When the server is software, it can be implemented as multiple software or software modules, such as software or software modules used to provide distributed servers, or it can be implemented as a single software or software module, which is not specifically limited here.

[0065] like Figure 2As shown, the change detection sample generation method provided by the present application is mainly divided into three parts: input, instance positioning, and instance generation. The input is an original image. Instance positioning is to obtain the mask position information of the object to be detected in the image as a label for generating samples. Instance generation is the re-creation of the original image based on the mask position information and additional auxiliary information. The re-created image background semantic environment is consistent with the original image, and each specific instance will be different, which meets the requirements of change detection training data.

[0066] like Figure 1 As shown, the specific steps are as follows:

[0067] Step S11: Acquire the original image.

[0068] In an embodiment of the present application, the sample generating device may input an original image, and the original image may be any image stored in a storage medium, or may be any frame of video extracted from a video captured in real time by a video capture device.

[0069] Step S12: Input the preset target prompt information and the original image into the detection model to obtain the target detection frame information in the original image.

[0070] In the embodiment of the present application, the sample generation device combines the original image with the prompt information of the target to be detected, such as Figure 2 The detection prompt words shown in are input into the detection model to obtain the target detection box information in the original image.

[0071] In a specific implementation, the detection model of the present application can be adjusted according to specific input sample data. For example, if a public outdoor scene data set is used as the input sample, a trained model such as the YOLO (You Only Look Once) series can be directly loaded. If the input is a simulated remote sensing scene, a trained remote sensing scene detection network can also be selected. If the acquired samples have not been systematically trained, a general detection model such as Ground-Dino, GLEE (General Object Foundation Model for Images and Videos at Scale), etc. can be used, and a prompt word can be input to specify the content to be detected. The prompt word can also be changed according to the specific scene, such as a mobile phone, a computer or any object.

[0072] This application uses a detection model to identify multiple targets in the original image, which can identify untrained target types and greatly enhance the recognition ability and recognition range of the original image.

[0073] Step S13: Input the target detection box information and the original image into the segmentation model to obtain the target mask map.

[0074] In an embodiment of the present application, the sample generation device inputs the output result of the detection model, i.e., the target detection box information, as a prompt word and the original image into the segmentation model. The segmentation model is end-to-end, i.e., the entire process from the input image to the segmentation result is completed by one model.

[0075] The target detection frame information may be a target detection frame or coordinate information indicating the position of the target detection frame.

[0076] The segmentation model used in this application can also be a large segmentation model or a general segmentation model. Taking the large segmentation model as an example, the large segmentation model can learn complex and accurate feature representations after training with large-scale data sets, thereby achieving high-quality image segmentation.

[0077] The sample generation device can obtain the mask information corresponding to the target object on the original image through the segmentation model, such as the target mask map. ,The mask information output by the segmentation model is .

[0078] Step S14: Generate a sample image based on the change information of the target mask image at the target position of the original image.

[0079] In the embodiment of the present application, the sample generation device can distinguish the target position and the background position in the original image according to the target mask map, and then change the image content of the target position to generate change information. The sample generation device generates a sample image according to the change information and the background content corresponding to the background position.

[0080] Finally, the sample generation device will and As the true label of the generated sample, the output result of the generated model is used as image B to form an image pair with the original image A, and all the training elements of the change detection model can be obtained.

[0081] The methods of changing the image content at the target location include but are not limited to: Figure 2 In other embodiments, the image content at the target location may be changed by weak data enhancement, strong data enhancement, etc. The sample generation device may select any one or more content change methods to change the image content at the target location according to the requirements.

[0082] For details on how to generate diffusion, please refer to Figure 3 , Figure 3 yes Figure 1The flow chart of the first embodiment of the sample generation method step S14 is shown. The diffusion generation method is to use a probability generation model based on the diffusion process to gradually generate samples in the target distribution by introducing noise. In the diffusion generation model, the generation process is regarded as a gradual improvement process from the initial noise distribution to the target distribution.

[0083] like Figure 3 As shown, the specific steps are as follows:

[0084] Step S21: determining a changed region image of the target position in the original image based on the target mask image.

[0085] In an embodiment of the present application, the sample generating device determines the image content of the target position in the original image according to the target mask image.

[0086] Step S22: Generate Gaussian white noise corresponding to the prompt word based on the prompt word in the prompt library of the diffusion generation model.

[0087] In the embodiment of the present application, the sample generation device can use the Wensheng graph model based on the diffusion model as an implementation method for diffusion generation of samples. Figure 2 As shown, the sample generation device is input into the prompt library in the diffusion generation model The set of corresponding prompt words is expressed as follows:

[0088]

[0089] in, It is the change target prompt word, which can be adjusted according to the specific scene. For example, the remote sensing scene can be set to "water body", "vegetation", etc., and if it is an open scene, it can be set to natural objects such as "cat", "dog", "object", etc.

[0090] It should be noted that the diffusion generation model pre-trained in the present application can generate corresponding Gaussian white noise according to the input prompt word, and the Gaussian white noise is used to process the image content of the target position in the original image, thereby changing the image content of the target position in the original image.

[0091] The training process of this application for the diffusion generation model is as follows:

[0092] The forward diffusion training process of the diffusion generation model is regarded as a gradual improvement process from the initial noise distribution to the target distribution. This process is completed through multiple iterations. In each iteration, the model combines the samples generated by the previous iteration with the noise of the current iteration. Then, the reverse diffusion training process of the diffusion generation model transforms this combined data through a reversible diffusion process to gradually approach the target distribution. Through multiple iterations, the diffusion generation model can eventually generate samples that match the target distribution.

[0093] Specifically, each forward diffusion process is carried out as follows:

[0094]

[0095] Among them, t is the number of iterations. After 𝑇 iterations, the whole process transforms the image from Convert to Gaussian white noise, This factor changes with t for variance reduction.

[0096] Specifically, the inverse diffusion process is to restore the image by iterative denoising given a Gaussian white noise. Each inverse diffusion process is performed as follows:

[0097]

[0098] Among them, the process is predicted by neural network modeling Gaussian distribution parameters and After t iterations, the original image can be restored.

[0099] Step S23: Process the changed area image by diffusing the Gaussian white noise of the generation model to obtain change information.

[0100] In the embodiment of the present application, the sample generation device uses the diffusion generation model to perform noise processing on the change area image according to the Gaussian white noise determined by the prompt word, thereby changing the change area image and generating change information.

[0101] Step S24: Generate a sample image based on the change information and the original image.

[0102] In the embodiment of the present application, the sample generating device generates a sample image according to the change information and the background content corresponding to the background position.

[0103] For details, see Object Replacement. Figure 4 and Figure 5 , Figure 4 yes Figure 1 The flow chart of the second implementation of the sample generation method step S14 is shown, Figure 5It is a flowchart of the object replacement sample generation method provided by this application.

[0104] like Figure 4 As shown, the specific steps are as follows:

[0105] Step S31: Acquire a sample library, wherein the sample library includes a number of example images.

[0106] In the embodiments of the present application, Figure 5 As shown, the sample generation device imports a sample library, and the sample library includes several instance images. The sample library is generated as follows: the staff imports several annotated images, and the sample generation device can directly use the entire image of the annotated image as the instance image, or segment the annotated image according to the annotation content and use the segmented image as the instance image. The annotated image can be obtained from an existing public data set and must contain the instance object and its mask map, so that the generated sample will be more realistic and accurate while retaining a higher resolution.

[0107] Set the upper limit of samples in the sample library to (It can be adjusted according to computer performance). Each time an instance is randomly selected, the random queue of the sample library is dequeued, and the instance in the annotated image is loaded again for enqueuing, and the number of samples in the sample library is maintained at ,This method can save memory overhead while ensuring the diversity of generated samples.

[0108] Step S32: input any instance image in the sample library, the target mask image, and the original image into the object replacement model to obtain a sample image output by the object replacement model.

[0109] In the embodiment of the present application, the object replacement mode model of the present application is also based on the diffusion model. The only difference is the sample library of the replacement model. It is composed of images, which correspond to the objects you want to replace. Unlike text prompts, image prompts are more diverse. Use a random queue for storage:

[0110]

[0111] like Figure 5 As shown, the object replacement model randomly selects any instance image from the sample library to replace the changed area image of the target position in the original image, thereby generating a new sample image.

[0112] For more information on sample wiping methods, please refer to Figure 6 , Figure 6 yes Figure 1The schematic diagram of the flow chart of the third embodiment of the sample generation method step S14 is shown. The essence of the sample wiping method is to fill the object pixels in the detection area with the background through a neural network algorithm.

[0113] Step S41: Obtain a sample erasure model of the prediction process of the pre-trained erasure network.

[0114] Step S42: input the target mask image and the original image into the sample erasing model, so that the sample erasing model fills the target position in the original image with the image background according to the prediction process of the erasing network, and outputs the sample image.

[0115] In the embodiment of the present application, the previous process of the sample wiping model is performed by the following formula:

[0116]

[0117] in, is the original input image, m is the mask of the instance to be erased, In order to erase the prediction process of the network, the patent does not limit the specific network structure. A GAN-style restoration model consisting of a convolutional neural network and a fast Fourier convolutional network can be used, or a restoration model based on a diffusion-generated network architecture can be used.

[0118] The sample generation device of the present application obtains an original image; inputs the preset target prompt information and the original image into a detection model to obtain target detection frame information in the original image; inputs the target detection frame information and the original image into a segmentation model to obtain a target mask map; and generates a sample image based on the change information of the target mask map at the target position of the original image. The above sample generation method processes unlabeled images and automatically generates samples and annotations.

[0119] Based on the sample generation method of the above embodiment, the sample generation device can obtain a large number of sample images, and further quality screening of the generated sample images is required. The following is a detailed description of the automatic quality screening part of the generated data:

[0120] Please refer to Figure 7 and Figure 8 , Figure 7 is a flow chart of another embodiment of the sample generation method provided by the present application. Figure 8 It is a process framework diagram of the sample quality screening method provided in this application.

[0121] like Figure 7 As shown, the specific steps are as follows:

[0122] Step S51: input the sample image and the original image into the change detection model to obtain the detection mask image output by the change detection model.

[0123] In the embodiment of the present application, the sample generation device inputs image A and image B into the change detection model to obtain the pred output by the change detection model, that is, the detection mask map. Figure 1 Sample images generated by the sample generation method shown.

[0124] Furthermore, if Figure 8 As shown, the training process of the change detection model used in this application is as follows;

[0125] The sample generation device inputs the training image and the sample image generated according to the training image into the change detection model to be trained. Figure 1 The mask images obtained by the detection model and segmentation model shown are used as the real images for training iteration. After the sample generation device prepares the training set, the training set images are input into the change detection model for cyclic iteration, network training is performed, and then the optimal change detection model is selected for use.

[0126] Specifically, when the sample generation device prepares the training set, the training data is a mixture of the generated image, that is, the sample image and the existing labeled sample image, that is, the real image, and the mixing ratio is controlled to be close to 1:1. The generated data is divided into K training data sets according to the size of the generated data. The test data does not contain the generated samples and are all known real data.

[0127] The sample generation device repeatedly cross-trains the model based on K training sets, so that the real data is trained K times and the generated data is trained once. At the same time, a richer data enhancement method is set, including but not limited to: image scaling, flipping, rotation and random jitter, etc., to ensure that the network will not overfit the generated data.

[0128] Specifically, during the training process, the change detection model obtains a predicted output mask map based on the original image and the sample image, and compares the output mask map with the real annotated image to obtain the change detection result of the change detection model, which is represented by the loss value. The smaller the loss value, the smaller the gap between the output mask map and the real annotated image, indicating that the generated quality of the sample image is higher.

[0129] Step S52: input the detection mask image and the target mask image of the original image into a comparison control module to obtain a comparison result.

[0130] In the embodiment of the present application, the sample generation device inputs the detection mask image and the target mask image of the original image into the comparison control module to determine whether the sample image is available.

[0131] The basic logic of the comparison module uses pixel consistency judgment, because the bad images are basically caused by inaccurate segmentation of the segmentation model during instance positioning, or the detection logic of the model does not match the sample annotation. Such samples will not only fail to enhance the model's expressiveness, but will also mislead the network. Therefore, we use the following formula to calculate the mask score to determine whether the sample is usable. The calculation formula for the mask score is as follows:

[0132]

[0133]

[0134] Among them, the tolerance is set according to the size calculated by socer To screen the sample threshold, samples below the threshold are set as samples that do not meet the quality requirements, thus completing the screening of samples.

[0135] Step S53: In response to the comparison result that the sample image meets the quality requirement, the sample image is output as a target sample image.

[0136] In the embodiment of the present application, the entire sample generation process is completed, and only one image is needed to obtain multiple sample images that meet the change detection training material, and the detection information output by the detection model is labeled The mask information output by the segmentation model is .

[0137] In a specific implementation, due to the complexity of generating sample semantic information, this application provides a change detection model T-Change specifically for quality screening. For details of its algorithm structure, please refer to Fig. 9 , Fig. 9 It is a schematic diagram of the framework of an embodiment of the change detection algorithm structure provided by the present application.

[0138] like Fig. 9 As shown in the figure, the change detection model T-Change provided by this application adopts a twin network architecture, with TransformerEncoder as the feature extractor and SAF (spatial attention fusion) module as the feature fuser to train and predict the generated image. Before the network input, the image is scaled to a feature image of size W*H and then the feature is divided. The feature image of size W*H pixels is then divided into L*L feature image blocks, each of which is recorded as . Send the image block into the feature extraction structure to start network training and prediction.

[0139] Among them, regarding the above-mentioned TransformerEncoder feature extractor, Fig. 9 The feature extraction part of the network shown in the figure adopts the deep learning method. The main structural diagram of the feature extraction network is as follows: Fig.10 As shown, Fig.10 yes Fig. 9 A schematic diagram of the framework of the basic structure of feature extraction in the change detection algorithm structure shown.

[0140] like Fig.10 As shown, the feature map of the network input , where L*L is the number of blocks of the input image, and the vector obtained by adding the pixel information and position information of each image block , the calculation formula is as follows:

[0141]

[0142] in, is the feature map input, Sem and Pos are linear layer functions, the purpose is to Dimensionality reduction is performed to reduce the amount of calculation and improve the network operation speed. The dimension after dimensionality reduction is recorded as D (recommended value 512). is the position encoding mapping function, Used to mark the location information of image block pixels. Encodes information for the mapping dimension index, where The calculation formula is as follows:

[0143]

[0144]

[0145] In this application, the entire feature extraction module consists of N (schematically Fig.10 Where N is 3) feature extraction basic modules, the basic feature extraction basic module structure diagram is as follows Fig.10 As shown, it consists of autocorrelation calculation and linear layer. Autocorrelation calculation is a calculation method that uses the correlation of image information at different positions to model it. The specific calculation formula is as follows:

[0146]

[0147] in, , , Each autocorrelation calculation can use a different linear layer to extract different , , The results are concatenated in the feature dimension, the number of concatenations is n, and the concatenated results are finally input into the linear layer to maintain the consistency of the dimension. Each autocorrelation and linear layer module is connected by a skip layer structure, and the output result is added and normalized to the feature input and then input into the next autocorrelation operation. At this point, the feature extraction basic module is built, and N feature extraction basic modules are connected in a Fig.10 The basic feature extraction network is built serially as shown.

[0148] Furthermore, the SAF feature fusion module is responsible for processing network features at multiple levels. The module first uses the cat method to fuse the feature information of the corresponding levels, and then reduces the output feature dimension to . n is selected according to the specific category, such as when it changes, n is 2 (change, no change), when it appears and disappears, n is 3 (appear, disappear, no change), After post-processing, it is used as the output mask map of the network . Then the network uses the Cross-Entropy function as the loss function for training the network model. The calculation formula is as follows:

[0149]

[0150] At this point, the forward propagation process of the training part is completed. Then the optimization function is used to perform back propagation to repeatedly update the network parameters and complete the network training.

[0151] This application proposes a change detection sample generation method based on large model technology. This method can adjust model changes according to specific usage scenarios. Therefore, the proposed method can be used as a paradigm for amplifying change detection data using large model technology.

[0152] This application proposes a change detection model T-Change with an innovative architecture and its training strategy. The model has strong feature extraction and expression capabilities for big data algorithms and can meet the training requirements of massive data.

[0153] This application proposes a quality screening method for generated samples. This method, combined with the T-Change model, can significantly increase the expressive power of the change detection model and improve the detection accuracy and recall rate without any labeling manpower.

[0154] Those skilled in the art will appreciate that, in the above method of specific implementation, the order in which the steps are written does not imply a strict execution order and does not constitute any limitation on the implementation process. The specific execution order of the steps should be determined by their functions and possible internal logic.

[0155] In order to implement the above sample generation method, this application also proposes a sample generation device, please refer to Fig.11 , Fig.11 It is a structural schematic diagram of an embodiment of a sample generation device provided in the present application.

[0156] The sample generation device 400 of this embodiment includes a processor 41 , a memory 42 , an input / output device 43 , and a bus 44 .

[0157] The processor 41 , the memory 42 , and the input / output device 43 are respectively connected to the bus 44 . The memory 42 stores program data. The processor 41 is used to execute the program data to implement the sample generation method described in the above embodiment.

[0158] In the embodiment of the present application, the processor 41 may also be referred to as a CPU (Central Processing Unit). The processor 41 may be an integrated circuit chip having the ability to process signals. The processor 41 may also be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), or other programmable logic devices, discrete gates or transistor logic devices, or discrete hardware components. A general-purpose processor may be a microprocessor or the processor 41 may also be any conventional processor, etc.

[0159] This application also provides a computer storage medium, please continue to refer to Fig.12 , Fig.12 6 is a schematic diagram of the structure of an embodiment of a computer storage medium provided in the present application. The computer storage medium 600 stores a computer program 61. When the computer program 61 is executed by a processor, it is used to implement the sample generation method of the above embodiment.

[0160] When the embodiments of the present application are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) or a processor to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program code.

[0161] The above description is only an implementation method of the present application, and does not limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the present application specification and drawings, or directly or indirectly used in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A sample generation method, characterized in that: The sample generation method comprises: Get the original image; Inputting the preset target prompt information and the original image into the detection model to obtain the target detection frame information in the original image; Input the target detection frame information and the original image into a segmentation model to obtain a target mask map; Generate a sample image based on change information of the target mask image at the target position of the original image; After generating the sample image based on the change information of the target position of the original image based on the target mask image, the sample generation method further includes: Inputting the sample image and the original image into a change detection model to obtain a detection mask image output by the change detection model; Inputting the detection mask image and the target mask image of the original image into a comparison control module to obtain a comparison result; In response to the comparison result that the sample image meets the quality requirement, the sample image is output as a target sample image.

2. The sample generation method according to claim 1, characterized in that: The generating a sample image based on the change information of the target mask image at the target position of the original image includes: Determine a changed region image of a target position in the original image based on the target mask image; Generate Gaussian white noise corresponding to the prompt words in the prompt library based on the diffusion generation model; Processing the changed region image by using Gaussian white noise of the diffusion generation model to obtain the change information; The sample image is generated based on the change information and the original image.

3. The sample generation method according to claim 2, characterized in that: The sample generation method comprises: Inputting the first training image into the diffusion generation model to be trained, iterating multiple times according to the forward diffusion process, and generating a forward diffusion loss based on the predicted white noise and the preset white noise generated on the training image in each iteration; Inputting the second training image into the diffusion generation model to be trained, iterating multiple times according to the reverse diffusion process, and generating reverse diffusion loss based on the restored image and the target image obtained in each iteration; The diffusion generation model to be trained is trained based on the forward diffusion loss and the reverse diffusion loss.

4. The sample generation method according to claim 1, characterized in that: The generating a sample image based on the change information of the target mask image at the target position of the original image includes: Acquire a sample library, wherein the sample library includes a plurality of example images; Any instance image in the sample library, the target mask image, and the original image are input into the object substitution model to obtain a sample image output by the object substitution model.

5. The sample generation method according to claim 1, characterized in that: The generating a sample image based on the change information of the target mask image at the target position of the original image includes: Get a sample erasure model of the prediction process of the pre-trained erasure network; The target mask map and the original image are input into the sample erasure model, so that the sample erasure model fills the target position in the original image with the image background according to the prediction process of the erasure network, and outputs the sample image.

6. The sample generation method according to claim 1, characterized in that: The step of inputting the detection mask image and the target mask image of the original image into a comparison control module to obtain a comparison result includes: Inputting the detection mask image and the target mask image of the original image into a comparison control module, and calculating the number of pixels having the same pixel value in the detection mask image and the target mask image; Based on the number of pixels, obtaining a mask score of the sample image; When the mask score is greater than or equal to a preset threshold, a comparison result indicating that the sample image meets the quality requirement is output.

7. The sample generation method according to claim 1, characterized in that: The sample generation method further includes: Acquire a training image and a generated image generated based on the training image, and a real annotated image corresponding to the training image; Based on the generated images and the real annotated images, forming several training sets; Inputting the generated image in each training set and the training image into the change detection model to be trained to obtain a predicted output mask image; Calculating a training loss value based on the output mask image and the true annotated image; The change detection model to be trained is trained based on the training loss value.

8. The sample generation method according to claim 7, characterized in that: The number of the generated images is greater than the number of the real annotated images, the generated images and the real annotated images form the training set according to a preset mixing ratio, and each real annotated image is trained more times than each generated image.

9. A sample generating device, characterized in that: The sample generating device includes a memory and a processor coupled to the memory; The memory is used to store program data, and the processor is used to execute the program data to implement the sample generation method according to any one of claims 1 to 8.

10. A computer storage medium, characterized in that: The computer storage medium is used to store program data, and when the program data is executed by a computer, it is used to implement the sample generation method according to any one of claims 1 to 8.