An image sample generation method based on key frame point detection

Abstract

The application discloses an image sample generation method based on key frame point detection and belongs to the field of computer vision. The method uses Gaussian random noise as input, generates a key frame point generation network, presets a target generation position and embedding of a rectangle, uses Mahalanobis-Gaussian heatmap as an initial mask, combines a Swin-Transformer-SPADE generator to generate a mask for controlling the target generation position and size, generates a foreground and a background of a sample image respectively, and finally generates the sample image. Controllable generation refers to the ability to directly decouple the object coordinate information in the image, decompose the image into the position and appearance of the object, and generate the image by repositioning and exchanging key points to rearrange the image. Labelled generation refers to the ability to directly save the key frame position as target position information. The labelled image samples in the few-sample database are supplemented, the overfitting problem of a target detection model caused by the lack of samples of a target detection network is relieved, and the target detection accuracy is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to an image sample generation method for target detection expansion, in particular to a labeled and editable target detection sample generation method based on key frame detection, and belongs to the field of computer vision. BACKGROUND

[0002] Target detection is an important research content in the field of computer. It has wide applications in many fields, such as video surveillance, human-computer interaction, and unmanned driving. Through the use of deep learning and computer vision technology, target detection can accurately detect and locate target objects in images, thereby realizing automated and intelligent solutions. This not only improves the efficiency and safety of various fields, but also promotes the continuous progress of technology and the sustainable development of society.

[0003] Deep learning has great advantages in target detection, but it needs a large amount of sample data to drive. Under the condition of a certain network structure, the more sample data, the stronger the detection ability of the trained model. However, in practical applications, especially in some special fields, it is difficult to obtain a large amount of sample data, or a large amount of manpower, material resources and financial resources are needed to obtain a large amount of data samples, which makes the endless demand for large data samples contradict the actual acquisition amount that is difficult to meet the requirements. This makes the small sample problem a hot and difficult point in the application of target detection and tracking. When the number of samples is too small, the model is prone to overfitting, and the generalization ability of the trained model is poor, which will directly affect the performance of the final detection network. The essence of overfitting is the mismatch between the number of samples and the complexity of the model. Therefore, the problem of preventing overfitting is actually the problem of preventing the mismatch between the number of samples and the complexity of the model.

[0004] Sample enhancement algorithm is a technology for improving the performance of deep learning-based target detection and tracking algorithms. It increases the diversity and quantity of training data by performing a series of transformations and distortions on the original image. These operations include image rotation, scaling, translation, mirror flipping, brightness adjustment, color transformation, and adding noise, etc. By randomly combining and transforming the original image, the sample enhancement algorithm can generate diverse new samples, providing more images under different angles, scales, lighting conditions and background environments.

[0005] As an effective way of sample enhancement, generating image samples is the most basic application of GAN. For well-constrained fields such as MNIST handwritten data set, CIFAR-10 small picture, LSUN Bedroom data set and CelebA face data set, new sample generation with photo-realism has been achieved by using StyleGAN, SPADE algorithm and the like. StyleGAN can unsupervisedly separate high-level attributes, such as expression during training on a face, identity features and random changes in generated images (such as freckles and hair). However, even if different parts can be adjusted, the adjustments are uncontrollable. The SPADE algorithm can generate images that meet expectations according to input conditional information (such as semantic labels) while generating realistic images. This makes SPADE have great potential in generating images of various styles, categories and image editing, making the image generation process more controllable and flexible. However, semantic labels or categories as semantic information are not easy to obtain. And when the image faces the problems of occlusion, deformation, lighting conditions and background environment changes in target detection, the generated samples not only need to ensure diversity, but also need to ensure the practicality of the generated data. However, controllable image generation and image labeling are still obstacles to the application of GAN in target detection tasks. Therefore, making such image generation have the characteristics of being editable and self-labeled can better meet the needs of sample supplement in target detection. SUMMARY

[0006] In order to solve the problem that the sample cannot be controllably generated and cannot be labeled in the image sample generation based on the deep learning of the target detection task at present. The purpose of the present application is to provide an image sample generation method based on key frame detection, which can effectively supplement the image samples required for target detection model training, and has the advantages of controllable generation and labeled generation. The controllable generation refers to the ability to directly decouple the object coordinate information in the image, decompose the image into the position and appearance of the object, facilitate the spatial position editing of the image sample for the purpose of target detection, and generate the image by repositioning and exchanging key points. The labeled generation refers to the ability to directly save the key frame position as the target position information, avoiding the manual labeling process. The labeled image samples in the small sample database are supplemented, the problem of overfitting of the target detection model caused by the lack of samples of the target detection network is alleviated, and the target detection accuracy is improved.

[0007] The purpose of the present application is achieved by the following technical solutions:

[0008] The image sample generation method based on key frame point detection disclosed in the application is a labeled and editable target detection sample generation method based on key frame point detection, utilizes an independent noise vector, generates K key point positions and key frame width and height through a key frame point generation network, specifies the coordinate positions of the four points of a rectangular key frame and the appearance corresponding to the key points (calculates the size of the specified part of each key frame point according to the half diagonal length of the rectangle surrounded by the key frame points), models the local part using a Mahalanobis-Gaussian heatmap and a SPADE generator, combines the key point coordinate positions and the half diagonal length of the rectangle surrounded by the key frame points, generates a mask corresponding to the key point positions, and generates a sample image using a SwinTransformer-SPADE generator based on the generated mask and embedding. The discriminator constantly assists the image generation network to generate more realistic image samples by comparing the generated image and the real image, and the image generation network is composed of a key frame point generation network, a mask generation network, a foreground generation network and a background generation network. The image generation model obtained through training can generate editable and labeled image samples. On the basis of generating sample images, the positions of the corresponding objects are manipulated by moving the positions of the key points, and the characteristics of the objects are changed by modifying the embedding. At the same time, the positions of the key frame points directly correspond to the generated positions of the objects, so that subsequent labeling work is not needed, the labeled image samples in the small sample database are supplemented, the overfitting problem of the target detection model caused by the lack of samples is relieved, and the target detection accuracy is improved.

[0009] The image sample generation method based on key frame point detection disclosed in the application comprises the following steps:

[0010] Step 0: Make a target detection image sample dataset, and perform image preprocessing on the pictures in the dataset to achieve the purpose of data enhancement. The image preprocessing includes random horizontal rotation, cropping, random region cropping, scaling and standardization.

[0011] An input data set containing real images is preprocessed: randomly horizontally rotated, cropped to the same size of a square, randomly cropped a region and scaled to the size of the aforementioned square, standardized, and the standardized image is obtained. The above operations can achieve the effect of data enhancement, reduce the overfitting problem of the image generation model training, and improve the model generalization ability.

[0012] Step 1: Generate K keypoint positions by keypoint generation network using independent noise vectors, each keypoint position corresponds to a keybox width and keybox width height, determine the coordinate position of the four points of the rectangular keybox according to each keypoint position, keybox width and keybox width height. Determine the scale of each keypoint designated component according to the half diagonal length of the rectangle surrounded by each keypoint. Generate a random appearance vector for each component using the keypoint generation network, and multiply the fixed embedding vector to obtain the appearance embedding of each component.

[0013] The keypoint generation network generates the positions and appearances of K components by using independent noise vectors. In order to prevent the position and appearance information from interfering with each other, three groups of independent noise vectors As input, use multi-layer perception MLP to generate independent noise z keypoint and z WH Generate K keypoint and K groups of width W, height H. The network uses MLP, and each hidden layer is a full connection layer and uses LeakyReLU as the activation function. The output layer still uses a full connection layer to control the output dimension. Then the position of the keybox point corresponding to each keypoint is The keybox point is specified as a rectangle, and the keypoint is the center point of the keybox point position. The position of the 4xK keybox points is In addition, the component scale is represented as {σ1,...,σ K}, then,

[0014]

[0015] where k = 1,...,K, (x row ,x col ) represents the horizontal and vertical coordinates of each keypoint coordinate.

[0016] Map z app to component appearance vector Define a fixed embedding vector for each component. The fixed embedding is randomly generated using nn.Embeddings, which encodes the semantic information of the keypoint and is continuously updated during the training phase. Element-wise multiplication with the component embedding

[0017] w dynamic = MLP app (z app )

[0018]

[0019] The above appearance and position noise sources are independent of each other.

[0020] Step 2: According to the key frame point coordinate position and component size determined in step 1, use Mahalanobis-Gaussian heatmap to calculate the initial mask of each component, and multiply it with the corresponding component embedding to obtain the component embedding graph, add all K component embedding graphs to form a single embedding graph, and use the SPADE generator to generate a mask corresponding to the predicted component position in combination with the initial position encoding.

[0021] The present application does not use semantic labels, i.e. semantic segmentation map, so it is necessary to generate the mask required by the SPADE generator to generate an image based on Mahalanobis-Gaussian heatmap and initial position encoding, which uses relative position encoding. Among them, Mahalanobis-Gaussian heatmap is used to model the local independence of components, and relative position encoding is used to maintain the distance relationship between components.

[0022] According to the key frame point coordinate position generated in step 1, use Mahalanobis-Gaussian heatmap to model local components to calculate the initial mask of each component, and use the component size determined in step 1 to control the size of the initial mask. The initial mask H k Multiply it with the corresponding component embedding to obtain the component embedding graph, and add all K component embedding graphs to form a single embedding graph Each component embedding graph is centered on the key point position x k , and d controls the Gaussian value calculated at each pixel position, while σ k controls the range of action, and each component initial mask presents a Gaussian shape.

[0023]

[0024] d(p,x k )=(p-x k ) T ∑ -1 (p-x k )

[0025]

[0026] Where ∑ -1 is the inverse matrix of the covariance matrix between sample variables.

[0027] As another input of the SPADE generator to generate a mask, the initial position encoding of the mask In order to avoid leaking absolute position information, use the low-frequency position encoding of the difference between the pixel position and the 4xK points:

[0028]

[0029] The avoiding of leaking absolute position information means that using absolute position encoding position information can cause the model to over-remember specific positions in the training data, thereby limiting the generalization ability when applied to new data. Relative position encoding better assists the image generation network to capture the relationship and relative arrangement between objects, thereby not relying on absolute position information.

[0030] To improve the ability of SPADE generator to understand the global structure of image context information and feature representation, and to improve the quality of image generation, SwinTransformer Block is introduced in SPADE ResBlock, and the generation process of mask M = [M bg ,M1,...,M K ] is represented as:

[0031] M (i) = SPADE ResBlock(Swin Transformer Block(M (i-1) ),W mask )

[0032]

[0033] i = 1,...,T mask

[0034] Where M bg is the background mask, and T mask is the number of SPADE ResBlock. In addition, the generation of the background mask uses a separate channel.

[0035] Step 3: Perform initial position encoding according to the key point positions obtained in step 1, multiply the corresponding part embedding to obtain each part embedding map, add each part embedding map to obtain a single foreground embedding map, and use the SPADE generator to generate the image sample foreground in combination with the mask obtained in step 2. Use independent noise to generate background embedding, and use AdaIN ConvBlock to generate sample background in combination with background initial position encoding. Linearly mix the foreground and background to generate a sample image.

[0036] According to the mask M = [M bg ,M1,...,M K ] generated in step 2, the generation of foreground embedding is similar to the generation of mask embedding W mask , and the initial position encoding and foreground embedding are also used to generate foreground feature map

[0037]

[0038] F (0)(p) = [sin(πFC([p - xi,..., p - x K ])),

[0039] cos(πFC([p - xi,..., p - x K ]))]

[0040] F (i) = SPADEResBlock(Swin Transformer Block(F (i-1) ), W fg )

[0041]

[0042] where i = 1,..., T fg is the number of SPADEResBlock.

[0043] For the generation of background, two sets of noise and u bg_pos directly correspond to the center position of the background, and the background embedding is generated using MLP:

[0044] W bg = MLP bg_app (z bg_app )

[0045] Similarly, the initial position encoding and the background center position u bg_pos are used to generate the background feature map

[0046] B (0) (p) = [sin(πFC([p - u bg_pos ])), cos(πFC([p - u bg_pos ]))]

[0047] B (i) = AdaINConvBlock(B (i-1) , W bg )

[0048]

[0049] where i = 1,..., T fg is the number of AdaIN ConvBlock.

[0050] The foreground and background feature maps are combined and input into two layers of CNN network to generate the final image:

[0051]

[0052] Step 4: Build a discriminator network for discriminating real images and images generated by the image generation network. Discriminate whether the image is a real image through the discriminator network. The discriminator network is composed of multiple DiscriminatorBlock, and DiscriminatorBlock is composed of a series of convolutional layers and LeakyReLU activation functions. Discriminate whether the input image is real through the fully connected layer.

[0053] During the training of the discriminator network, the real images preprocessed in step 0 are input into the discriminator network according to the batch size to train the discriminator network. The images generated by the image generation network and the real images are input into the discriminator network to calculate the discrimination result and obtain the discrimination loss. The discrimination loss is backpropagated to calculate the gradient, and the discrimination loss value is accumulated. The discriminator network parameters are updated. Train the generation network, input the real images preprocessed in step 0 and the images generated by the image generation network into the discriminator network to calculate the discrimination result and obtain the generation loss. The generation loss is backpropagated to calculate the gradient, and the generation loss value is accumulated. The discriminator compares the generated image and the real image to constantly assist the image generation network to generate more realistic image samples, and the image generation network is composed of a key frame point generation network, a mask generation network, a foreground generation network and a background generation network. Stop training after reaching the specified training round to obtain the trained image generation model, and the image generation model is realized based on the image generation network.

[0054] The total loss function L is summarized as:

[0055]

[0056] Where, represents the generator, represents the discriminator; x represents the sample obtained by sampling from the real data distribution p data ; z is random noise input from the noise space N.

[0057] Step 5: Use the image generation model trained in step 4 to input noise to generate image samples. Based on the key point position and the corresponding embedding vector, keep the embedding vector unchanged, move the key point position to generate images, obtain the image samples before and after the object position moves, and save the position of the corresponding key frame; keep the key point position unchanged, exchange the embedding vectors of each object, obtain the image samples before and after the object appearance editing, and save the key frame position to obtain the labeled image samples. Supplement the labeled image samples in the small sample database to alleviate the overfitting problem of the target detection model caused by the lack of samples, and improve the target detection accuracy.

[0058] Beneficial effects:

[0059] 1. For the sample demand of training target detection network, the image sample generation method based on key frame point detection disclosed by the application uses Gaussian random noise as input, generates a key frame point through a key frame point generation network, first presets the target generation position and embedding of a rectangle, uses Mahalanobis-Gaussian heatmap as an initial mask, combines a Swin-Transformer-SPADE generator to first generate a mask for controlling the target generation position and size, then further generates a foreground and a background for generating a sample image respectively, and finally generates a sample image. This method not only can supplement the number of samples, but also can decouple the target position and appearance information in the image generation process, can realize intuitive editing, and can supplement the diversity of samples.

[0060] 2. The image sample generation method based on key frame point detection disclosed by the application, the rectangle surrounded by the key frame point obtained by the key frame point generation network directly corresponds to the position of each object in the generated image sample, can replace the target bounding box label in the target detection data set, and avoids manual labeling after image generation.

[0061] 3. The image sample generation method based on key frame point detection disclosed by the application, Mahalanobis-Gaussian heatmap makes the generation of the initial mask more accurate: the Mahalanobis distance considers the correlation between features, has scale invariance, and is beneficial to modeling each part of the object in the image.

[0062] 4. The image sample generation method based on key frame point detection disclosed by the application, Swin-Transformer-SPADE is beneficial to improving the long-range dependency capturing ability of the image generator, that is, helping the generator to understand the global structure and context relationship of the image; further enhancing the spatial adaptive performance of the generator; improving the image generation quality and generating more image details.

[0063] 5. The image sample generation method based on key frame point detection disclosed by the application, in the SPADE ResBlock, the global context information refers to the mutual relationship and overall structure between each part in the image, by understanding the context information to realize deeper representation of objects, scenes and relationships in the image, improve the understanding ability and feature representation ability of the global structure context information of the image, and improve the image generation quality. BRIEF DESCRIPTION OF DRAWINGS

[0064] Figure 1 It is a network structure schematic diagram of the image sample generation method based on key frame point detection;

[0065] Figure 2 It is a network structure schematic diagram of the key frame point generation network;

[0066] Figure 3 It is a network structure schematic diagram of the mask generation network;

[0067] Figure 4 Fig. 1 is a structural schematic diagram of a SwinTransformer-SPADE network structure;

[0068] Figure 5 Fig. 3 is a structural schematic diagram of a foreground generation network structure;

[0069] Figure 6 Fig. 4 is a structural schematic diagram of a background generation network structure;

[0070] Figure 7 Fig. 5 is a network training flowchart of an image sample generation method based on key frame point detection;

[0071] Figure 8 Fig. 6 is a network test flowchart of an image sample generation method based on key frame point detection;

[0072] Figure 9 Fig. 7 is a training image example (320x240);

[0073] Figure 10 Fig. 8 is an image generation result example (128x128);

[0074] Figure 11 Fig. 9 is an image editing result example;

[0075] Figure 12 Fig. 10 is a key point and key frame point example. Figure 12 (a) A generated vehicle with key point labeling; Figure 12 (b) A generated vehicle with key frame point labeling DETAILED DESCRIPTION

[0076] In order to better illustrate the purpose and points of the present application, the content of the application is further described below in combination with the drawings and examples.

[0077] Example 1:

[0078] As Figure 1As shown, the image sample generation method based on key frame point detection disclosed in this embodiment is a labeled and editable target detection sample generation method based on key frame point detection. First, (three sets of) independent Gaussian noise vectors are used to generate K key point positions and key frame width and height through a key frame point generation network, finally specifying the coordinate positions of the four points of the rectangular key frame and the appearance corresponding to the key points (calculating the size of the specified part of each key frame point according to the half diagonal length of the rectangle surrounded by the key frame points); then, the local part is modeled using Mahalanobis-Gaussian heatmap and SPADE generator, and the mask corresponding to the key point position is generated in combination with the key point coordinate position and the half diagonal length of the rectangle surrounded by the key frame points; finally, the sample image is generated based on the generated mask and embedding using the SwinTransformer-SPADE generator. The discriminator constantly guides the aforementioned network to generate more realistic image samples by comparing the generated image and the real image. On the basis of the generated sample image, the position of the corresponding object is manipulated by moving the position of the key point, and the object features are changed by modifying the embedding; at the same time, the position of the key frame point directly corresponds to the generated position of the object, so there is no need for subsequent labeling work. The network structure diagram of this embodiment is shown in the accompanying Figure 1

[0079] The training sample expansion method of the target detection system for intelligent driving, unmanned aerial vehicle application, security monitoring, human-computer interaction, etc. disclosed in this embodiment. The target detection system needs to accurately detect the target in the dynamic scene, but it is very time-consuming and expensive to obtain a large number of labeled video image samples in the real scene, so data expansion technology needs to be used to increase the training samples. Through sample expansion, the diversity of sample data can be increased, occlusion and background interference can be handled, illumination changes can be handled, etc., and finally the performance and robustness of the target detection model can be improved. In order to ensure the tracking ability of the video monitoring system, the anti-occlusion target tracking method disclosed in this embodiment specifically includes the following steps:

[0080] Step 0: Make a target detection image sample dataset, and perform image preprocessing on the pictures in the dataset to achieve the purpose of data enhancement. The image preprocessing includes random horizontal rotation, cropping, random region cropping, scaling and standardization.

[0081] ​Taking generating a 128x128 image as an example, an input data set containing real images is taken, and the images in the data set are preprocessed (taking an image size of 320x240 as an example): random horizontal rotation (probability p is 0.5), cropping to a square of 128x128, randomly cropping a region and scaling to 128x128 (scale=(0.9, 1.0), ratio=(0.9, 1, 1)), standardization processing (mean=(0.5, 0.5, 0.5), std=(0.5, 0.5, 0.5)), and obtaining the image after standardization processing. The above operations can achieve the effect of data augmentation, alleviate the overfitting problem of image generation model training, and improve the generalization ability of the model.

[0082] Step 1: Use an independent noise vector to generate K key point positions through a key frame point generation network, each key point position corresponds to a key frame width and a key frame width height, and the coordinates of the four points of the rectangular key frame are determined according to each key point position, the key frame width and the key frame width height. The scale of each key point specified component is determined according to the half diagonal length of the rectangle surrounded by each key frame point. A random appearance vector of each component is generated by using the key frame point generation network, and the appearance embedding of each component is obtained by dot product of the appearance vector and the fixed embedding vector.

[0083] The key frame point generation network generates the positions and appearances of K components by using independent noise vectors. In order to prevent the position and appearance information from interfering with each other, three groups of independent noise vectors As input, D noise = 256, the independent noise z keypoint and z WH are used to generate K key points and K = 8 groups of width W, height H. The network uses 5 layers of MLP, each hidden layer is a fully connected layer and uses LeakyReLU as the activation function, and the output layer still uses a fully connected layer to control the output dimension. Then the position of the key frame point corresponding to each key point is The specified key frame point is a rectangle, the key point is the center point of the key frame point position, and the positions of the 4xK (=8) key frame points are The structure diagram of the key frame point generation network is shown in the accompanying Figure 2 In addition, the component scale is represented as {σ1,...,σ K}, and the component scale is shown in Figure 3 , then,

[0084]

[0085] where k = 1,..., K, (x row , x col) represents the horizontal and vertical coordinates of each key point coordinate.

[0086] z is mapped to component appearance vector using MLP app A fixed embedding vector is defined for each component The fixed embedding is randomly generated using nn.Embeddings, which encodes the semantic information of the key points and is continuously updated during the training phase. The component embedding obtained by element-wise multiplication with the key point position x

[0087] w dynamic = MLP app (z app )

[0088]

[0089] The above appearance and position noise sources are independent of each other.

[0090] Step 2: According to the key frame point coordinate position and component size determined in step 1, use Mahalanobis-Gaussian heatmap to calculate the initial mask of each component, and multiply it with the corresponding component embedding to obtain the component embedding graph, and then add the single mask embedding graph, and use SPADE generator to generate the mask corresponding to the predicted component position combined with the initial position encoding.

[0091] The present application does not use semantic labels, i.e. semantic segmentation map, so it is necessary to generate the mask required by SPADE generator to generate image based on Mahalanobis-Gaussian heatmap and initial position encoding, which uses relative position encoding. The mask generation network structure diagram is shown in Figure 3 , where Mahalanobis-Gaussian heatmap is used to model the local independence of components, and relative position encoding is used to maintain the distance relationship between components.

[0092] According to the key frame point coordinate position generated in step 1, use Mahalanobis-Gaussian heatmap to model the local component to calculate the initial mask of each component, and use the component size determined in step 1 to control the size of the initial mask. The initial mask H k is multiplied by the corresponding component embedding to obtain the component embedding graph, and the embeddings of all K=8 components are added to form a single embedding graph Each component embedding graph is centered at the key point position x k , and d controls the Gaussian value calculated at each pixel position, while σ k controls the range of action, and each component initial mask presents a Gaussian shape.

[0093]

[0094] d(p,x​k ) = (p - x k ) T ∑ -1 (p - x k )

[0095]

[0096] where ∑ -1 is the inverse of the covariance matrix of the variables in the sample.

[0097] As another input to the SPADE generator to generate the mask, the initial positional encoding of the mask To avoid leaking absolute position information, low-frequency positional encoding is used between the pixel position and the 4xK(=8) points:

[0098]

[0099] The avoidance of leaking absolute position information means that using absolute position encoding position information may cause the model to over-remember the specific positions in the training data, thereby limiting the generalization ability when applied to new data. Relative position encoding better assists the image generation network to capture the relationship and relative arrangement between objects, thereby not relying on absolute position information.

[0100] To improve the ability of SPADE generator to understand the global structure of the image context information and the feature representation ability, and improve the image generation quality, Swin Transformer Block is introduced in SPADE ResBlock, and the structure diagram of the proposed SwinTransformer-SPADE network is shown in FIG. 1. Figure 4 To reduce the amount of calculation, here only the SwinTransformerBlock is placed in the second SPADE module of the third SPADE ResBlock. Then the generation process of the mask M=[M bg ,M1,...,M K ] is represented as:

[0101] M (i) = SPADE ResBlock(Swin Transformer Block(M (i-1) ),W mask )

[0102]

[0103] i = 1,...,T mask

[0104] where M bg is the background mask, T maskThe number of SPADE ResBlocks. Here T mask is set to 4. In addition, the generation of the background mask uses a separate channel.

[0105] Step 3: Perform initial position encoding on the keypoint positions obtained in step 1, multiply the corresponding part embedding to obtain each part embedding map, add each part embedding map to obtain a single foreground embedding map, and use the SPADE generator to generate the foreground of the image sample using the mask obtained in step 2. Use independent noise to generate a background embedding, and use AdaIN ConvBlock to generate a sample background combined with the background initial position encoding. Linearly mix the foreground and background to generate a sample image.

[0106] According to the mask M = [M bg , M1,..., M K ] generated in step 2, the generation of the foreground embedding is similar to the generation of the mask embedding W mask , and the foreground feature map

[0107]

[0108] F (0) (p) = [sin(πFC([p-x1,...,p-x K ]),

[0109] cos(πFC([p-x1,...,p-x K ]))]

[0110] F (i) = SPADE ResBlock(Swin Transformer Block(F (i-1) ), W fg )

[0111]

[0112] where i = 1,..., T fg , the number of SPADE ResBlocks, which is set to 4 here.

[0113] The generation of the background, then needs to use two groups of noise and u bg_pos directly corresponds to the background center position, and a 3-layer MLP is used to generate a background embedding:

[0114] W bg = MLP bg_app (z bg_app )

[0115] Similarly, the initial position encoding and the background center position u bg_pos Generate background feature map

[0116] B (0) (p) = [sin(pi*FC([p-u bg_pos ])), cos(pi*FC([p-u bg_pos ]))]

[0117] B (i) = AdaINConvBlock(B (i-1) , W bg )

[0118]

[0119] Where i = 1,...,T fg , is the number of AdaIN ConvBlock, which is set to 4 here.

[0120] Combine the foreground and background feature maps into two layers of CNN network to generate the final image:

[0121]

[0122] Step 4: Build a discriminator network for discriminating real images and images generated by the image generation network. Discriminate whether the image is real or not through the discriminator network. The discriminator network is composed of multiple DiscriminatorBlock, and DiscriminatorBlock is composed of a series of convolutional layers and LeakyReLU activation functions. Discriminate whether the input image is real or not through the fully connected layer.

[0123] When training the discriminator network, according to batch size = 32, input the real image preprocessed in step 0 into the discriminator network to train the discriminator network. The learning rate of the discriminator network is set to 4x10 -4 -4. Input the image generated by the image generation network and the real image into the discriminator network to calculate the discrimination result and get the discrimination loss. Backpropagate the discrimination loss, calculate the gradient, accumulate the discrimination loss value, and update the discriminator network parameters. Train the generation network, and the learning rate of the generation network is set to 1x10 -4The real image and the image generated by the image generation network after the preprocessing of step 0 are input into the discriminator network to calculate the discrimination result, and the generation loss is obtained. The generation loss is back propagated to calculate the gradient, and the generation loss value is accumulated. The discriminator constantly assists the image generation network to generate more real image samples by comparing the generated image and the real image, and the image generation network is composed of the key frame point generation network, the mask generation network, the foreground generation network and the background generation network. After reaching the specified training round, the training is stopped, and the trained image generation model is obtained, and the image generation model is realized based on the image generation network. The network training flow chart is shown in the accompanying Figure 7

[0124] The total loss function L is summarized as:

[0125]

[0126] Wherein, represents the generator, represents the discriminator. Wherein, represents the generator, represents the discriminator; x represents the sample obtained by sampling from the real data distribution p data ; z is random noise input from the noise space N.

[0127] Step 5: using the image generation model trained in step 4, inputting noise to generate image samples. The network test flow chart is shown in the accompanying Figure 8 Based on the key point position and the corresponding embedding vector, the embedding vector is kept unchanged, the key point position is moved to generate the image, the image samples before and after the object position is moved are obtained, and the position of the corresponding key frame is saved; the key point position is kept unchanged, and the embedding vectors of each object are exchanged to obtain the image samples before and after the object appearance editing, and the key frame position is saved to obtain the labeled image samples. The labeled image samples in the small sample database are supplemented to alleviate the overfitting problem of the target detection model caused by the lack of samples of the target detection network, and the target detection accuracy is improved.

[0128] Experimental conditions

[0129] The labeled and editable target detection sample generation method based on key frame point detection disclosed in this embodiment is trained and tested using the Pytorch deep learning framework in the environment of Intel(R) Xeon(R) Silver 4214 CPU 2.20GHz and Nvidia RTX 2080Ti GPU.

[0130] Experimental results

[0131] The training image example of the application is shown in the accompanying Figure 9 , and the generated image example is shown in the accompanying​Figure 10 The image editing example is shown in the attached Figure 11 As shown in the attached Figure 12 As shown in the attached Figure 12 (b) The key point frame point can be directly used as the label of the image sample, so as to realize the labeled generation.

[0132] The above specific description further describes the purpose, technical scheme and beneficial effects of the application. It should be understood that the above description is only a specific embodiment of the application and is not used to limit the protection scope of the application. Any modification, equivalent replacement, improvement, etc. within the spirit and principles of the application shall be included in the protection scope of the application.

Claims

1. An image sample generation method based on key frame detection, characterized in that: Includes the following steps, Step 0: Create a dataset of object detection image samples, and perform image preprocessing on the images in the dataset to achieve data augmentation. The image preprocessing includes random horizontal rotation, cropping, random region cropping, scaling, and normalization; Input a dataset containing real images, preprocess the images in the dataset: randomly rotate them horizontally, crop them to squares of the same size, randomly crop a region and scale it to the size of the aforementioned square, and perform standardization to obtain the standardized image; Step 1: Using independent noise vectors, generate K keypoint locations through a keyframe generation network. Each keypoint location corresponds to a keyframe width and keyframe height. Determine the coordinate positions of the four points of a rectangular keyframe based on each keypoint location, keyframe width, and keyframe height. Determine the scale of the component specified by each keypoint based on the length of the half-diagonal of the rectangle enclosed by each keyframe. Generate a random appearance vector for each component using the keyframe generation network. Multiply this appearance vector by a fixed embedding vector to obtain the appearance embedding of each component. Step 2: Based on the key frame coordinates and component dimensions determined in Step 1, calculate the initial mask for each component using the Markov-Gaussian heatmap, and multiply it with the corresponding component embedding to obtain the component embedding map. Add the component embedding maps to obtain a single mask embedding map. Combine the initial position encoding with the SPADE generator to generate a mask corresponding to the predicted component position. Step 3: Perform initial position encoding based on the key point positions obtained in Step 1, multiply with the corresponding component embedding to obtain the component embedding map, add the component embedding maps to obtain a single foreground embedding map, and use the SPADE generator to generate image sample foreground using the mask obtained in Step 2. Background embeddings are generated using independent noise, and sample backgrounds are generated using AdaINConvBlock in conjunction with initial background location encoding; foreground and background are linearly blended to generate sample images. Step 4: Build a discriminant network to distinguish between real images and images generated by the image generation network, and use the discriminant network to determine whether an image is a real image; The discriminator network consists of multiple DiscriminatorBlocks, each composed of a series of convolutional layers and LeakyReLU activation functions; it uses fully connected layers to determine whether the input image is real or not. During the training of the discriminator network, the real images after preprocessing in step 0 are input into the discriminator network according to the batch size to train the discriminator network. The images generated by the image generation network and the real images are input into the discriminator network respectively to calculate the discrimination result and obtain the discrimination loss. The discriminant loss is backpropagated to calculate the gradient and accumulate the discriminant loss value. And update the discrimination network parameters; The generator network is trained by inputting the real image after preprocessing in step 0 and the image generated by the image generator into the discriminator network to calculate the discrimination result and obtain the generation loss. The generation loss is backpropagated to calculate the gradient and accumulate the generation loss value. The discriminator continuously assists the image generation network in generating more realistic image samples by comparing generated images with real images. The image generation network consists of a keyframe generation network, a mask generation network, a foreground generation network, and a background generation network. Training stops after a specified number of training rounds, resulting in a trained image generation model. The image generation model is implemented based on the image generation network. Step 5: Using the image generation model trained in Step 4, input noise to generate image samples; based on the obtained key point positions and corresponding embedding vectors, keep the embedding vectors unchanged, move the key point positions to generate images, obtain image samples before and after the object position is moved, and save the corresponding key box positions; keep the key point positions unchanged, swap the embedding vectors of each object to obtain image samples before and after the object appearance is edited, and save the key box positions to obtain labeled image samples, which supplement the labeled image samples in the few-sample database, alleviate the overfitting problem of the object detection model caused by the lack of samples, and improve the object detection accuracy.

2. The image sample generation method based on key frame detection as described in claim 1, characterized in that: In step 1, The keyframe generation network generates the positions and appearances of K components using independent noise vectors. To prevent interference between position and appearance information, three sets of independent noise vectors are used. As input, the independent noise z is processed using a multilayer perceptron (MLP). keypoint and z WH Generate K key points The network has K groups with width W and height H; each hidden layer of this network is a fully connected layer using an MLP and LeakyReLU is used as the activation function. The output layer also uses a fully connected layer to control the output dimension; therefore, the location of the keyframe corresponding to each keypoint is... The keyframes are defined as rectangles, and each keyframe is the center point of its location. The positions of the 4×K keyframes are... Furthermore, the component dimensions are represented as {σ1,...,σ K },but, Where k = 1,...,K, (x row ,x col () represents the x and y coordinates of each key point; Use MLP to z app Mapping to component appearance vector Define a fixed embedding vector for each component. This fixed embedding is randomly generated using nn.Embeddings to encode semantic information of keypoints and is continuously updated during the training phase; w dynamic and Component embedding obtained by performing element-wise dot product In dynamic =MLP app (With app ) The noise sources in terms of appearance and location are independent of each other.

3. The image sample generation method based on key frame detection as described in claim 2, characterized in that: In step 2, The mask required for generating the SPADE image is generated based on the Mahalanobis-Gaussian heatmap and the initial position encoding, wherein the initial position encoding uses relative position encoding; wherein the local independence of the components is modeled using the Mahalanobis-Gaussian heatmap, and the distance relationship between the components is maintained using relative position encoding; Based on the keyframe coordinates generated in step 1, a Markov-Gaussian heatmap is used to model the local components and calculate the initial mask for each component. The component scale determined in step 1 controls the size of the initial mask; the initial mask H k Multiplying the embeddings of the corresponding components yields the embedding maps of each component. Adding the embeddings of all K components together forms a single embedding map. Each component's embedded image is represented by the key point position x. k Centered on d, the Gaussian value is calculated for each pixel position, while σ... k To control the effective range, the initial mask of each component exhibits a Gaussian shape; d(p,x k )=(p-x k ) T ∑ -1 (p-x k ) Where, ∑ -1 It is the inverse of the covariance matrix among the variables in the sample; As another input to the SPADE generator to generate the mask, the initial position encoding of the mask is... To avoid leaking absolute position information, low-frequency position encoding is used, based on the difference between the pixel position and the 4×K points: Introducing the Swing Transformer Block into SPADE ResBlock, the mask M = [M bg ,M1,...,M K The generation process of ] is represented as follows: M (i) =SPADE ResBlock(Swin Transformer Block(M (i-1) ),W mask ) i=1,...,T mask Among them, M bg As the background mask, T mask The number of SPADE ResBlocks; in addition, the background mask is generated using a separate channel.

4. The image sample generation method based on key frame detection as described in claim 3, characterized in that: In step 3, Based on the mask M generated in step 2, M = [M bg ,M1,...,M K ], foreground embedding The generation is similar to mask embedding W. mask The generation of foreground feature maps is performed using the same initial position encoding and foreground embedding. F (0) (p)=[sin(πFC([p-x1,...,p-x K ])),cos(πFC([p-x1,...,p-x K ]))] F (i) =SPADE ResBlock(Swin Transformer Block(F (i-1) ),W fg ) Where i = 1,...,T fg , where is the number of SPADE ResBlocks; Background generation requires the use of two sets of noise. and u bg_pos The background embedding is generated using MLP, directly corresponding to the center position of the background. IN bg =MLP bg_app (With bg_app ) Similarly, using the initial position encoding and the background center position u bg_pos Generate background feature map B (0) (p)=[sin(πFC([p-u bg_pos ])),cos(πFC([p-u bg_pos ]))] B (i) =AdaIN ConvBlock(B (i-1) ,W bg ) Where i = 1,...,T fg , where is the number of AdaIN ConvBlocks; The foreground and background feature maps are combined and input into a two-layer CNN network to generate the final image.

5. The image sample generation method based on key frame detection as described in claim 4, characterized in that: In step 4, The overall loss function can be summarized as follows: in, Represents generator, Let x represent the discriminator, and p represent the distribution of the real data. data The sample obtained by sampling; z is random noise input from the noise space N.

Images