Data enhancement system and method based on target pose disturbance

Abstract

The invention discloses a data enhancement system and method based on target pose disturbance. The system comprises a key point extraction module, a key point disturbance module and a deformation module. The key point extraction module is used for extracting key points in the original image; the method comprises the steps of inputting an original image, and outputting key points in the original image as key points before disturbance; the key point disturbance module is used for carrying out position disturbance on the key points and generating new key points; key points before disturbance are input, and new key points are output and serve as key points after disturbance; the deformation module is used for shifting the key points before disturbance to the key points after disturbance based on the original image and generating an enhanced image corresponding to the original image; according to the method, key points before disturbance on an original image are shifted to the positions of corresponding key points after disturbance through an iterative interpolation method, and an enhanced image is output. The method based on target pose disturbance is adopted, and the problem that samples generated through a traditional data enhancement method are free of geometric deformation and lack of diversity is solved.

Description

Technical Field

[0001] The present invention relates to the field of machine learning, and in particular to a data enhancement system and method based on target posture disturbance. Background Art

[0002] At present, the task of data augmentation is to modify or expand the original data in a targeted manner through a set of predefined transformation or conversion operations using some given data (usually some images or parts of images), that is, to generate new samples that conform to the original distribution from a large number of existing data samples through some means. Existing data samples usually refer to images that are already in the model training data and belong to known categories and environments. New samples refer to images of new styles that are generated. These images place higher requirements on the generalization ability of the model.

[0003] Traditional data augmentation methods are usually limited to processing single or multiple images at the pixel level. This type of method based on simple image space operations has great limitations on changes in texture and geometric morphology, and it is difficult to significantly improve the diversity and novelty of the potential information of the original data, thereby limiting the effect of introducing new knowledge and improving the generalization ability of the model. By introducing strategies to change the characteristics of geometric shapes, the representation capacity of the original data can be effectively expanded, thereby providing the model with more new knowledge, which is crucial to enhancing model performance. However, most existing methods are limited to local geometric transformations and fail to fully consider a wider range of scenarios, resulting in limited performance in complex applications.

[0004] In addition, most existing data augmentation methods based on feature space perform augmentation by operating data in the middle layers of the neural network and utilizing information at the feature level. In the feature space, the data has been transformed into more abstract representations by the first few layers of the neural network, which usually capture the key semantic features of the input data. Therefore, data augmentation in the feature space can better maintain the semantic consistency of the samples, making the generated enhanced samples more reasonable and meaningful. However, compared with the enhancement methods in the pixel space, the operations in the feature space are often difficult to visualize and intuitively understand, and may introduce pseudo samples that are inconsistent with the distribution of the original data, thereby negatively affecting model training.

[0005] In summary, in order to maintain the performance of the model in a wider range of application scenarios, especially when data is scarce or the data distribution is unbalanced, it is urgent to introduce more universal data augmentation methods. These methods should not only be able to effectively expand the size of the dataset, but also avoid introducing noise or invalid information to ensure the quality of the enhanced data. Therefore, for data augmentation tasks, we should not only pay attention to the diversity and rationality of the generated images, but also take into account the consistency and relevance of the generated samples with the original data distribution, so as to achieve the goal of truly improving the generalization ability and robustness of the model. Summary of the invention

[0006] The present invention provides a data enhancement system and method based on target posture disturbance to solve the technical problems existing in the known technology.

[0007] The technical solution adopted by the present invention to solve the technical problems existing in the known technology is:

[0008] A data enhancement system based on target posture perturbation, comprising a key point extraction module, a key point perturbation module and a deformation module;

[0009] The key point extraction module is used to extract the key points in the original image; it inputs the original image and outputs the key points in the original image; the key points it outputs are used as the key points before disturbance;

[0010] The key point perturbation module is used to perturb the key points and generate new key points. It inputs the key points before perturbation and outputs new key points. The key points it outputs are used as the key points after perturbation.

[0011] The deformation module is used to shift the key points before disturbance to the key points after disturbance based on the original image, and generate an enhanced image corresponding to the original image; it inputs the original image and the corresponding key points before and after disturbance; it shifts the key points before disturbance on the original image to the corresponding key points after disturbance through an iterative interpolation method, and outputs an enhanced image.

[0012] Furthermore, the deformation module shifts the pre-perturbation key points on the original image to the corresponding post-perturbation key points through the thin plate spline interpolation method.

[0013] Furthermore, the key point extraction module includes a Gaussian difference pyramid and / or a skeleton extraction module; the Gaussian difference pyramid is used to extract candidate key points from the original image; the skeleton extraction module is used to extract the skeleton from the original image; and the key point extraction module correspondingly extracts key points from the candidate key points and / or the skeleton.

[0014] Furthermore, it also includes a feature extraction module and a deformation loss calculation module;

[0015] The feature extraction module is used to extract features of the original image and the enhanced image; it inputs the original image and the enhanced image respectively, and outputs a feature map of the original image and a feature map of the enhanced image respectively;

[0016] The deformation loss calculation module is used to calculate the difference between the feature maps of the original image and the enhanced image; this difference is used as the deformation loss; when the deformation loss reaches the set threshold range, it outputs a deformation loss standard-reaching signal to the deformation module. After receiving the deformation loss standard-reaching signal, the deformation module stops iterative interpolation and uses the image generated before stopping iterative interpolation as the final enhanced image.

[0017] Furthermore, the feature extraction module includes a VGG feature extraction network.

[0018] The present invention also provides a data enhancement method based on target posture disturbance, which sets a key point extraction module, a key point disturbance module and a deformation module;

[0019] A key point extraction module is used to extract key points in the original image; the module inputs the original image and outputs the key points in the original image; the output key points are used as the key points before disturbance;

[0020] The key point perturbation module is used to perturb the position of the key point and generate a new key point; the key point before the perturbation is input and the new key point is output; the key point output is used as the key point after the perturbation;

[0021] A deformation module is used to shift the key points before disturbance to the key points after disturbance based on the original image, and generate an enhanced image corresponding to the original image; the original image and the corresponding key points before and after disturbance are input; the key points before disturbance on the original image are shifted to the corresponding key points after disturbance through an iterative interpolation method, and the enhanced image is output.

[0022] Furthermore, the deformation module shifts the pre-disturbance key points on the original image to the corresponding post-disturbance key point positions through the thin plate spline interpolation method.

[0023] Furthermore, the key point extraction module sets a Gaussian difference pyramid and / or a skeleton extraction module; uses the Gaussian difference pyramid to extract candidate key points from the original image; uses the skeleton extraction module to extract the skeleton from the original image; and enables the key point extraction module to extract key points from the candidate key points and / or the skeleton accordingly.

[0024] Furthermore, a feature extraction module and a deformation loss calculation module are also provided;

[0025] A feature extraction module is used to extract features of the original image and the enhanced image; the module is used to input the original image and the enhanced image respectively, and to output a feature map of the original image and a feature map of the enhanced image respectively;

[0026] A deformation loss calculation module is used to calculate the difference between the feature maps of the original image and the enhanced image; this difference is used as the deformation loss; when the deformation loss reaches the set threshold range, it outputs a deformation loss standard signal to the deformation module, so that the deformation module stops iterative interpolation after receiving the deformation loss standard signal, and uses the image generated before stopping iterative interpolation as the final enhanced image.

[0027] Furthermore, the deformation loss calculation formula of the deformation loss calculation module is as follows:

[0028] Ltotal =L content +αL wrap +βL tv +E;

[0029]

[0030] L tv =∑ e，f ((I e+1，f -I e，f ) 2 +(I e，f +1-I e，f ) 2 );

[0031] E=∑ j w i φ(||p j -p i -θ i ||)+b[cot(θ i，j )L(i,j)+cot(θ j，i )L(j,i)](x i -x j );

[0032] Let r = || p j -p i -θ i ||, then: φ(Y)=r 2 logr;

[0033] Where:

[0034] L total is the total deformation loss;

[0035] L content is the content deformation loss between the original image and the enhanced image;

[0036] L wrap is the deformation loss between the original image and the enhanced image;

[0037] L tv is the total variational deformation loss between the original image and the enhanced image;

[0038] E is the rigid deformation energy;

[0039] α is the coefficient of deformation loss;

[0040] β is the coefficient of total variational deformation loss;

[0041] To enhance the feature representation of the image at the vth layer position of the uth block of the feature extraction module;

[0042] is the feature representation of the original image at the position of the vth layer of the uth block of the feature extraction module;

[0043] w i is the weight of the i-th key point;

[0044] i and j are key point numbers;

[0045] θ i is the vector of perturbation of the i-th key point;

[0046] p i is the vector of the coordinates of the i-th key point in the original image relative to the origin;

[0047] p j is the vector of the coordinates of the jth key point in the original image relative to the origin;

[0048] p i ′ is the vector of the coordinates of the point corresponding to the i-th key point in the original image in the enhanced image relative to the origin;

[0049] I e，f is the enhanced image, pixel at position (e, f); e is the enhanced image, position abscissa, f is the enhanced image, position ordinate;

[0050] I e+1，f To enhance the image, the pixel at position (e+1, f);

[0051] I e，f+1 To enhance the image, the pixel at position (e, f+1);

[0052] θ i，j is the angle between the perturbation vector of the i-th key point and the perturbation vector of the j-th key point;

[0053] θ i，j is the angle between the vector of the j-th key point perturbation and the vector of the i-th key point perturbation;

[0054] L(i, j) is the Laplace operator of the i-th key point and the j-th key point;

[0055] L(j, i) is the Laplace operator of the jth key point and the ith key point;

[0056] x i For p i The horizontal axis of

[0057] x j is the horizontal coordinate of pj;

[0058] r is the intermediate variable;

[0059] b is the coefficient;

[0060] k is the number of key points.

[0061] The advantages and positive effects of the present invention are:

[0062] 1. This paper proposes a data enhancement method based on target pose perturbation, which aims to solve the problem that traditional pixel-level data enhancement methods cannot effectively change the target geometry, and also makes up for the deficiency that feature-level data enhancement is difficult to intuitively understand. Pose perturbation can not only change the appearance of the target, but also enhance the diversity of the image without changing its semantic features.

[0063] 2. The present invention proposes to combine the iterative framework of style transfer and the method of deformation as rigid as possible to solve the problems of smoothness and rationality of geometric deformation in the generated image. Through this method, the generated image can be locally deformed while maintaining the overall structure, making the deformation natural and distortion-free. At the same time, combined with the iterative framework of style transfer, a variety of potential styles can be effectively generated. When the same perturbation is input to the key points, the segmentation mask can also produce the same geometric changes as the original image, which is suitable for semantic segmentation tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0064] Figure 1 It is a schematic diagram of the workflow of a data enhancement system based on target posture disturbance of the present invention.

[0065] In the picture:

[0066] P represents the key point; θ represents the perturbation angle. DETAILED DESCRIPTION

[0067] The present invention will be described in detail below with reference to the accompanying drawings and in combination with embodiments. It should be understood that the preferred embodiments described herein are only used to illustrate and explain the present invention, and are not used to limit the present invention.

[0068] See also Figure 1 , a data enhancement system based on target pose perturbation, including a key point extraction module, a key point perturbation module and a deformation module;

[0069] The key point extraction module is used to extract the key points in the original image; it inputs the original image and outputs the key points in the original image; the key points it outputs are used as the key points before disturbance;

[0070] The key point perturbation module is used to perturb the key points and generate new key points. It inputs the key points before perturbation and outputs new key points. The key points it outputs are used as the key points after perturbation.

[0071] The deformation module is used to shift the key points before disturbance to the key points after disturbance based on the original image, and generate an enhanced image corresponding to the original image; it inputs the original image and the corresponding key points before and after disturbance; it shifts the key points before disturbance on the original image to the corresponding key points after disturbance through an iterative interpolation method, and outputs an enhanced image.

[0072] The key point perturbation module randomly perturbs the extracted key points, so that the generated image presents geometric deformation.

[0073] Preferably, the deformation module can shift the pre-disturbance key points on the original image to the corresponding post-disturbance key point positions by thin plate spline interpolation method.

[0074] The deformation module uses thin plate spline interpolation to generate a smooth distortion function for a given set of key points before and after the disturbance, mapping the key points before the disturbance to the key points after the disturbance. The deformation module uses thin plate spline interpolation to minimize the energy function, not only accurately matching the target points at the given control points, but also minimizing the energy of the total deformation through the physical model, thereby ensuring the smoothness and rationality of the interpolation results.

[0075] Preferably, the key point extraction module may include a Gaussian difference pyramid and / or a skeleton extraction module; the Gaussian difference pyramid can be used to extract candidate key points from the original image; the skeleton extraction module can be used to extract the skeleton from the original image; the key point extraction module can extract key points from the candidate key points and / or the skeleton accordingly.

[0076] The Gaussian difference pyramid may correspond to images containing ordinary objects, and the skeleton extraction module may correspond to images containing animal objects.

[0077] Key point extraction module: This module is used to sort the candidate key points generated by the Gaussian difference pyramid and / or skeleton extraction module according to the response value and select the best key points.

[0078] Gaussian difference pyramid is an image processing technique used for multi-scale image feature detection and is widely used in image matching, feature extraction and computer vision tasks. Gaussian difference pyramid consists of two main parts:

[0079] Gaussian blur: This part blurs the input image using Gaussian kernels of different scales, each of which corresponds to a different standard deviation. The low-level Gaussian blurred image retains more global or large-scale information, while the high-level blurred image emphasizes more details and local information. In this way, the Gaussian difference pyramid can obtain image features at multiple scales. Low-scale images are suitable for processing large, smooth areas, while high-scale images retain more local edge and corner features, providing a basis for subsequent edge, corner and other feature detection.

[0080] Gaussian difference: This part performs a difference operation on Gaussian blurred images of adjacent scales to generate a Gaussian difference image. This method detects edges and feature points by approximating the Laplace operator. The difference operation can effectively remove irrelevant factors such as lighting and color in the image, while retaining key information such as edges and contours. Specifically, difference images of different scales can simultaneously detect feature points and edges of different sizes, thereby generating a multi-scale, multi-information feature map. This process utilizes the smoothness and gradient characteristics of each layer of Gaussian blur images, enhances the feature extraction capability of the image at different scales, and helps to deal with complex geometric change problems.

[0081] Skeleton extraction module: This module uses the animal skeleton extraction method of the MMPose open source framework, which mainly includes a skeleton regression network and a head network for detecting and regressing skeleton key points. The skeleton regression network is used to extract multi-scale features and help combine multi-scale information to enhance the effect of key point detection; the head network is used to generate key point heat maps.

[0082] Preferably, it may also include a feature extraction module and a deformation loss calculation module;

[0083] The feature extraction module can be used to extract features of the original image and the enhanced image; it can input the original image and the enhanced image respectively, and output the feature map of the original image and the feature map of the enhanced image respectively;

[0084] The deformation loss calculation module can be used to calculate the difference between the feature maps of the original image and the enhanced image; this difference can be used as the deformation loss; when the deformation loss reaches the set threshold range, it can output a deformation loss standard signal to the deformation module. After receiving the deformation loss standard signal, the deformation module stops iterative interpolation and uses the image generated before stopping iterative interpolation as the final enhanced image.

[0085] Preferably, the feature extraction module may include a VGG feature extraction network.

[0086] VGG feature extraction network: VGG16 is a classic convolutional neural network architecture, first proposed by the Visual Geometry Group (VGG) of Oxford University for image classification and feature extraction. Its main function is to extract feature maps of the original image and the generated image, so as to calculate deformation loss in the subsequent network. The convolution layers of the first few layers of the VGG network can extract low-level features such as edges, textures, etc., while the middle layers begin to focus on more shapes and contours, and the high-level convolution layers capture more complex image structure information. When used for feature extraction, only the convolution layer part is used, and the pre-trained weights are fixed.

[0087] The present invention also provides a data enhancement method based on target posture disturbance, which sets a key point extraction module, a key point disturbance module and a deformation module;

[0088] A key point extraction module is used to extract key points in the original image; the module inputs the original image and outputs the key points in the original image; the output key points are used as the key points before disturbance;

[0089] The key point perturbation module is used to perturb the position of the key point and generate a new key point; the key point before the perturbation is input and the new key point is output; the key point output is used as the key point after the perturbation;

[0090] A deformation module is used to shift the key points before disturbance to the key points after disturbance based on the original image, and generate an enhanced image corresponding to the original image; the original image and the corresponding key points before and after disturbance are input; the key points before disturbance on the original image are shifted to the corresponding key points after disturbance through an iterative interpolation method, and the enhanced image is output.

[0091] The number of iterations of iterative interpolation can be constrained by setting a fixed value or by setting other parameters. The number of iterations can be set to 200 to 300 times.

[0092] Preferably, the deformation module can shift the pre-disturbance key points on the original image to the corresponding post-disturbance key point positions by thin plate spline interpolation method.

[0093] Preferably, the key point extraction module can be set with a Gaussian difference pyramid and / or a skeleton extraction module; the Gaussian difference pyramid can be used to extract candidate key points from the original image; the skeleton extraction module can be used to extract the skeleton from the original image; and the key point extraction module is enabled to extract key points from the candidate key points and / or the skeleton accordingly.

[0094] Preferably, a feature extraction module and a deformation loss calculation module may also be provided;

[0095] The feature extraction module can be used to extract the features of the original image and the enhanced image; the original image and the enhanced image can be input respectively, and the feature map of the original image and the feature map of the enhanced image can be output respectively;

[0096] The deformation loss calculation module can be used to calculate the difference between the feature maps of the original image and the enhanced image; this difference can be used as the deformation loss; when the deformation loss reaches the set threshold range, it can output a deformation loss standard signal to the deformation module, so that the deformation module stops iterative interpolation after receiving the deformation loss standard signal, and uses the image generated before stopping iterative interpolation as the final enhanced image.

[0097] The deformation loss calculation module can calculate the deformation loss between the original image and the generated image, and feed it back to the deformation module to obtain a better enhanced image.

[0098] Through the organic combination of these modules, this data augmentation method can generate more diverse samples, enrich the data set, and is widely used in segmentation tasks in the field of computer vision.

[0099] Preferably, the deformation loss calculation formula of the deformation loss calculation module may be as follows:

[0100] L total =L content +αL wrap +βL tv +E;

[0101]

[0102] L tv =∑ e，f ((I e+1，f -I e，f ) 2 +(I e，f+1 -I e，f ) 2 );

[0103] E=∑ j w i φ(||p j -p i -θ i ||)+b[cot(θ i，j )L(i,j)+cot(θ j，i )L(j,i)](x i -x j );

[0104] Let r = || p j -p i -θ i ||, then: φ(r)=r 2 logr;

[0105] Where:

[0106] L totai is the total deformation loss;

[0107] L content is the content deformation loss between the original image and the enhanced image;

[0108] L wrap is the deformation loss between the original image and the enhanced image;

[0109] L tv is the total variational deformation loss between the original image and the enhanced image;

[0110] E is the rigid deformation energy;

[0111] α is the coefficient of deformation loss;

[0112] β is the coefficient of total variational deformation loss;

[0113] To enhance the feature representation of the image at the vth layer position of the uth block of the feature extraction module;

[0114] is the feature representation of the original image at the position of the vth layer of the uth block of the feature extraction module;

[0115] w i is the weight of the i-th key point;

[0116] i and j are key point numbers;

[0117] α i is the vector of perturbation of the i-th key point;

[0118] p i is the vector of the coordinates of the i-th key point in the original image relative to the origin;

[0119] p j is the vector of the coordinates of the jth key point in the original image relative to the origin;

[0120] p i ′ is the vector of the coordinates of the point corresponding to the i-th key point in the original image in the enhanced image relative to the origin;

[0121] I e，f is the enhanced image, pixel at position (e, f); e is the enhanced image, position abscissa, f is the enhanced image, position ordinate;

[0122] I e+1，f To enhance the image, the pixel at position (e+1, f);

[0123] I e，f+1 To enhance the image, the pixel at position (e, f+1);

[0124] θ i，j is the angle between the perturbation vector of the i-th key point and the perturbation vector of the j-th key point;

[0125] θ i，j is the angle between the vector of the j-th key point perturbation and the vector of the i-th key point perturbation;

[0126] L(i, j) is the Laplace operator of the i-th key point and the j-th key point;

[0127] L(j, i) is the Laplace operator of the jth key point and the ith key point;

[0128] x i For p i The horizontal axis of

[0129] x j For p j The horizontal axis of

[0130] r is the intermediate variable;

[0131] b is the coefficient;

[0132] k is the number of key points.

[0133] φ(r) represents a function of r.

[0134] The structure, working process and working principle of the present invention are further described below with reference to a preferred embodiment of the present invention:

[0135] See also Figure 1 ,A data enhancement system based on target posture perturbation, including a key point extraction module, a key point perturbation module, a deformation module, a feature extraction module and a deformation loss calculation module;;

[0136] The key point extraction module is used to extract the key points in the original image; it inputs the original image and outputs the key points in the original image; the key points it outputs are used as the key points before disturbance;

[0137] The key point perturbation module is used to perturb the key points and generate new key points. It inputs the key points before perturbation and outputs new key points. The key points it outputs are used as the key points after perturbation.

[0138] The deformation module is used to shift the key points before disturbance to the key points after disturbance based on the original image, and generate an enhanced image corresponding to the original image; it inputs the original image and the corresponding key points before and after disturbance; it shifts the key points before disturbance on the original image to the corresponding key points after disturbance through an iterative interpolation method, and outputs an enhanced image.

[0139] The feature extraction module is used to extract the features of the original image and the enhanced image; it can input the original image and the enhanced image respectively, and output the feature map of the original image and the feature map of the enhanced image respectively;

[0140] The deformation loss calculation module is used to calculate the difference between the feature maps of the original image and the enhanced image; this difference can be used as the deformation loss; when the deformation loss reaches the set threshold range, it can output a deformation loss standard signal to the deformation module. After receiving the deformation loss standard signal, the deformation module stops iterative interpolation and uses the image generated before stopping iterative interpolation as the final enhanced image.

[0141] In order to improve the performance in data enhancement tasks, this paper proposes a new data enhancement branch. The deformation module realizes the perturbation of the target pose, solving the problem that the traditional pixel-level data enhancement method cannot effectively change the geometric shape of the target. The deformation caused by the pose perturbation can not only change the appearance of the target, but also enhance the diversity of the image without changing its semantic features.

[0142] The present invention considers the key points in the original image as part of image generation. The method of the present invention is based on iterative style transfer and as rigid as possible deformation algorithm, and generates more diverse samples through iterative generation based on key points, thereby enhancing the generalization ability of other segmentation models.

[0143] The key point extraction module may include a Gaussian difference pyramid and / or a skeleton extraction module; the Gaussian difference pyramid is used to extract candidate key points from the original image; the skeleton extraction module is used to extract the skeleton from the original image; the key point extraction module correspondingly extracts key points from the candidate key points and / or the skeleton.

[0144] Both the Gaussian difference pyramid and the skeleton extraction module can be used to extract key points, and one or the other can be used simultaneously.

[0145] The Gaussian difference pyramid uses the kernel function to perform Gaussian blurring of different standard deviations of the image layer by layer to produce blurred images of different scales. The kernel function G(x, y, σ) is as follows:

[0146]

[0147] Where:

[0148] x is the horizontal coordinate of the current pixel in the image relative to the blurred center pixel;

[0149] y is the ordinate of the current pixel in the image relative to the blurred center pixel;

[0150] σ is the standard deviation of the Gaussian function;

[0151] is the normalized coefficient of the Gaussian function;

[0152] is the exponential part of the Gaussian function, which is the influence of the distance between the pixel and the center point in the image on the blur weight. The farther the pixel is from the center, the faster its weight decays.

[0153] The Gaussian difference pyramid is used to extract blurred images of different scales from the original image. The values ​​of each pixel are compared with its adjacent pixels in space and scale. If the pixel is a maximum or minimum value in space and scale, the pixel is considered to be a potential key point. The Gaussian difference pyramid uses the kernel function to perform Gaussian blurring of different standard deviations on the image layer by layer to generate blurred images of different scales.

[0154] The key point extraction module sorts the key points according to the response values ​​and selects the key points with greater intensity;

[0155] The deformation module uses thin plate spline interpolation to minimize the energy function and obtain the optimal generation result. Its energy E m The calculation formula is as follows:

[0156]

[0157] Let s = || q i -p i ||

[0158] φ(s)=s 2 logs;

[0159] Where:

[0160] s is an intermediate variable;

[0161] p i is the vector of the coordinates of the i-th key point in the original image relative to the origin;

[0162] q i is the vector of the coordinates of the i-th perturbed key point in the original image relative to the origin;

[0163] w i is the weight of the i-th key point;

[0164] i is the key point number;

[0165] n is the number of key points.

[0166] φ(s) represents a function of s.

[0167] Preferably, the VGG feature extraction network may adopt a VGG-16 network. The skeleton extraction module may adopt an MMPose skeleton extraction module.

[0168] Preferably, the deformation loss calculation module calculates the difference between the feature maps of the original image and the enhanced image to obtain the deformation loss. The deformation loss function calculation formula is as follows:

[0169] L total =L content +αL wrap +βL tv +E;

[0170]

[0171] L tv =∑ e，f ((I e+1,f -I e,f ) 2 +(I e，f+1 -I e,f ) 2 );

[0172] E=∑ j w i φ(||p j -p i -θ i ||)+b[cot(θ i，j )L(i,j)+cot(θ j，i )L(j,i)](x i -x j );

[0173] Let r = || p j -p i -θ i ||, then: φ(r)=r 2 logr;

[0174] Where:

[0175] L total is the total deformation loss;

[0176] L content is the content deformation loss between the original image and the enhanced image;

[0177] L wrap is the deformation loss between the original image and the enhanced image;

[0178] L tvis the total variational deformation loss between the original image and the enhanced image;

[0179] E is the rigid deformation energy;

[0180] α is the coefficient of deformation loss;

[0181] β is the coefficient of total variational deformation loss;

[0182] To enhance the feature representation of the image at the vth layer position of the uth block of the feature extraction module;

[0183] is the feature representation of the original image at the position of the vth layer of the uth block of the feature extraction module;

[0184] w i is the weight of the i-th key point;

[0185] i and j are key point numbers;

[0186] θ i is the vector of perturbation of the i-th key point;

[0187] p i is the vector of the coordinates of the i-th key point in the original image relative to the origin;

[0188] p j is the vector of the coordinates of the jth key point in the original image relative to the origin;

[0189] p i ′ is the vector of the coordinates of the point corresponding to the i-th key point in the original image in the enhanced image relative to the origin;

[0190] I e，f is the enhanced image, pixel at position (e, f); e is the enhanced image, position abscissa, f is the enhanced image, position ordinate;

[0191] I e+1，f To enhance the image, the pixel at position (e+1, f);

[0192] I e，f+1 To enhance the image, the pixel at position (e, f+1);

[0193] θ i，j is the angle between the perturbation vector of the i-th key point and the perturbation vector of the j-th key point;

[0194] θ i，j is the angle between the vector of the j-th key point perturbation and the vector of the i-th key point perturbation;

[0195] L(i, j) is the Laplace operator of the i-th key point and the j-th key point;

[0196] L(j, i) is the Laplace operator of the jth key point and the ith key point;

[0197] x i For p i The horizontal axis of

[0198] x j is the horizontal coordinate of pj;

[0199] r is the intermediate variable;

[0200] b is the coefficient;

[0201] k is the number of key points.

[0202] The present invention also provides a data enhancement method based on target posture disturbance, which method sets a key point extraction module, a key point disturbance module, a deformation module, a feature extraction module and a deformation loss calculation module;

[0203] A key point extraction module is used to extract key points in the original image; the module inputs the original image and outputs the key points in the original image; the output key points are used as the key points before disturbance;

[0204] The key point perturbation module is used to perturb the position of the key point and generate a new key point; the key point before the perturbation is input and the new key point is output; the key point output is used as the key point after the perturbation;

[0205] A deformation module is used to shift the key points before disturbance to the key points after disturbance based on the original image, and generate an enhanced image corresponding to the original image; the original image and the corresponding key points before and after disturbance are input; the key points before disturbance on the original image are shifted to the corresponding key points after disturbance through an iterative interpolation method, and the enhanced image is output.

[0206] A feature extraction module is used to extract features of the original image and the enhanced image; the module is used to input the original image and the enhanced image respectively, and to output a feature map of the original image and a feature map of the enhanced image respectively;

[0207] A deformation loss calculation module is used to calculate the difference between the feature maps of the original image and the enhanced image; this difference is used as the deformation loss; when the deformation loss reaches the set threshold range, it outputs a deformation loss standard signal to the deformation module, so that the deformation module stops iterative interpolation after receiving the deformation loss standard signal, and uses the image generated before stopping iterative interpolation as the final enhanced image.

[0208] The key point extraction module sets a Gaussian difference pyramid, a key point screening module, and / or a skeleton extraction module, a skeleton key point extraction module; uses a Gaussian difference pyramid to extract candidate key points from the original image; uses the skeleton extraction module to extract the skeleton from the original image; enables the key point screening module to screen key points from candidate key points, and uses the skeleton key point extraction module to extract key points on the skeleton; that is, the key point extraction module correspondingly extracts key points from candidate key points and / or skeletons.

[0209] The method comprises the following steps:

[0210] Step 1: Use Gaussian difference pyramid to extract candidate key points from the original image, or use skeleton extraction module to extract skeleton from the original image;

[0211] Step 2: Use the key point screening module to screen key points, or use the skeleton key point extraction module to extract key points on the skeleton; use the skeleton extraction module to extract image features of different scales of the original image and generate a key point heat map

[0212] Step 3: Use the key point perturbation module to generate the perturbed key point coordinates;

[0213] Step 4: Use the deformation module to generate an enhanced image;

[0214] Step 5: Use VGG feature extraction network to extract enhanced image and original image features;

[0215] Step 6, using a deformation loss calculation module to calculate the deformation loss between the enhanced image and the original image;

[0216] Step 7: Repeat steps 4 to 6 until the deformation loss reaches the target or the number of iterations reaches the set value, and the iteration is completed.

[0217] Step eight, obtain the final enhanced image.

[0218] The above-mentioned key point extraction module, key point perturbation module, deformation module, feature extraction module and deformation loss calculation module, Gaussian difference pyramid, key point screening module, skeleton extraction module, skeleton key point extraction module, VGG feature extraction network, VGG-16 network, MMPose skeleton extraction module and other functional modules can all adopt applicable functional modules in the existing technology, or adopt functional modules and software in the existing technology and be constructed by conventional technical means.

[0219] The embodiments described above are only used to illustrate the technical ideas and features of the present invention, and their purpose is to enable those skilled in the art to understand the contents of the present invention and implement them accordingly. The patent scope of the present invention cannot be limited only by this embodiment, that is, any equivalent changes or modifications made to the spirit disclosed by the present invention still fall within the patent scope of the present invention.

Claims

1. A data augmentation system based on target pose perturbation, characterized in that: It includes key point extraction module, key point perturbation module and deformation module; The key point extraction module is used to extract the key points in the original image; It inputs the original image and outputs the key points in the original image; the key points outputted are used as the key points before disturbance; The key point perturbation module is used to perturb the position of key points and generate new key points; It inputs the key point before the disturbance and outputs the new key point; The key points of its output are used as the key points after disturbance; The deformation module is used to shift the key points before the disturbance to the key points after the disturbance based on the original image, and generate an enhanced image corresponding to the original image; Its input is the original image and the corresponding key points before and after the perturbation; It uses an iterative interpolation method to shift the key points before disturbance on the original image to the corresponding key points after disturbance, and outputs an enhanced image.

2. The data enhancement system based on target pose perturbation according to claim 1, characterized in that: The deformation module uses the thin plate spline interpolation method to shift the key points before disturbance on the original image to the corresponding key points after disturbance.

3. The data enhancement system based on target pose disturbance according to claim 1, characterized in that: The key point extraction module includes a Gaussian difference pyramid and / or a skeleton extraction module; the Gaussian difference pyramid is used to extract candidate key points from the original image; the skeleton extraction module is used to extract the skeleton from the original image; and the key point extraction module correspondingly extracts key points from the candidate key points and / or the skeleton.

4. The data enhancement system based on target pose perturbation according to claim 1, characterized in that: It also includes a feature extraction module and a deformation loss calculation module; The feature extraction module is used to extract the features of the original image and the enhanced image; It inputs the original image and the enhanced image respectively, and outputs the feature map of the original image and the feature map of the enhanced image respectively; The deformation loss calculation module is used to calculate the difference between the feature maps of the original image and the enhanced image; this difference is used as the deformation loss; when the deformation loss reaches the set threshold range, it outputs a deformation loss standard-reaching signal to the deformation module. After receiving the deformation loss standard-reaching signal, the deformation module stops iterative interpolation and uses the image generated before stopping iterative interpolation as the final enhanced image.

5. The data enhancement system based on target pose disturbance according to claim 1, characterized in that: The feature extraction module includes the VGG feature extraction network.

6. A data enhancement method based on target pose perturbation, characterized in that: Set up a key point extraction module, a key point perturbation module and a deformation module; A key point extraction module is used to extract key points in the original image; the module inputs the original image and outputs the key points in the original image; The key points of its output are used as the key points before disturbance; The key point perturbation module is used to perturb the position of the key point and generate a new key point; the key point before the perturbation is input and the new key point is output; The key points of its output are used as the key points after disturbance; A deformation module is used to shift the key points before disturbance to the key points after disturbance based on the original image, and generate an enhanced image corresponding to the original image; the original image and the corresponding key points before and after disturbance are input; the key points before disturbance on the original image are shifted to the corresponding key points after disturbance through an iterative interpolation method, and the enhanced image is output.

7. The data enhancement method based on target pose perturbation according to claim 6, characterized in that: The deformation module uses thin plate spline interpolation to shift the key points before disturbance on the original image to the corresponding key points after disturbance.

8. The data enhancement method based on target pose perturbation according to claim 6, characterized in that: The key point extraction module sets up a Gaussian difference pyramid and / or a skeleton extraction module; A Gaussian difference pyramid is used to extract candidate key points from the original image; a skeleton extraction module is used to extract a skeleton from the original image; and the key point extraction module is used to extract key points from the candidate key points and / or the skeleton.

9. The data enhancement method based on target pose perturbation according to claim 6, characterized in that: A feature extraction module and a deformation loss calculation module are also provided; A feature extraction module is used to extract features of the original image and the enhanced image; the module is used to input the original image and the enhanced image respectively, and to output a feature map of the original image and a feature map of the enhanced image respectively; A deformation loss calculation module is used to calculate the difference between the feature maps of the original image and the enhanced image; this difference is used as the deformation loss; when the deformation loss reaches the set threshold range, it outputs a deformation loss standard signal to the deformation module, so that the deformation module stops iterative interpolation after receiving the deformation loss standard signal, and uses the image generated before stopping iterative interpolation as the final enhanced image.

10. The data enhancement method based on target pose disturbance according to claim 1, characterized in that: The deformation loss calculation formula of the deformation loss calculation module is as follows: L total =L content +αL wrap +βL tv +E; L tv ==∑ e，f ((I e+1，f -I e，f ) 2 +(I e，f+1 -I e，f ) 2 ); E=∑ j w i φ(||p j -p i -θ i ||)+b[cot(θ i，j )L(i,j)+cot(θ j，i )L(j,i)](x i -x j ); Let r = || p j -p i -θ i ||, then: φ(r)=r 2 logr; Where: L total is the total deformation loss; L content is the content deformation loss between the original image and the enhanced image; L wrap is the deformation loss between the original image and the enhanced image; L tv is the total variational deformation loss between the original image and the enhanced image; E is the rigid deformation energy; α is the coefficient of deformation loss; β is the coefficient of total variational deformation loss; To enhance the feature representation of the image at the vth layer position of the uth block of the feature extraction module; is the feature representation of the original image at the position of the vth layer of the uth block of the feature extraction module; w i is the weight of the i-th key point; i and j are key point numbers; θ i is the vector of perturbation of the i-th key point; p i is the vector of the coordinates of the i-th key point in the original image relative to the origin; p j is the vector of the coordinates of the jth key point in the original image relative to the origin; p i ′ is the vector of the coordinates of the point corresponding to the i-th key point in the original image in the enhanced image relative to the origin; I e，f To enhance the image, the pixel at position (e, f); e is the abscissa of the position of enhanced image I, and f is the ordinate of the position of enhanced image I; I e+1，f To enhance the image, the pixel at position (e+1, f); I e，f+1 To enhance the pixel at position (e, f+1) of image I; θ i，j is the angle between the vector of the perturbation of the i-th key point and the vector of the perturbation of the j-th key point; θ i，j is the angle between the vector of the j-th key point perturbation and the vector of the i-th key point perturbation; L(i, j) is the Laplace operator of the i-th key point and the j-th key point; L(j, i) is the Laplace operator of the jth key point and the ith key point; x i For p i The horizontal axis of x j For p j The horizontal axis of r is the intermediate variable; b is the coefficient; k is the number of key points.

Images