Training sample image enhancement method and system based on neural network

Abstract

The present invention provides a training sample image enhancement method and system based on a neural network, the method comprising the following steps: manually labeling sample images in an initial sample set to obtain semantic segmentation labels; using the semantic segmentation labels to segment the sample images, and generating multiple combination information according to the similarity of the segmented images; calculating the preference of each combination information, and selecting a pre-expanded number of combination information as an augmented image in descending order of preference; using a training discriminator to judge the authenticity of the augmented image, and optimizing the image fusion parameters according to the authenticity of the augmented image to obtain an augmented sample set. The present invention amplifies more similar images that meet the requirements by splicing and fusion methods such as the interchange of foreground objects and backgrounds in different sample images, which not only increases the quantity and richness of sample data, but also reduces the corresponding manual labeling workload.

Description

Technical Field

[0001] The present invention relates to a training sample enhancement method, and in particular to a training sample image enhancement method and system based on a neural network. Background Art

[0002] At present, artificial intelligence is applied to all walks of life. The deep neural network technology it relies on is widely used in image, video, natural language processing and other fields due to its strong adaptability, excellent ability to handle nonlinear relationships, and fast parallel computing speed. However, the construction of deep neural networks requires massive and rich data to support it. With the continuous improvement of deep learning algorithms, the main task of its construction is almost concentrated in the early data collection part. In addition, the quality and richness of the training data used by the neural network affect the final performance of the network.

[0003] The collection of training data usually requires a lot of manpower and material resources, and it is often difficult to collect the amount of data that meets the training task requirements in practical applications. At present, the enhancement of training data is mostly through translation, rotation, etc. to transform one image into multiple different images, but the richness of the training set has not increased effectively, it has simply expanded the number. CN109948562B discloses manually splicing and editing images through image editing software PS, but this method relies on manual operation, requires a lot of labor and time costs, and has a low degree of automation.

[0004] In order to solve the above problems, people have been seeking an ideal technical solution. Summary of the invention

[0005] The purpose of the present invention is to address the deficiencies of the prior art and thus provide a training sample image enhancement method and system based on a neural network.

[0006] In order to achieve the above object, the technical solution adopted by the present invention is: a training sample image enhancement method based on a neural network, comprising the following steps:

[0007] Manually annotate the sample images in the initial sample set to obtain semantic segmentation labels;

[0008] Use semantic segmentation labels to segment sample images and obtain background and foreground objects; calculate the similarity between any two foreground objects and any two backgrounds;

[0009] According to the order of sample images, for sample images containing both background and foreground targets, the background is replaced according to the similarity of the foreground target, and the foreground target is replaced according to the similarity of the background to obtain combined information; for sample images containing only background, the foreground target in the sample image similar to its background is added to obtain combined information;

[0010] The preference of each combination of information is calculated, and the combination of information of the pre-expanded quantity is selected in descending order of preference. The corresponding area corresponding to each combination of information is subjected to Poisson fusion to obtain an amplified image, which constitutes an amplified sample set.

[0011] Based on the above, when calculating the similarity of any two foreground objects, the following steps are performed:

[0012] Connected domain analysis is performed on each foreground object to obtain the coordinate data of each point of the edge contour and the coordinate data of the center point of the region. The polar coordinates of the edge contour and the variance of the corresponding path length of each point on the edge are calculated with the center point of the region as the origin; the similarity of the edge contours of two foreground objects is calculated based on the variance of the corresponding path length of each point on the edge of the foreground object;

[0013] Perform corner point detection and description on the foreground object, match the corner points of two foreground objects, and calculate the similarity of the internal areas of the two foreground objects based on the matching results of the corner points;

[0014] The similarity of two foreground targets is calculated according to the formula QS(A, B)=α*bs(A, B)+β*ns(A, B), where QS(A, B) represents the similarity between foreground target A and foreground target B, bs(A, B) represents the edge contour similarity between foreground target A and foreground target B, ns(A, B) represents the internal area similarity between foreground target A and foreground target B, and α and β represent the influence weight of edge contour similarity and the influence weight of internal area similarity, respectively.

[0015] Based on the above, Calculate the similarity of the edge contours of any two foreground objects, where R i represents the distance from the i-th edge point of foreground target A to the center point of the region of foreground target A, N is the number of edge points of foreground target A, RU A is the average distance from the N edge points of foreground object A to the center point of the region of foreground object A; R j represents the distance from the jth edge point of foreground object B to the center point of the region of foreground object B, M is the number of edge points of foreground object B, RU B It is the average distance from the M edge points of foreground object B to the center point of the region of foreground object B.

[0016] Based on the above, when calculating the similarity of the internal areas of two foreground objects based on the matching results of the corner points, the following steps are performed:

[0017] Calculate the corner point matching rate pl of any two foreground targets:

[0018] Among them, DS A Indicates the number of corner points of the foreground target A, DS B Indicates the number of corner points of foreground object B; PS indicates the number of successfully matched corner points of foreground object A and foreground object B; kd is the preset ratio parameter;

[0019] Calculate the discreteness ls of the matching points:

[0020]

[0021] Where js(A) represents the number of angles of successfully matched corner points in foreground target A, and js(B) represents the number of angles of successfully matched corner points in foreground target B; rs(A) represents the number of path length categories of successfully matched corner points in foreground target A, and rs(B) represents the number of path length categories of successfully matched corner points in foreground target B; max(R A，m ) represents the maximum value of the corresponding path length in the foreground object A; max(R B，m ) represents the maximum value of the corresponding path length in the foreground object B.

[0022] Based on the above, when calculating the similarity of any two backgrounds, the following steps are performed:

[0023] Use the pre-trained VGG network to extract the local feature image and global feature image of the background of each sample image, and calculate the local Gram matrix and the global Gram matrix;

[0024] Compute the difference between the local Gram matrices of two backgrounds and the difference between the global Gram matrices:

[0025]

[0026]

[0027] Where jc(C, D) represents the difference between the local feature matrices of background C and background D, qc(C, D) represents the difference between the global feature matrices of background C and background D; jg (C，i，j ) represents the element value at the i-th row and j-th column in the local feature matrix of the background C, jg (D，i，j) Represents the element value of the i-th row and j-th column in the local feature matrix of the background D; qg (C，i，j ) table represents the element value of the i-th row and j-th column in the global feature matrix of the background C, qg (D，i，j) table represents the element value of the i-th row and j-th column position in the global feature matrix of the background D; I and J are the number of rows and columns respectively;

[0028] Calculate the similarity of the backgrounds of the two sample images: BS(C, D) = γ / jc(C, D) + δ / qc(C, D);

[0029] Among them, γ and δ represent the influence weights of the difference of local feature matrix and the difference of global feature matrix respectively.

[0030] Based on the above, when calculating the priority of each combination of information, the following steps are performed:

[0031] For sample images containing both background and foreground targets, the rationality of each combination of information is calculated using the formula Hp = QS (A, B);

[0032] For the sample image containing only the background, the rationality of each combination of information is calculated using the formula Hp=BS(A, B);

[0033] The rationality of each combination of information is normalized to obtain the priority of each combination of information.

[0034] Based on the above, the structure of the training discriminator is an encoder + fully connected layer structure, and the convolution and pooling layers in the encoder extract the features in the combined information to obtain a feature sequence. The fully connected layer network judges the confidence of the authenticity of the image based on the feature sequence.

[0035] The present invention also provides a training sample image enhancement system based on a neural network, comprising a manual annotation module and a combination splicing module.

[0036] The manual annotation module is used to manually annotate sample images in the initial sample set to obtain semantic segmentation labels;

[0037] The combined stitching module is used to segment the sample image using the semantic segmentation label, obtain the background and foreground objects, calculate the similarity of any two foreground objects, and calculate the similarity of any two backgrounds;

[0038] And for, in the order of sample images, for a sample image containing both background and foreground objects, replacing the background according to the similarity of the foreground object, and replacing the foreground object according to the similarity of the background, to obtain combined information; for a sample image containing only the background, adding the foreground object in the sample image similar to the background, to obtain combined information;

[0039] It is also used to calculate the preference of each combination information, select the pre-expanded number of combination information in descending order of preference, perform Poisson fusion on the corresponding area corresponding to each combination information, obtain the amplified image, and form an amplified sample set.

[0040] Based on the above, it also includes a training discriminator, wherein the training discriminator is used to judge the authenticity of the amplified image, and selects the amplified images with the result being true to form an amplified sample set;

[0041] The structure of the training discriminator is an encoder + fully connected layer structure, and the convolution and pooling layers in the encoder extract features in the combined information to obtain a feature sequence, and the fully connected layer network judges the confidence of the authenticity of the image based on the feature sequence.

[0042] The present invention has outstanding substantive features and significant progress compared to the prior art, specifically:

[0043] The present invention uses a small number of sample images currently collected to form an initial sample set, manually annotates the sample images in the initial sample set to obtain semantic labels, obtains the foreground objects and backgrounds corresponding to each label through semantic segmentation, and then amplifies more similar images that meet the requirements through splicing and fusion methods such as exchanging foreground objects and backgrounds in different sample images, thereby increasing the quantity and richness of sample data and reducing the corresponding manual annotation workload;

[0044] The present invention provides a training discriminator to screen out amplified images with obvious splicing traces, thereby improving the fusion effect and authenticity of the amplified images. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 It is a schematic diagram of the process of Example 1 of the present invention.

[0046] Figure 2 It is a schematic diagram of the effect of embodiment 1 of the present invention.

[0047] Figure 3 It is a schematic diagram of the flow chart of Example 2 of the present invention. DETAILED DESCRIPTION

[0048] The technical solution of the present invention is further described in detail below through specific implementation methods.

[0049] like Figure 1 As shown, a training sample image enhancement method based on a neural network includes the following steps:

[0050] Manually annotating sample images in the initial sample set to obtain an initial training set, wherein the initial training set includes semantic segmentation labels and sample images;

[0051] Use semantic segmentation labels to segment sample images and obtain background and foreground objects; calculate the similarity between any two foreground objects and any two backgrounds;

[0052] According to the order of sample images, for sample images containing both background and foreground targets, the background is replaced according to the similarity of the foreground target, and the foreground target is replaced according to the similarity of the background to obtain combined information; for sample images containing only background, the foreground target in the sample image similar to its background is added to obtain combined information;

[0053] The preference of each combination of information is calculated, and the combination of information of the pre-expanded quantity is selected in descending order of preference. The corresponding area corresponding to each combination of information is subjected to Poisson fusion to obtain an amplified image, which constitutes an amplified sample set.

[0054] The present invention utilizes the currently collected small amount, obtains the foreground target and background corresponding to each label through semantic segmentation, and then expands more similar images that meet the requirements through splicing and fusion methods such as exchanging the foreground target and background in different sample images, which not only increases the quantity and richness of sample data, but also reduces the corresponding manual labeling workload.

[0055] In the specific implementation, the sample images in the initial sample set are manually annotated as follows: an equal-sized, single-channel label image is established, and the pixels at the corresponding positions in the original RGB image are labeled as 0 for the background class, the pixels at the corresponding positions in the original RGB image are labeled as 1 for the class A target, and the pixels at the corresponding positions in the original RGB image are labeled as 2 for the class B target, and so on, and the corresponding values ​​are assigned in the label image according to the category to which the image pixels belong, until every pixel in the entire sample image is labeled. After completing the manual labeling, the sample image can be segmented using the semantic segmentation label to obtain the background and foreground targets.

[0056] Since there are many types of foreground targets, such as people, cars, buildings, animals, etc., they are all foreground targets, so a more general method is used to identify the similarity of foreground targets.

[0057] Specifically, the present invention compares the foreground target from two aspects, the internal and external aspects, namely the external edge contour similarity and the internal corner point similarity of the foreground target; that is, firstly the edge contour of the foreground target is obtained and the external edge contour similarity is compared, and then the corner point detection is performed on the target to obtain the description of the corner point, and the internal corner point similarity of the foreground target is judged according to the matching result of the corner point.

[0058] In the specific implementation, when calculating the similarity between any two foreground objects, the following steps are performed:

[0059] 1) Perform connected domain analysis on each foreground object to obtain the coordinate data of each point of the edge contour and the coordinate data of the center point of the region, and calculate the polar coordinates of the edge contour and the variance of the corresponding path length of each point on the edge with the center point of the region as the origin; calculate the similarity of the edge contours of two foreground objects based on the variance of the corresponding path length of each point on the edge of the foreground object;

[0060] Among them, the similarity of the edge contours of any two foreground objects is calculated using in, R i represents the distance from the i-th edge point of foreground target A to the center point of the region of foreground target A, N is the number of edge points of foreground target A, RU A is the average distance from the N edge points of foreground object A to the center point of the region of foreground object A; R j represents the distance from the jth edge point of foreground object B to the center point of the region of foreground object B, M is the number of edge points of foreground object B, RU B It is the average distance from the M edge points of foreground object B to the center point of the region of foreground object B.

[0061] 2) Perform corner point detection and description on the foreground target, match the corner points of two foreground targets, and calculate the similarity of the internal areas of the two foreground targets based on the corner point matching results.

[0062] Corner detection is a common method for image contrast matching in image stitching. Its logic is to describe the changes in the pixel values ​​around the image, and then compare the description values ​​for matching. For example, the ORB algorithm can be used to detect corners in the image area and describe them. Through this algorithm, the position and description of each corner point in the grayscale image can be obtained, and the description is the grayscale difference around each corner point.

[0063] After obtaining the description of the corner points, if the description of the corner points is consistent or meets the set error range, it is a successfully matched corner point pair. Then the similarity is judged based on the number of matched corner points and the positions of the successfully matched corner points.

[0064] In specific implementation, when calculating the similarity of the internal regions of two foreground objects based on the matching results of the corner points, the following steps are performed:

[0065] a. Calculate the corner point matching rate pl of any two foreground objects:

[0066] Among them, DS A Indicates the number of corner points of the foreground target A, DS B Indicates the number of corner points of foreground object B; PS indicates the number of successfully matched corner points of foreground object A and foreground object B; kd is the preset ratio parameter; the empirical value is 3.

[0067] The corner point matching rate can reflect the overall similarity of the internal area of ​​the foreground target.

[0068] b. Calculate the discreteness ls of the matching points:

[0069]

[0070] Among them, js(A) represents the number of angles of successfully matched corner points in foreground target A, and js(B) represents the number of angles of successfully matched corner points in foreground target B. For example, there are two successfully matched points in foreground target A, one is 10° and the other is 30°, then the number is 2; rs(A) represents the number of path length categories of successfully matched corner points in foreground target A, and rs(B) represents the number of path length categories of successfully matched corner points in foreground target B; max(R A，m ) represents the maximum value of the corresponding path length in the foreground object A; max(R B，m ) represents the maximum value of the corresponding path length in the foreground object B.

[0071] The formula is divided into two parts. The first part is the degree of dispersion in angle, that is, whether it is concentrated in a certain angle range. The second part is the degree of dispersion in path length, that is, whether it is concentrated in a certain distance range.

[0072] This part is designed to avoid the influence of the matching rate of local area and global area on the similarity. For example, area 1 is the upper body area of ​​the human body, and area 2 is the whole body area of ​​the human body. At this time, although the matching rate is high (because the upper body corner points match more and the matching points are too concentrated), the two areas are not very similar.

[0073] 3) Calculate the similarity of two foreground objects according to the formula QS(A, B)=α*bs(A, B)+β*ns(A, B), where QS(A, B) represents the similarity between foreground object A and foreground object B, bs(A, B) represents the edge contour similarity between foreground object A and foreground object B, ns(A, B) represents the internal area similarity between foreground object A and foreground object B, α and β represent the influence weights of edge contour similarity and internal area similarity, respectively, and the empirical values ​​are 0.45 and 0.55.

[0074] The present invention uses the color and texture style of the background area to perform similarity judgment, and the Gram matrix of the feature image obtained by the convolution operation of the image can well judge the style difference between the images. Therefore, the present invention needs to judge the Gram matrix of the feature images of the two background images to obtain the similarity of the background.

[0075] In specific implementation, when calculating the similarity between any two backgrounds, the following steps are performed:

[0076] i. Use the pre-trained VGG network to extract the local feature image and global feature image of the background of each sample image, and calculate the local Gram matrix and the global Gram matrix;

[0077] The trained VGG network can extract features from the processed images, that is, it can obtain the feature maps corresponding to each hidden layer through convolution and pooling operations with determined weights. The shallow feature images focus on the local similarity of the image, while the deep feature images focus on the similarity of the overall content of the image.

[0078] Therefore, the first M layers of convolution feature images are extracted as local features corresponding to the background image, and the last N layers of convolution feature images are extracted as global features corresponding to the background image.

[0079] Specifically, the steps for obtaining the local feature matrix are as follows: flatten the first M feature images to obtain the corresponding feature matrix, that is, each two-dimensional feature image is expanded into a one-dimensional feature vector as a row of data in the feature matrix; M feature vectors obtain M rows of data, which are combined to obtain the corresponding local feature matrix.

[0080] Similarly, the global feature matrix corresponding to the last N convolutional layers can be obtained.

[0081] ii. Calculate the difference between the local Gram matrix and the global Gram matrix of the background of the two sample images;

[0082] The feature matrix is ​​multiplied by its transposed matrix to obtain the corresponding Gram matrix. The mean square error formula can be used to find the difference between feature matrices, that is, each position in the matrix is ​​compared, the square of the difference is obtained, and then the average is calculated.

[0083] The following formula is used to calculate the difference between the local Gram matrices of two backgrounds and the difference between the global Gram matrices:

[0084]

[0085]

[0086] Where jc(C, D) represents the difference between the local feature matrices of background C and background D, qc(C, D) represents the difference between the global feature matrices of background C and background D; jg (C，i，j) Represents the element value at the i-th row and j-th column in the local feature matrix of the background C, jg (D，i，j) Represents the element value of the i-th row and j-th column in the local feature matrix of the background D; qg (C，i，j) The table represents the element value of the i-th row and j-th column in the global feature matrix of the background C, qg (D，i，j) Table represents the element value of the i-th row and j-th column position in the global feature matrix of the background D; I and J are the number of rows and columns respectively;

[0087] Calculate the similarity of the backgrounds of the two sample images: BS(C, D) = γ / jc(C, D) + δ / qc(C, D);

[0088] Among them, γ and δ represent the influence weights of the difference of local feature matrix and the difference of global feature matrix respectively.

[0089] Based on the above, when calculating the priority of each combination of information, the following steps are performed:

[0090] For sample images containing both background and foreground targets, the rationality of each combination of information is calculated using the formula Hp = QS (A, B);

[0091] Take two images with similar foreground and background objects as an example:

[0092] Image 1 is composed of: background 1+A1+B1+C1

[0093] Image 2 is composed of: background 2+A2+B2+C2

[0094] Then similar substitutions can be obtained:

[0095] Background 1+A2+B1+C1

[0096] Background 1+A1+B2+C1

[0097] Background 1+A1+B1+C2

[0098] Background 1+A2+B2+C1

[0099] Background 1+A1+B2+C2

[0100] Background 1+A2+B1+C2

[0101] Background 1+A2+B2+C2

[0102] …(same as background 2)

[0103] For images combined in this way, the rationality Hp is judged as follows:

[0104] Hp=QS(A,B)

[0105] That is, similar replacement. The rationality of this type of combined image is: the similarity of the replacement target. The more similar the replacement target is, the more reasonable it is.

[0106] Sometimes, in order to expand the richness of the data, we will collect the corresponding background, but no target images. At this time, we need to superimpose the corresponding foreground on the background.

[0107] For example: Image 1: Background 1 + A1

[0108] Image 2: Background 2

[0109] Then we can get: background 2+A1;

[0110] Therefore, for the sample image containing only the background, the rationality of each combination of information is calculated using the formula Hp = BS (A, B);

[0111] The rationality of each combination of information is normalized to obtain the priority of each combination of information.

[0112] by Figure 2 For example, Figure 2 The original image on the left side is 1 swimming dog, 2 swimming person, 3 sea water from top to bottom; superimposing the swimming dog 1 and the swimming person 2 on the sea water 3, we can get a new image, that is, Figure 2 The middle image has obvious superposition traces and poor image quality. Poisson fusion is performed on the combined superimposed images to obtain Figure 2 The image on the right has a better fusion effect and is more realistic, so it can be used as a new image to augment the training data.

[0113] Example 2

[0114] The difference between this embodiment and embodiment 1 is that: Figure 3 As shown, after the amplified image is formed, the authenticity of the amplified image is judged by the trained discriminator, and the amplified images with true results are selected to form an amplified sample set.

[0115] The combination information corresponding to each target image can be obtained by adopting the method described in the embodiment, but the combined image is likely to have obvious traces of splicing combination (simple superposition of images), which will affect the training effect of the subsequent network. The present invention uses a discriminator network to judge the authenticity of the fused image to screen out the amplified image with obvious splicing traces, thereby improving the fusion effect and authenticity of the amplified image.

[0116] It should be noted that the discriminator network of the present invention is implemented by a convolutional neural network. Compared with the above-mentioned network that needs to expand training data, the network has different mission objectives. The discriminator network is to distinguish the authenticity of images, while the target recognition network is to identify different targets from various scenes. Secondly, the difficulty of collecting training data is different. The discriminator network is more general and does not require images of specific scenes and targets, while the target detection network requires target images in different scenes, and it is difficult to collect a large number of images. Finally, the difficulty of annotation is also different. The discriminator only needs to manually annotate the confidence of the authenticity of the image, while the target detection network requires complex semantic segmentation labels.

[0117] The specific contents of the training discriminator of the present invention are as follows:

[0118] 1) The structure of the discriminator network is Encoder + FC (fully connected layer) structure, that is, the convolution and pooling layers in the encoder are used to extract features in the image to obtain a feature sequence, and then the feature sequence is used to make a confidence judgment on the authenticity of the image through a fully connected network.

[0119] 2) The input of the network is the image after combination and fusion according to the corresponding parameters, and the output of the network is the confidence level of the corresponding image authenticity.

[0120] 3) The labeling process is as follows: collect some images from real scenes, without any necessary requirements for the image content, and directly label these images as real images; collect some materials and simply superimpose them on the background, use the fusion algorithm to perform random parameter images, and then manually label the authenticity. Among them, images with obvious combination traces are marked as pseudo images [0,1], and images with good fusion effects are marked as real images [1,0].

[0121] 3) Using the above-mentioned annotated real scene images and combined images, the discriminator is trained. The loss function used by the network is the binary cross-entropy loss function, and the formula is as follows:

[0122]

[0123] Among them, y i is a binary label 0 or 1, p(y i ) is the output belonging to y i The probability of the label, N is the number of samples, and ω is the parameter to be optimized.

[0124] Furthermore, after using the trained discriminator to judge the authenticity of the amplified image, the image fusion parameters can be optimized according to the authenticity of the amplified image to achieve adaptive optimal parameter selection.

[0125] The parameters that need to be adjusted for Poisson fusion are adjusted within the corresponding range to obtain the image after the combination of each parameter, which is input into the above discriminator network, and the images with true results are selected and retained in the expanded data set.

[0126] After the fused image is obtained, the semantic labels corresponding to the fused image can be combined according to the semantic labels corresponding to each region and retained in the label set corresponding to the augmented data.

[0127] Example 3

[0128] This embodiment provides a training sample image enhancement system based on a neural network, including a manual annotation module and a combination splicing module.

[0129] The manual annotation module is used to manually annotate sample images in the initial sample set to obtain semantic segmentation labels;

[0130] The combined stitching module is used to segment the sample image using the semantic segmentation label, obtain the background and foreground objects, calculate the similarity of any two foreground objects, and calculate the similarity of any two backgrounds;

[0131] And for, in the order of sample images, for a sample image containing both background and foreground objects, replacing the background according to the similarity of the foreground object, and replacing the foreground object according to the similarity of the background, to obtain combined information; for a sample image containing only the background, adding the foreground object in the sample image similar to the background, to obtain combined information;

[0132] It is also used to calculate the preference of each combination information, select the pre-expanded number of combination information in descending order of preference, perform Poisson fusion on the corresponding area corresponding to each combination information, obtain the amplified image, and form an amplified sample set.

[0133] In a specific implementation, the training sample image enhancement system further includes a fusion optimization module, the fusion optimization module includes a training discriminator, the training discriminator is used to judge the authenticity of the augmented image, and selects augmented images with true results to form an augmented sample set;

[0134] The structure of the training discriminator is an encoder + fully connected layer structure, and the convolution and pooling layers in the encoder extract features in the combined information to obtain a feature sequence, and the fully connected layer network judges the confidence of the authenticity of the image based on the feature sequence.

[0135] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, ordinary technicians in the field should understand that the specific implementation methods of the present invention can still be modified or some technical features can be replaced by equivalents without departing from the spirit of the technical solution of the present invention, which should be included in the scope of the technical solution for protection of the present invention.

Claims

1. A training sample image enhancement method based on a neural network, characterized in that: The following steps are involved: Manually annotate the sample images in the initial sample set to obtain semantic segmentation labels; Use semantic segmentation labels to segment sample images and obtain background and foreground objects; Calculate the similarity between any two foreground objects and any two background objects; According to the order of sample images, for sample images containing both background and foreground targets, the background is replaced according to the similarity of the foreground target, and the foreground target is replaced according to the similarity of the background, so as to obtain combined information; For a sample image containing only the background, add the foreground object in the sample image that is similar to its background to obtain the combined information; The preference of each combination of information is calculated, and the combination of information of the pre-expanded quantity is selected in descending order of preference. The corresponding area corresponding to each combination of information is subjected to Poisson fusion to obtain an amplified image, which constitutes an amplified sample set.

2. The method for enhancing training sample images based on a neural network according to claim 1, characterized in that: When calculating the similarity between any two foreground objects, the following steps are performed: Connected domain analysis is performed on each foreground object to obtain the coordinate data of each point of the edge contour and the coordinate data of the center point of the region. The polar coordinates of the edge contour and the variance of the corresponding path length of each point on the edge are calculated with the center point of the region as the origin; the similarity of the edge contours of two foreground objects is calculated based on the variance of the corresponding path length of each point on the edge of the foreground object; Perform corner point detection and description on the foreground object, match the corner points of two foreground objects, and calculate the similarity of the internal areas of the two foreground objects based on the matching results of the corner points; The similarity of two foreground targets is calculated according to the formula QS(A, B)=α*bs(A, B)+β*ns(A, B), where QS(A, B) represents the similarity between foreground target A and foreground target B, bs(A, B) represents the edge contour similarity between foreground target A and foreground target B, ns(A, B) represents the internal area similarity between foreground target A and foreground target B, and α and β represent the influence weight of edge contour similarity and the influence weight of internal area similarity, respectively.

3. The method for enhancing training sample images based on a neural network according to claim 2, characterized in that: use Calculate the similarity of the edge contours of any two foreground objects, where R i represents the distance from the i-th edge point of foreground target A to the center point of the region of foreground target A, N is the number of edge points of foreground target A, RU A is the average distance from the N edge points of foreground object A to the center point of the region of foreground object A; R j represents the distance from the jth edge point of foreground object B to the center point of the region of foreground object B, M is the number of edge points of foreground object B, RU B It is the average distance from the M edge points of foreground object B to the center point of the region of foreground object B.

4. The method for enhancing training sample images based on a neural network according to claim 2, characterized in that: When calculating the similarity of the internal regions of two foreground objects based on the matching results of the corner points, the following steps are performed: Calculate the corner point matching rate pl of any two foreground targets: Among them, DS A Indicates the number of corner points of the foreground target A, DS B Indicates the number of corner points of foreground object B; PS indicates the number of successfully matched corner points of foreground object A and foreground object B; kd is the preset ratio parameter; Calculate the discreteness ls of the matching points: Where js(A) represents the number of angles of successfully matched corner points in foreground target A, and js(B) represents the number of angles of successfully matched corner points in foreground target B; rs(A) represents the number of path length categories of successfully matched corner points in foreground target A, and rs(B) represents the number of path length categories of successfully matched corner points in foreground target B; max(R A，m ) represents the maximum value of the corresponding path length in the foreground object A; max(R B，m ) represents the maximum value of the corresponding path length in the foreground object B.

5. The method for enhancing training sample images based on a neural network according to claim 1, characterized in that: When calculating the similarity between any two backgrounds, the following steps are performed: Use the pre-trained VGG network to extract the local feature image and global feature image of the background in each sample image, and calculate the local Gram matrix and the global Gram matrix; Compute the difference between the local Gram matrices of two backgrounds and the difference between the global Gram matrices: Where jc(C, D) represents the difference between the local feature matrices of background C and background D, qc(C, D) represents the difference between the global feature matrices of background C and background D; jg (C，i，j) Represents the element value at the i-th row and j-th column in the local feature matrix of the background C, jg (D，i，j) Represents the element value of the i-th row and j-th column in the local feature matrix of the background D; qg (C，i，j) The table represents the element value of the i-th row and j-th column in the global feature matrix of the background C, qg (D，i，j) Table represents the element value of the i-th row and j-th column position in the global feature matrix of the background D; I and J are the number of rows and columns respectively; Calculate the similarity of the backgrounds of the two sample images: BS(C, D) = γ / jc(C, D) + δ / qc(C, D); Among them, γ and δ represent the influence weights of the difference of local feature matrix and the difference of global feature matrix respectively.

6. The method for enhancing training sample images based on a neural network according to claim 1, characterized in that: When calculating the preference of each combination of information, the following steps are performed: For sample images containing both background and foreground targets, the rationality of each combination of information is calculated using the formula Hp = QS (A, B); For the sample image containing only the background, the rationality of each combination of information is calculated using the formula Hp=BS(A, B); The rationality of each combination of information is normalized to obtain the priority of each combination of information.

7. The method for enhancing training sample images based on a neural network according to claim 1, characterized in that: After obtaining the amplified sample set, the semantic labels corresponding to each region in each amplified sample image are combined to obtain the semantic labels corresponding to the amplified sample image, and the labels are retained in the amplified label set corresponding to the amplified sample set.

8. The method for enhancing training sample images based on a neural network according to claim 1, characterized in that: After the augmented image is formed, the authenticity of the augmented image is judged by the trained discriminator, and the augmented images with true results are selected to form an augmented sample set; The structure of the training discriminator is an encoder + fully connected layer structure, and the convolution and pooling layers in the encoder extract features in the combined information to obtain a feature sequence, and the fully connected layer network judges the confidence of the authenticity of the image based on the feature sequence.

9. A training sample image enhancement system based on a neural network, characterized in that: Manual annotation module and combination splicing module, The manual annotation module is used to manually annotate sample images in the initial sample set to obtain semantic segmentation labels; The combined stitching module is used to segment the sample image using the semantic segmentation label, obtain the background and foreground objects, calculate the similarity of any two foreground objects, and calculate the similarity of any two backgrounds; and for replacing the background of a sample image containing both background and foreground objects according to the similarity of the foreground objects and replacing the foreground objects according to the similarity of the background in the order of the sample images, thereby obtaining combined information; For a sample image containing only the background, add the foreground object in the sample image that is similar to its background to obtain the combined information; It is also used to calculate the preference of each combination information, select the pre-expanded number of combination information in descending order of preference, perform Poisson fusion on the corresponding area corresponding to each combination information, obtain the amplified image, and form an amplified sample set.

10. The neural network-based training sample image enhancement system according to claim 9, characterized in that: It also includes a training discriminator, which is used to judge the authenticity of the amplified image and select the amplified images with true results to form an amplified sample set; The structure of the training discriminator is an encoder + fully connected layer structure, and the convolution and pooling layers in the encoder extract features in the combined information to obtain a feature sequence, and the fully connected layer network judges the confidence of the authenticity of the image based on the feature sequence.

Images