Image synthesis method and system

Abstract

The application provides an image synthesis method and system, the method comprising: obtaining a target augmentation strategy for image synthesis of an original image based on a training mode of adversarial reinforcement learning between a first policy controller and a second policy controller; and performing image synthesis on the original image based on the target augmentation strategy to obtain a target image. The system executes the method. The application obtains an optimal abnormal sample synthesis strategy, i.e., a target augmentation strategy, for image synthesis of an original image by designing a training mode of adversarial reinforcement learning between two policy controllers, solves the problem of overfitting of a detection model trained based on a target image due to a single abnormal sample (i.e., target image) synthesis mode, and improves the generalization ability of the detection model.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer, and in particular, to an image synthesis method and system. BACKGROUND

[0002] Defect detection is an important link in the industrial production process. In recent years, with the continuous improvement of the performance of image processing units, deep learning related technologies have been fully developed. Many methods realize defect detection through deep learning, and most of the work belongs to the category of supervised learning, that is, a large number of defect samples labeled by artificial need to be used to train the model. However, the collection of defect samples will face the following problems in many industrial scenarios: (1) sample collection is difficult. The probability of occurrence of defective products is generally low, and the time cost of collecting and labeling a sufficient number of samples is high. (2) For the same production line, the model of the product may change frequently, making it difficult for the supervised learning model to quickly adapt to the new detection task. (3) The model often has difficulty accurately detecting new defect types in the test phase. The essential reason for the above problems is that the performance of the supervised detection model is limited by the number and shape of the training set samples. From the perspective of artificial detection, by memorizing the shape of the normal sample, the quality inspection worker can easily find various structural abnormal defect areas. Therefore, how to make the detection model get rid of the dependence on defect samples and use normal samples to train an unsupervised model to locate the defect area has become a key means and a popular research topic in the intelligent detection of industrial product appearance.

[0003] Most of the current unsupervised-based defect detection methods will use normal samples to train the model, and in the test phase, the difference between the normal sample features is compared to locate the defect area. However, the accuracy of the model trained only with normal samples is often insufficient, and a common improvement idea is to artificially synthesize a series of artificial abnormal samples based on the collection of a certain amount of normal samples to participate in the training, so as to train the model to distinguish between normal image texture features and potential abnormal texture features.

[0004] Introducing artificial abnormal samples in the training process of the unsupervised defect detection model can effectively improve the generalization ability of the model. The current mainstream method relies on artificially designed data augmentation strategies to synthesize abnormal samples. A common idea is to first select a region that needs to be synthesized in a normal image by two-dimensional Bernoulli noise, and the abnormal texture structure that needs to be filled can be randomly selected from an unrelated image dataset to construct an abnormal texture, or the normal image can be processed by common deformation or brightness and contrast change to construct an abnormal texture.

[0005] And in the training process by letting the network adapt to various abnormal structure in advance, the model can have better recognition ability to the potential abnormal defect target in the actual test stage, and in this training method, the effect of the model in the test stage has high correlation with the quality of the abnormal sample synthesis.

[0006] However, according to the effect of the existing method and the inventor's previous experiments, the model trained by using artificial abnormal samples has the risk of overfitting. The model can achieve accurate reconstruction and defect positioning for the images in the training stage, but the effect on the actual defect samples is obviously worse. This is mainly because the use of fixed synthesis strategy for sample synthesis greatly limits the diversity of the synthesized samples in the abnormal area texture and abnormal area shape, the abnormal texture pattern is single and easy to learn, which leads to insufficient training of the reconstruction and discrimination network, so that it is difficult to adapt to the actual defect sample. SUMMARY

[0007] The image synthesis method and system provided by the application are used to solve the problem of overfitting of the detection model trained based on the abnormal sample due to the single synthesis mode of the abnormal sample in the prior art.

[0008] The image synthesis method provided by the application comprises:

[0009] Based on the training mode of the adversarial reinforcement learning between the first strategy controller and the second strategy controller, a target augmentation strategy for image synthesis of the original image is obtained, the first strategy controller is used to input the original image and output a first augmentation strategy, and the second strategy controller is used to input the original image and output a second augmentation strategy;

[0010] Based on the target augmentation strategy, the original image is synthesized to obtain a target image.

[0011] According to the image synthesis method provided by the application, the training mode of the adversarial reinforcement learning between the first strategy controller and the second strategy controller is used to obtain the target augmentation strategy for image synthesis of the original image, which comprises:

[0012] According to a first evaluation index of the first policy controller, a first augmented policy output by the first policy controller is updated until the first evaluation index is greater than a preset value, the first augmented policy is obtained by inputting the original image into the first policy controller, the first evaluation index is determined according to a detection result of a first training sample by a first detection model, the first training sample is obtained by image synthesis on the original image based on the first augmented policy, the first detection model is obtained by training a first preset detection model based on a second training sample, and the second training sample is obtained by image synthesis on the original image based on a second augmented policy output by a second policy controller;

[0013] According to a second evaluation index of the second policy controller, a second augmented policy output by the second policy controller is updated until the second evaluation index is greater than the preset value, the second evaluation index is determined according to a detection result of the second training sample by a second detection model, and the second detection model is obtained by training a second preset detection model based on the first training sample.

[0014] According to the updated first augmented policy and the updated second augmented policy, the target augmented policy is obtained.

[0015] According to the image synthesis method provided by the application, the first evaluation index and the second evaluation index are obtained in the following manner:

[0016] According to a first accuracy rate of a detection result of the second training sample by the second detection model, a second accuracy rate of a detection result of the first training sample by the first detection model, mutual information of the first training sample and the second training sample, and intra-class dispersion of the first training sample, the first evaluation index is obtained.

[0017] According to a first accuracy rate of a detection result of the first training sample by the first detection model, a second accuracy rate of a detection result of the second training sample by the second detection model, mutual information of the second training sample and the first training sample, and intra-class dispersion of the second training sample, the second evaluation index is obtained.

[0018] According to the image synthesis method provided by the application, the original image is synthesized based on the target augmented policy to obtain a target image, which comprises:

[0019] The original image is synthesized based on the updated first augmented policy and the updated second augmented policy to obtain the target image.

[0020] According to the image synthesis method provided by the application, the original image is synthesized based on the updated first augmentation strategy and the updated second augmentation strategy to obtain the target image, and the method comprises the following steps:

[0021] The original image is synthesized based on the updated first augmentation strategy and the updated second augmentation strategy to obtain an original synthesized image.

[0022] The augmentation probability corresponding to each region in the original image is obtained, and the augmentation probability is used to represent the probability that the corresponding region in the original synthesized image is retained.

[0023] The target image is obtained according to the augmentation probability and the original synthesized image.

[0024] According to the image synthesis method provided by the application, the augmentation probability corresponding to each region in the original image is obtained, and the method comprises the following steps:

[0025] The original image is divided into a plurality of regions, and the image features of each region are obtained.

[0026] The augmentation probability of each region is determined according to the correlation between the image features of each region.

[0027] The application further provides an image synthesis system, comprising:

[0028] An acquisition module is configured to acquire a target augmentation strategy for image synthesis of an original image based on a training mode of adversarial reinforcement learning between a first strategy controller and a second strategy controller, the first strategy controller being configured to input the original image and output a first augmentation strategy, and the second strategy controller being configured to input the original image and output a second augmentation strategy.

[0029] A synthesis module is configured to synthesize the original image based on the target augmentation strategy to obtain a target image.

[0030] The application further provides an electronic device comprising a processor and a memory storing a computer program, wherein the processor implements the image synthesis method according to any one of the above-mentioned image synthesis methods when executing the program.

[0031] The application further provides a non-transitory computer-readable storage medium storing a computer program, wherein the computer program is executed by a processor to implement the image synthesis method according to any one of the above-mentioned image synthesis methods.

[0032] The application further provides a computer program product comprising a computer program, wherein the computer program is executed by a processor to implement the image synthesis method according to any one of the above-mentioned image synthesis methods.

[0033] The image synthesis method and system provided by the application obtain an optimal abnormal sample synthesis strategy, that is, a target augmentation strategy, for image synthesis of an original image by designing a training mode of adversarial reinforcement learning between two strategy controllers, solve the problem of overfitting of a detection model trained based on a target image due to a single abnormal sample (that is, target image) synthesis mode, and improve the generalization ability of the detection model. BRIEF DESCRIPTION OF DRAWINGS

[0034] In order to more clearly illustrate the technical solutions in the application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or the prior art description. Obviously, the drawings in the following description are some embodiments of the application, and other drawings can be obtained by those skilled in the art without creative labor on the basis of these drawings.

[0035] Figure 1 is a flowchart of the image synthesis method provided by the application;

[0036] Figure 2 is a schematic diagram of a double-strategy controller sample synthesis algorithm based on adversarial reinforcement learning provided by the application;

[0037] Figure 3 is a schematic diagram of a classical strategy search algorithm provided by the application;

[0038] Figure 4 is a schematic diagram of a target image synthesis process provided by the application;

[0039] Figure 5 is a schematic diagram of a double-strategy controller optimization process provided by the application;

[0040] Figure 6 is a schematic diagram of a strategy controller based on regional probability control provided by the application;

[0041] Figure 7 is a structural schematic diagram of an image synthesis system provided by the application;

[0042] Figure 8 is a schematic diagram of the physical structure of an electronic device provided by the application. DETAILED DESCRIPTION

[0043] In order to make the objects, technical solutions and advantages of the application clearer, the technical solutions in the application will be described clearly and completely below in combination with the drawings in the application. Obviously, the described embodiments are some embodiments of the application, not all embodiments. Based on the embodiments in the application, all other embodiments obtained by those skilled in the art without creative labor are within the protection scope of the application.

[0044] Figure 1 is a flowchart of the image synthesis method provided by the present application, as shown in Figure 1 The image synthesis method provided by the present application can specifically include the following steps:

[0045] In step 110, based on the training mode of the adversarial reinforcement learning between the first policy controller and the second policy controller, a target augmentation policy for image synthesis of the original image is obtained, the first policy controller is used for inputting the original image and outputting a first augmentation policy, and the second policy controller is used for inputting the original image and outputting a second augmentation policy.

[0046] In step 120, based on the target augmentation policy, the original image is subjected to image synthesis to obtain a target image.

[0047] It should be noted that the execution subject of the above method can be a computer device.

[0048] Optionally, the image synthesis method provided by the present application is specifically to synthesize an original image (i.e. a normal sample) into a target image (i.e. an abnormal sample) through an image synthesis algorithm based on adversarial reinforcement learning, and the core target is to improve the generalization ability of a detection model trained based on the target image by adaptively synthesizing diversified artificial abnormal samples, and the overall framework of the algorithm is as shown in Figure 2 The flow of the algorithm will be described in detail.

[0049] Referring to Figure 2 The present application designs two augmentation policy controllers (referred to as policy controllers) based on regional probability control: augmentation policy controller A and augmentation policy controller B, which correspond to the first policy controller and the second policy controller, respectively.

[0050] Based on the training mode of the adversarial reinforcement learning between the first policy controller and the second policy controller, the best abnormal sample synthesis policy (i.e. the target augmentation policy) for image synthesis of the original image is searched from the policy search space. The policy search space is mainly composed of the augmentation policy (i.e. the first augmentation policy) output by the first policy controller according to the input original image, the augmentation policy (i.e. the second augmentation policy) output by the second policy controller according to the input original image, the probability and amplitude corresponding to the first augmentation policy, and the probability and amplitude corresponding to the second augmentation policy.

[0051] Optionally, the first augmentation policy and the second augmentation policy can specifically include an abnormal morphology generation algorithm, an abnormal texture generation algorithm composed of traditional image augmentation and irrelevant dataset images, and a Poisson image fusion algorithm.

[0052] Based on the target augmentation strategy, the original image (i.e. normal sample) is synthesized to obtain a synthesized abnormal sample.

[0053] It should be noted that the strategy controller used in the present application can be a regression model, for example, a regression model generated based on Resnet-18.

[0054] The image synthesis method provided by the present application designs an adversarial reinforcement learning training mode between two strategy controllers to obtain an optimal abnormal sample synthesis strategy for image synthesis of the original image, i.e. a target augmentation strategy, solves the problem of overfitting of the detection model trained based on the target image due to the single synthesis mode of the abnormal sample (i.e. the target image), and improves the generalization ability of the detection model.

[0055] Further, in one embodiment, the training mode based on adversarial reinforcement learning between the first strategy controller and the second strategy controller to obtain the target augmentation strategy for image synthesis of the original image can specifically include:

[0056] According to the first evaluation index of the first strategy controller, the first augmentation strategy output by the first strategy controller is updated until the first evaluation index is greater than a preset value, the first augmentation strategy is obtained by inputting the original image into the first strategy controller, the first evaluation index is determined according to the detection result of the first detection model on the first training sample, the first training sample is obtained by image synthesis of the original image based on the first augmentation strategy, and the first detection model is obtained by training a first preset detection model based on a second training sample, the second training sample is obtained by image synthesis of the original image based on the second augmentation strategy output by the second strategy controller;

[0057] According to the second evaluation index of the second strategy controller, the second augmentation strategy output by the second strategy controller is updated until the second evaluation index is greater than the preset value, the second evaluation index is determined according to the detection result of the second detection model on the second training sample, and the second detection model is obtained by training a second preset detection model based on the first training sample;

[0058] According to the updated first augmentation strategy and the updated second augmentation strategy, the target augmentation strategy is obtained.

[0059] Optionally, considering that the original strategy search space is too large, in order to reduce the search time of the strategy, the augmented strategy control vector designed by the application no longer controls the probability and amplitude of each type of augmented strategy respectively, but uniformly sets the effect probability of each augmented strategy, and only selects the corresponding interval for uniform sampling to determine the final augmented amplitude in the amplitude selection process. And in the interval setting process, part of the lower amplitude will be excluded according to the actual detection task demand, so as to avoid overfitting caused by the abnormal sample too close to the normal sample.

[0060] Figure 3 is a schematic diagram of a classical strategy search algorithm provided by the application, referring to Figure 3 , the application is improved on the basis of the existing image augmentation strategy search algorithm based on reinforcement learning. The strategy controller outputs a group of values according to the input original image, which corresponds to the type, probability and amplitude of the selected augmented strategy and other information. Then the generated augmented strategy is used to train the sub-model. After training, the evaluation is carried out on the validation set (i.e. the abnormal sample set generated by the strategy controller), and the evaluation index is used as the reward value feedback to update the parameters of the strategy controller. After the sub-model converges, a group of good augmented strategies on the existing data set can be obtained.

[0061] Among them, the sub-model can specifically include a first preset detection model and a second preset detection model, for example, a segmentation model U-net. Whether the sub-model converges can be judged according to whether the preset training times are reached or whether the reward value is greater than the preset value. When the preset training times are reached or the reward value is greater than the preset value, it is determined that the sub-model converges.

[0062] As can be seen from the above basic search algorithm, the validation set is needed to evaluate the effectiveness of the augmented strategy. However, in the unsupervised defect detection task, real defect samples cannot be used to construct the validation set. Therefore, the application designs a double-controller sample synthesis framework based on adversarial reinforcement learning, so that the detection model obtained under the augmented strategy output by a certain strategy controller is evaluated on the abnormal samples synthesized by another strategy controller. The effectiveness of the augmented strategy is evaluated in an adversarial training mode, and finally converges to two groups of optimal augmented strategies, as shown in Figure 4 and Figure 5 .

[0063] As shown in Figure 4 , the application based on the augmented strategy output by the strategy controller mainly includes three parts of the abnormal morphology generation algorithm based on Berlin Perlin noise, the abnormal texture generation algorithm composed of traditional image augmentation and irrelevant data set images, and the Poisson image fusion algorithm. The strategy search space is composed of parameters such as probability and amplitude involved in the three algorithms.

[0064] AsFigure 5 As shown, in order to evaluate the effectiveness of a certain augmented strategy, the first augmented strategy generated by the first strategy controller A is used to perform image synthesis on the original image to obtain a synthesized abnormal sample X A The second preset detection model is trained as a training sample, i.e., a first training sample, to obtain a first detection model M A The second augmented strategy generated by the second strategy controller B is used to perform image synthesis on the original image to obtain a synthesized abnormal sample X B The first preset detection model is trained as a training sample, i.e., a second training sample, to obtain a second detection model M B .

[0065] According to the reward value R A (or evaluation index), i.e., a first evaluation index, of the first strategy controller A, the first augmented strategy output by the first strategy controller A is updated until the first evaluation index (i.e., the reward value R A ) is greater than a preset value, and according to the reward value R B (i.e., a second evaluation index) of the second strategy controller B, the second augmented strategy output by the second strategy controller B is updated until the second evaluation index (i.e., the reward value R B ) is greater than a preset value. The first evaluation index is determined according to the detection result of the first training sample by the first detection model M B , and the second evaluation index is determined according to the detection result of the second training sample by the second detection model M A .

[0066] The second detection model M A trained by the strategy controller A is verified on the abnormal sample set (i.e., the second training sample) X B synthesized by the strategy controller B, and the higher the accuracy of the second detection model M A in the abnormal detection of the second training sample X B , the higher the generalization ability of the first augmented strategy output by the strategy controller A, and the higher the reward value given to the strategy controller A, and the lower the reward value given to the strategy controller B, so as to force the strategy controller B to optimize its synthesis strategy. Meanwhile, the first detection model M B obtained by the strategy controller B also undergoes the same verification process as described above, so as to construct an adversarial training framework between the two strategy controllers, until the evaluation indexes of the strategy controller A and the strategy controller are greater than the preset value, and the updated first augmented strategy output by the strategy controller A and the updated second augmented strategy output by the strategy controller B are obtained when the evaluation indexes of the strategy controller A and the strategy controller are greater than the preset value.

[0067] According to the updated first augmented strategy output by the policy controller A and the updated second augmented strategy output by the policy controller B, a target augmented strategy for image synthesis of the original image is obtained.

[0068] The image synthesis method provided by the application obtains an optimal abnormal sample synthesis strategy through the design of the training mode of the adversarial reinforcement learning between the two policy controllers, and improves the generalization ability of the detection model through adaptive search of the optimal abnormal sample synthesis strategy.

[0069] Further, in one embodiment, the first evaluation index and the second evaluation index can be obtained in the following manner:

[0070] According to the first accuracy of the detection result of the second training sample by the second detection model, the second accuracy of the detection result of the first training sample by the first detection model, the mutual information of the first training sample and the second training sample, and the intra-class dispersion of the first training sample, the first evaluation index is obtained;

[0071] According to the first accuracy of the detection result of the first training sample by the first detection model, the second accuracy of the detection result of the second training sample by the second detection model, the mutual information of the second training sample and the first training sample, and the intra-class dispersion of the second training sample, the second evaluation index is obtained.

[0072] Optionally, in order to further avoid the two policy controllers from improving their respective reward values by constructing the same or extremely single augmented strategy, the application introduces the intra-class difference and inter-class difference restrictions of two data distributions (training samples X A and X B ), increases the difference between the augmented strategies output by the two policy controllers by calculating and restricting the mutual information between the two data distributions, and combines the intra-class dispersion to improve the sample diversity of the two groups of data respectively, thereby avoiding the detection model from converging to a trivial solution. Based on this, the following reward value is designed for each policy controller in the application:

[0073] According to the first accuracy Acc(M A , X B ) of the detection result of the second training sample X A by the second detection model M B , the second accuracy Acc(M B , X A ) of the detection result of the first training sample X B by the first detection model M A , the mutual information I(X A , X B ) of the first training sample X A,X B ) and the first training sample X A intraclass divergence The first evaluation metric was obtained.

[0074] According to the first detection model M B For the first training sample X A The first accuracy of the test results Acc(M) B ,X A ), second detection model M A For the second training sample X B The second accuracy of the detection results Acc(M) A ,X B Mutual information I(X) between the second training sample and the first training sample B ,X A and the intra-class divergence of the second training sample The second evaluation metric was obtained.

[0075]

[0076]

[0077] Where Acc(M,X) represents the localization accuracy of detection model M on artificially generated anomaly samples (i.e., training samples) with data distribution X, that is, the accuracy of detection model M in detecting training samples X, I represents the mutual information calculation process, used to measure the correlation between two data distributions, S X This describes the within-class divergence of distribution X, where α, β, and γ represent the three weight coefficients. Regarding the optimization process of the policy controllers, this invention employs a classic robust adversarial reinforcement learning algorithm to optimize the two controllers. Specifically, the augmented policies of policy controllers A and B are alternately frozen and optimized separately until convergence. Subsequently, the augmented policies obtained by the two policy controllers are recorded and applied to the actual training process with equal probability.

[0078] The image synthesis method provided by this invention introduces intra-class and inter-class variability constraints on two data distributions (training samples XA and XB), increases the augmented policy variability of the outputs of the two policy controllers by calculating and constraining the mutual information between the two data distributions, and improves the sample diversity of the two sets of data by combining intra-class divergence, thus avoiding the detection model from eventually converging to a trivial solution.

[0079] Furthermore, in one embodiment, the step of performing image synthesis on the original image based on the target augmentation strategy to obtain the target image may specifically include:

[0080] perform image synthesis on the original image based on the updated first augmentation strategy and the updated second augmentation strategy to obtain the target image.

[0081] Optionally, an augmented strategy vector (including the type, probability, and amplitude of the updated first / second augmentation strategy) is obtained using the obtained updated first augmentation strategy and updated second augmentation strategy, image synthesis is performed on the original image to generate an abnormal morphology and an abnormal texture of the original image, and Poisson fusion is performed to obtain the target image.

[0082] Further, in an embodiment, the performing image synthesis on the original image based on the updated first augmentation strategy and the updated second augmentation strategy to obtain the target image can specifically include:

[0083] performing image synthesis on the original image based on the updated first augmentation strategy and the updated second augmentation strategy to obtain an original synthesis image;

[0084] obtaining an augmentation probability corresponding to each region in the original image, the augmentation probability being used to represent a probability that a corresponding region in the original synthesis image is retained;

[0085] obtaining the target image according to the augmentation probability and the original synthesis image.

[0086] Further, in an embodiment, the obtaining the augmentation probability corresponding to each region in the original image can specifically include:

[0087] dividing the original image into a preset number of regions and obtaining an image feature of each region;

[0088] determining an augmentation probability of each region according to a correlation between the image features of each region.

[0089] Optionally, considering that, in defect detection, most of the samples to be detected contain both background and foreground regions, for a background region with relatively simple texture, positioning of an abnormal region can be basically achieved without excessive training, and for a foreground region, more training is needed to distinguish abnormal structures because the structure of the foreground region is more complex, and defects have a higher probability of occurring in the foreground region. Therefore, in unsupervised defect detection model training, more attention needs to be paid to the synthesis of abnormal samples in the foreground region.

[0090] However, the traditional image augmentation strategy search method generally only considers how to augment from the perspective of the whole image, without determining the augmentation strategy in detail according to the characteristics of different regions of the image. Therefore, the application designs an augmentation strategy controller based on region probability control, which adaptively adjusts the generation probability of abnormal structures in different image regions, and the process is as shown in Figure 6

[0091] As shown in Figure 6 , the designed augmentation strategy controller mainly generates two kinds of control information. One is the augmentation strategy vector (including the type, probability and amplitude of the augmentation strategy) output by the strategy controller (including the image feature encoding module and the strategy control module), which determines the abnormal morphology, abnormal texture in the image synthesis process and various parameters in the image fusion process. The other is the augmentation probability generated by the region feature encoding module and the region probability control module, which is used to adjust the probability of the synthesized abnormal structure being retained in different image regions.

[0092] In order to control the generation probability of abnormal structures in the foreground and background regions respectively, the application first uniformly divides the input original image into a preset number of regions, for example, p x p regions, and obtains a total of p 2 image features in each local region of the original image based on the region feature encoding module (Resnet-18 model). Then, the correlation between the image features of different regions is analyzed by using a self-attention mechanism, for example, the correlation between the image features can be obtained by calculating the cosine distance, Euclidean distance, etc. between the image features of different regions. Considering that for the foreground region, there is a strong correlation between the regions, which are related context information, and the uniformly distributed background region has low correlation with other regions due to insufficient semantic information, therefore, based on the above characteristics, a region probability control module based on a self-attention mechanism is designed to assign higher augmentation probability to regions with stronger correlation (such as the foreground region), to obtain an augmentation probability map, and the corresponding region is more likely to retain the generated abnormal structure, so as to realize the detailed synthesis strategy control based on different region features.

[0093] Specifically, the augmentation strategy corresponding to the augmentation strategy vector obtained by the image encoding module and the strategy control module in the two strategy controllers is used to synthesize the normal sample (i.e. the original image) to obtain an original synthesis image.

[0094] By dividing the original image into a preset number of regions and obtaining the image features of each region based on the image feature encoding module, the augmentation probability of each region is obtained according to the correlation between the image features of each region, and the probability of the corresponding region being retained in the synthesized original synthesis image can be obtained according to the augmentation probability.

[0095] ​According to the augmented probability and the original composite image, a final composite image, i.e., a target image, is obtained.

[0096] The image synthesis method provided by the application controls the strategy controller based on the region probability control, so that the detection model pays more attention to the detection task in the foreground target region to be detected, and the network automatically learns the foreground and background of the image, and the generation probability of the abnormal sample in different image regions is controlled to strengthen the detection effect of the detection model on the complex foreground target.

[0097] The image synthesis system provided by the application is described below, and the image synthesis system described below can be correspondingly referred to the image synthesis method described above.

[0098] Figure 7 The image synthesis system provided by the application is described below, and the image synthesis system described below can be correspondingly referred to the image synthesis method described above. Figure 7 As shown in the figure, it comprises:

[0099] The acquisition module 710 is configured to acquire a target augmentation strategy for image synthesis of the original image based on an adversarial reinforcement learning training mode between the first strategy controller and the second strategy controller, the first strategy controller being configured to input the original image and output a first augmentation strategy, and the second strategy controller being configured to input the original image and output a second augmentation strategy.

[0100] The synthesis module 711 is configured to perform image synthesis on the original image based on the target augmentation strategy to obtain a target image.

[0101] The image synthesis system provided by the application acquires the best abnormal sample synthesis strategy, i.e., the target augmentation strategy, for image synthesis of the original image by designing an adversarial reinforcement learning training mode between the two strategy controllers, solves the overfitting problem of the detection model trained based on the target image due to the single abnormal sample (i.e., the target image) synthesis mode, and improves the generalization ability of the detection model.

[0102] Figure 8 The image synthesis system provided by the application acquires the best abnormal sample synthesis strategy, i.e., the target augmentation strategy, for image synthesis of the original image by designing an adversarial reinforcement learning training mode between the two strategy controllers, solves the overfitting problem of the detection model trained based on the target image due to the single abnormal sample (i.e., the target image) synthesis mode, and improves the generalization ability of the detection model. Figure 8 As shown in the figure, the electronic device can include a processor 810, a communication interface 811, a memory 812 and a bus 813, wherein the processor 810, the communication interface 811 and the memory 812 can communicate with each other through the bus 813. The processor 810 can call the logic instructions in the memory 812 to execute the following method:

[0103] Based on a training mode of adversarial reinforcement learning between the first policy controller and the second policy controller, a target augmentation policy for image synthesis on the original image is obtained, the first policy controller is used for inputting the original image and outputting a first augmentation policy, and the second policy controller is used for inputting the original image and outputting a second augmentation policy.

[0104] Based on the target augmentation policy, the original image is subjected to image synthesis to obtain a target image.

[0105] In addition, the logical instructions in the memory described above can be implemented in the form of a software function unit and sold or used as a stand-alone product, and can be stored in a computer-readable storage medium. Based on such understanding, the technical solutions of the present application essentially or the parts that contribute to the prior art or parts of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes a plurality of instructions for causing a computer power supply screen (which can be a personal computer, a server, or a network power supply screen) to execute all or part of the steps of the method described in the various embodiments of the present application. The foregoing storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM, Read-only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk or an optical disk, and various media that can store program codes.

[0106] Further, the present application discloses a computer program product, the computer program product comprising a computer program stored on a non-transitory computer-readable storage medium, the computer program comprising program instructions that, when executed by a computer, cause the computer to perform the image synthesis method provided by any of the above method embodiments, for example comprising:

[0107] Based on a training mode of adversarial reinforcement learning between the first policy controller and the second policy controller, a target augmentation policy for image synthesis on the original image is obtained, the first policy controller is used for inputting the original image and outputting a first augmentation policy, and the second policy controller is used for inputting the original image and outputting a second augmentation policy.

[0108] Based on the target augmentation policy, the original image is subjected to image synthesis to obtain a target image.

[0109] On the other hand, the present application also provides a non-transitory computer-readable storage medium having a computer program stored thereon, the computer program being executed by a processor to implement the image synthesis method provided by any of the above embodiments, for example comprising:

[0110] Based on a training mode of adversarial reinforcement learning between the first policy controller and the second policy controller, a target augmentation policy for image synthesis of the original image is obtained, the first policy controller is used for inputting the original image and outputting a first augmentation policy, and the second policy controller is used for inputting the original image and outputting a second augmentation policy.

[0111] Based on the target augmentation policy, the original image is subjected to image synthesis to obtain a target image.

[0112] The system embodiments described above are only illustrative, wherein the units illustrated as separate components can or can not be physically separated, and the components illustrated as units can or can not be physical units, i.e., can be located in one place or distributed on multiple network units. Part or all of the modules can be selected to achieve the purpose of the embodiment scheme according to actual needs. Those skilled in the art can understand and implement without creative labor.

[0113] Through the description of the above embodiments, those skilled in the art can clearly understand that each embodiment can be realized by means of software and necessary general hardware platform, and of course can also be realized by hardware. Based on such understanding, the above technical solutions can be embodied in the form of software product, which can be stored in a computer readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes a plurality of instructions to make a computer power supply screen (which can be a personal computer, server, or network power supply screen, etc.) execute the method described in each embodiment or some parts of the embodiment.

[0114] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present application, and not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that: it can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement for some technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. An image synthesis method, characterized in that, include: Based on the training mode of adversarial reinforcement learning between the first policy controller and the second policy controller, a target augmentation policy for image synthesis of the original image is obtained. The first policy controller is used to input the original image and output the first augmentation policy, and the second policy controller is used to input the original image and output the second augmentation policy. Based on the target augmentation strategy, the original image is synthesized to obtain the target image; The training mode based on adversarial reinforcement learning between the first and second policy controllers obtains a target augmentation strategy for image synthesis of the original image, including: The first augmentation strategy output by the first strategy controller is updated according to the first evaluation index of the first strategy controller until the first evaluation index is greater than a preset value. The first augmentation strategy is obtained by inputting the original image into the first strategy controller. The first evaluation index is determined based on the detection results of the first detection model on the first training sample. The first training sample is obtained by image synthesis of the original image based on the first augmentation strategy. The first detection model is obtained by training the first preset detection model based on the second training sample. The second training sample is obtained by image synthesis of the original image based on the second augmentation strategy output by the second strategy controller. The second augmentation strategy output by the second strategy controller is updated according to the second evaluation index of the second strategy controller until the second evaluation index is greater than the preset value. The second evaluation index is determined based on the detection results of the second training sample by the second detection model. The second detection model is obtained by training the second preset detection model based on the first training sample. The target augmentation strategy is obtained based on the updated first augmentation strategy and the updated second augmentation strategy.

2. The image synthesis method according to claim 1, characterized in that, The methods for obtaining the first evaluation indicator and the second evaluation indicator include: The first evaluation index is obtained based on the first accuracy of the detection results of the second detection model on the second training sample, the second accuracy of the detection results of the first detection model on the first training sample, the mutual information between the first training sample and the second training sample, and the intra-class divergence of the first training sample. The second evaluation index is obtained based on the first accuracy of the detection results of the first detection model on the first training sample, the second accuracy of the detection results of the second detection model on the second training sample, the mutual information between the second training sample and the first training sample, and the intra-class divergence of the second training sample.

3. The image synthesis method according to claim 1 or 2, characterized in that, The step of performing image synthesis on the original image based on the target augmentation strategy to obtain the target image includes: Based on the updated first augmentation strategy and the updated second augmentation strategy, the original image is synthesized to obtain the target image.

4. The image synthesis method according to claim 3, characterized in that, The step of performing image synthesis on the original image based on the updated first augmentation strategy and the updated second augmentation strategy to obtain the target image includes: Based on the updated first augmentation strategy and the updated second augmentation strategy, the original image is synthesized to obtain the original synthesized image; Obtain the augmentation probability corresponding to each region in the original image, and the augmentation probability is used to characterize the probability that the corresponding region is retained in the original synthesized image; The target image is obtained based on the augmentation probability and the original synthesized image.

5. The image synthesis method according to claim 4, characterized in that, The step of obtaining the augmentation probability corresponding to each region in the original image includes: The original image is divided into a preset number of regions, and the image features of each region are obtained; The augmentation probability of each region is determined based on the correlation between the image features of each region.

6. An image synthesis system, characterized in that, include: The acquisition module is used to acquire a target augmentation strategy for image synthesis of the original image based on the training mode of adversarial reinforcement learning between the first policy controller and the second policy controller. The first policy controller is used to input the original image and output a first augmentation strategy, and the second policy controller is used to input the original image and output a second augmentation strategy. The synthesis module is used to perform image synthesis on the original image based on the target augmentation strategy to obtain the target image; The training mode based on adversarial reinforcement learning between the first and second policy controllers obtains a target augmentation strategy for image synthesis of the original image, including: The first augmentation strategy output by the first strategy controller is updated according to the first evaluation index of the first strategy controller until the first evaluation index is greater than a preset value. The first augmentation strategy is obtained by inputting the original image into the first strategy controller. The first evaluation index is determined based on the detection results of the first detection model on the first training sample. The first training sample is obtained by image synthesis of the original image based on the first augmentation strategy. The first detection model is obtained by training the first preset detection model based on the second training sample. The second training sample is obtained by image synthesis of the original image based on the second augmentation strategy output by the second strategy controller. The second augmentation strategy output by the second strategy controller is updated according to the second evaluation index of the second strategy controller until the second evaluation index is greater than the preset value. The second evaluation index is determined based on the detection results of the second training sample by the second detection model. The second detection model is obtained by training the second preset detection model based on the first training sample. The target augmentation strategy is obtained based on the updated first augmentation strategy and the updated second augmentation strategy.

7. An electronic device comprising a processor and a memory storing a computer program, characterized in that, When the processor executes the computer program, it implements the image synthesis method according to any one of claims 1 to 5.

8. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the image synthesis method as described in any one of claims 1 to 5.

9. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the image synthesis method as described in any one of claims 1 to 5.

Images