Risk control image augmentation method and device, equipment and storage medium

Abstract

The invention relates to the field of image processing, and discloses a risk control image augmentation method and device, equipment and a storage medium. The risk control image augmentation method comprises the following steps: performing target segmentation on a risk control target screenshot to obtain a risk control target segmentation image; performing semantic augmentation on the risk control target segmentation map to obtain a risk control target augmentation map; and fusing the risk control target segmentation image and the risk control target augmented image into the risk control scene image to obtain a risk control augmented image. According to the method, the risk control images with few samples can be augmented, various and rich risk control augmented images suitable for different risk control scenes are obtained, the manual annotation cost is reduced, the data acquisition efficiency is improved, the generalization ability of a downstream risk control model is improved, the over-fitting risk is reduced, and the accuracy and robustness of the model are improved.

Description

Technical Field

[0001] The present invention relates to the field of image processing, and in particular to a wind control image augmentation method, device, equipment and storage medium. Background Art

[0002] With the rapid development of Internet technology, the demand for online live broadcasts and video publishing is increasing, generating a large amount of image data, which poses a great challenge to the supervision of image content. In order to ensure the health and safety of the network environment, it is particularly important to monitor, identify and manage uploaded and disseminated video images, and image risk control technology has emerged to detect and identify potential illegal, harmful or inappropriate image content.

[0003] In the field of image risk control, few-sample detection is a very challenging problem. Traditional target detection methods usually require a large amount of labeled data for training in order to build a high-precision risk control model. However, in actual situations, especially in image risk control scenarios, due to the rapid changes in risk control target categories, the scarcity of samples within a category, and the similar interference between illegal and normal categories in the fight against black industries, the number of risk control images is often limited, and images in risk control scenarios are not suitable for simulation and reproduction, resulting in high-quality training samples being difficult to obtain and high labeling costs.

[0004] For the few-sample image detection task, it is currently mainly implemented through metric learning, meta-learning and transfer learning. These methods attempt to improve the generalization ability and detection accuracy of the model through embedding space learning, multi-task training and pre-trained model fine-tuning from the model learning level. However, these methods often face the limitations of overfitting, poor generalization and limited accuracy improvement in the field of image risk control scenarios, and cannot effectively meet actual business needs. In addition, existing methods also model and learn the data distribution of target images through image generation models, such as generative adversarial networks, variational autoencoders and diffusion models, which can generate more data samples to make up for the lack of training data. However, in the few-sample scenario, training a generative model that can learn the distribution of image data is itself a challenge. Especially in the image risk control scenario, due to the complexity and variability of risk control targets, the existing generative model data generation methods often find it difficult to generate high-quality target sample data and cannot meet business needs. Summary of the invention

[0005] The main purpose of the present invention is to provide a wind control image augmentation method, device, equipment and storage medium, aiming to solve the technical problem that wind control images are difficult to obtain.

[0006] A first aspect of the present invention provides a wind control image augmentation method, the wind control image augmentation method comprising:

[0007] Perform target segmentation on the risk control target screenshot to obtain a risk control target segmentation map;

[0008] Performing semantic augmentation on the risk control target segmentation map to obtain a risk control target augmented map;

[0009] The risk control target segmentation map and the risk control target augmented map are fused into the risk control scene image to obtain a risk control augmented image.

[0010] Optionally, in a first implementation of the first aspect of the present invention, before segmenting the risk control target screenshot to obtain the risk control target segmentation map, the method further includes:

[0011] Obtain risk control sample images;

[0012] Determining, based on the received target selection instruction, an external bounding box of the risk control target in the risk control sample image;

[0013] Based on the external frame, the risk control sample image is captured to obtain the risk control target screenshot.

[0014] Optionally, in a second implementation of the first aspect of the present invention, the step of performing target segmentation on the risk control target screenshot to obtain a risk control target segmentation map includes:

[0015] Performing target segmentation on the risk control target screenshot based on multiple preset segmentation models to obtain multiple pending segmentation graphs;

[0016] Calculate the average of the segmentation values ​​of the corresponding pixel points in the plurality of pending segmentation images corresponding to each pixel point in the risk control target screenshot;

[0017] The pixel points in the risk control target screenshot corresponding to the segmentation value mean that meet the preset target value condition are taken as risk control target points to obtain the risk control target segmentation map composed of the risk control target points.

[0018] Optionally, in a third implementation of the first aspect of the present invention, the risk control target augmented graph includes an intra-class new sample graph and an inter-class difficult class sample graph;

[0019] The step of performing semantic augmentation on the risk control target segmentation map to obtain a risk control target augmented map includes:

[0020] Based on the pre-trained first augmentation model, the risk control target segmentation map is semantically augmented within the class to obtain a new sample map within the class of the same risk control target category;

[0021] Based on the pre-trained second augmentation model, inter-class semantic augmentation is performed on the risk control target segmentation map to obtain the inter-class difficult class sample maps of different risk control target categories;

[0022] The first augmented model is obtained based on a few-sample fine-tuning of a preset large-scale cultural graph model, and the second augmented model is obtained based on a coupled preset feature extraction model and a preset diffusion model.

[0023] Optionally, in a fourth implementation of the first aspect of the present invention, the first augmented model is obtained based on fine-tuning a preset large-scale cultural graph model with a small number of samples, including:

[0024] Calculating the loss value between the predicted value and the true value of the training sample image by the preset large-scale cultural graph model based on the preset loss function, wherein the training sample image includes a fine-tuning sample image and an original generated image, wherein the original generated image is an image generated by the preset large-scale cultural graph model using the risk control target category as a prompt before fine-tuning;

[0025] Updating the model parameters of the preset large-scale cultural graph model based on a preset back-propagation algorithm to iteratively reduce the loss value to obtain the first augmented model;

[0026] Wherein, the prediction loss function is:

[0027]

[0028] In the formula, represents the prediction loss function, t is the diffusion time step, ∈ is the initial noise of the sample, c is the conditional vector generated by the text encoder and text prompt in the preset large-scale text graph model, α t , σ t and w t To control the parameters of noise strategy and sampling quality, is the image generated by the preset large-scale cultural graph model, x (ori,new) is the training sample image, new represents the fine-tuning sample image, and .ri represents the original generated image.

[0029] Optionally, in a fifth implementation of the first aspect of the present invention, the pre-trained second augmented model performs inter-class semantic augmentation on the risk control target segmentation map to obtain the inter-class difficult-class sample maps of different risk control target categories, including:

[0030] Based on the preset feature extraction model, feature extraction is performed on the risk control target segmentation map to obtain sample feature data;

[0031] Inputting the sample characteristic data into the preset diffusion model to obtain a diffusion sample image;

[0032] The diffusion sample images are screened based on a preset screening and filtering mechanism, and unsupervised clustering is performed on the retained diffusion sample images to obtain a new risk control target category to update the risk control target category;

[0033] The updated sample feature data corresponding to the risk control target category is input into the preset diffusion model to obtain the inter-class difficult class sample map.

[0034] Optionally, in a sixth implementation manner of the first aspect of the present invention, fusing the risk control target segmentation map and the risk control target augmented map into a risk control scene image to obtain a risk control augmented image includes:

[0035] Paste the risk control target segmentation map and the risk control target augmented map within the image range of the risk control scene image to obtain an image to be processed;

[0036] The image to be processed is subjected to transition smoothing processing based on a preset fusion algorithm to obtain the wind control augmented image.

[0037] A second aspect of the present invention further provides a wind control image augmentation device, the wind control image augmentation device comprising:

[0038] The risk control target extraction module is used to segment the risk control target screenshot to obtain a risk control target segmentation map;

[0039] A risk control target generation module, used to perform semantic augmentation on the risk control target segmentation map to obtain a risk control target augmented map;

[0040] The risk control target fusion module is used to fuse the risk control target segmentation map and the risk control target augmented map into the risk control scene image to obtain a risk control augmented image.

[0041] The third aspect of the present invention also provides a computer device, comprising: a memory and at least one processor, wherein the memory stores instructions; the at least one processor calls the instructions in the memory so that the computer device executes the wind control image augmentation method as described above.

[0042] A fourth aspect of the present invention further provides a computer-readable storage medium, on which instructions are stored, and when the instructions are executed by a processor, the wind control image augmentation method as described above is implemented.

[0043] The embodiments of the present invention provide a method, device, equipment and storage medium for augmenting risk control images. The method first performs target segmentation on a screenshot of a risk control target to obtain a risk control target segmentation map, then performs semantic augmentation on the risk control target segmentation map to obtain a risk control target augmented map, and then fuses the risk control target segmentation map and the risk control target augmented map into the risk control scene image to obtain a risk control augmented image. In this way, a small number of risk control images can be augmented to obtain high-quality and diverse risk control augmented images suitable for different risk control scenarios, thereby reducing the need to collect new data, better adapting to risk control scenarios with small numbers of samples and difficulty in acquiring sample images, and reducing data acquisition costs. It can effectively improve downstream training efficiency, enrich sample image data sets used for downstream model training, help downstream models learn more image features of risk control scenarios, reduce overfitting of training data, and improve the generalization ability, accuracy and robustness of the model to meet a wider range of risk control business needs. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 This is a schematic diagram of a first embodiment of the flow chart of the wind control image augmentation method in an embodiment of the present invention;

[0045] Figure 2 for Figure 1 A schematic diagram of an embodiment flow chart before step 101 in the embodiment;

[0046] Figure 3 for Figure 1 A schematic flow chart of an embodiment of step 101 in the embodiment;

[0047] Figure 4 for Figure 1 A schematic flow chart of an embodiment of step 102 in the embodiment;

[0048] Figure 5 for Figure 1 A schematic flow chart of an embodiment of step 103 in the embodiment;

[0049] Figure 6 This is a schematic diagram of functional modules of an embodiment of a wind control image augmentation device according to an embodiment of the present invention;

[0050] Figure 7 The figure is a functional module diagram of an embodiment of a computer device in an embodiment of the present invention. DETAILED DESCRIPTION

[0051] The terms "first", "second", "third", "fourth", etc. (if any) in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments described herein can be implemented in an order other than that illustrated or described herein. In addition, the terms "including" or "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0052] For ease of understanding, the specific process of the wind control image augmentation method in the embodiment of the present invention is described below. The wind control image augmentation method of this embodiment is applied to electronic devices, which can be terminals, such as smart phones, laptops, tablet computers, desktop computers, etc., or development boards, chip systems, etc. Please refer to Figure 1 , Figure 1 1 is a flow chart of a first embodiment of a method for augmenting a wind control image in an embodiment of the present invention. In this embodiment, the method for augmenting a wind control image includes:

[0053] 101. Perform target segmentation on the risk control target screenshot to obtain a risk control target segmentation map;

[0054] In this embodiment, the risk control target screenshot refers to an image including the risk control target captured from an image containing a risk control scenario. For example, the risk control target may be a prohibited item, a violator, and an illegal behavior in the image of the risk control scenario, and the corresponding risk control target screenshot is obtained by capturing the risk control target as the main body. The acquisition of the risk control target screenshot can be performed by manually operating the screenshot tool to capture and annotate, or automatically capturing using image recognition technology. The risk control target segmentation map is obtained by separating the risk control target from the risk control target screenshot using image segmentation technology. The image segmentation technology can be threshold-based binary segmentation, edge detection, region growing, graph cutting algorithm, or specially trained image segmentation model, etc. The obtained risk control target segmentation map only includes the risk control target, so as to facilitate the subsequent generation and fusion processing based on the risk control target.

[0055] Optional, see Figure 2 In one embodiment, before segmenting the risk control target screenshot to obtain the risk control target segmentation map, the method further includes:

[0056] 1001. Obtain risk control sample images;

[0057] 1002. Determine an external frame of a risk control target in a risk control sample image based on the received target selection instruction;

[0058] 1003. Based on the external frame, the risk control sample image is intercepted to obtain a risk control target screenshot.

[0059] In this optional embodiment, the risk control sample image is the sample data originally used to train the risk control model. The image includes the risk control target and the corresponding risk control scene, which can be obtained from the relevant data risk control database or extracted from the graphic information. The selection of the risk control sample image can be random, or it can be selected in a targeted manner according to the needs of a specific risk control scene. For example, if it is necessary to enhance the model's ability to recognize a specific type of prohibited items, sample images containing the prohibited items can be specifically selected. However, due to the characteristics of image detection in the risk control scenario, the categories change quickly and in large numbers, the number of samples within the category is small, and the violations caused by the confrontation with the black industry are similar to the interference of normal categories, etc., the difficulty of obtaining relevant scene images is extremely large, and therefore the number of risk control sample images is usually scarce, and there are only a small number of such risk control sample images, which cannot meet the requirements of model training for the number and diversity of samples.

[0060] In this optional embodiment, the target selection instruction is issued by a manual or automated image recognition system to indicate the target area to be captured in the risk control sample image. The instruction can be based on a preset rule, for example, the shape and marking method of the bounding box are determined according to the shape, size or color of the risk control target. Once the bounding box is determined, the risk control sample image can be cropped to obtain a screenshot of the risk control target, thereby ensuring that the risk control target is accurately extracted from the sample image, providing a basis for subsequent image augmentation.

[0061] In this optional embodiment, it is preferred to determine the external frame and capture the risk control target screenshot based on manual annotation. In this process, the manual annotation method can provide higher accuracy and flexibility. The manual annotator can flexibly adjust the position and size of the external frame according to the characteristics and contextual information of the risk control target to ensure that the captured image area reflects the risk control target as accurately as possible. At the same time, manual annotation can also reduce the erroneous capture caused by the limitations of image recognition technology to a certain extent, thereby improving the quality of the risk control target screenshot. In addition, since the risk control sample images are relatively small, the existing manually operable screenshot and annotation tools are relatively advanced, and the actual manual annotation method has a small workload, low cost and high efficacy.

[0062] Optional, see Figure 3 In one embodiment, target segmentation is performed on the risk control target screenshot to obtain a risk control target segmentation map, including:

[0063] 1011. Performing target segmentation on the risk control target screenshot based on multiple preset segmentation models to obtain multiple pending segmentation graphs;

[0064] 1012. Calculate the average segmentation values ​​of corresponding pixel points in multiple pending segmentation images corresponding to each pixel point in the risk control target screenshot;

[0065] 1013. Pixel points whose corresponding segmentation value mean values ​​in the risk control target screenshot meet the preset target value conditions are taken as risk control target points to obtain a risk control target segmentation map composed of risk control target points.

[0066] In this optional embodiment, the preset segmentation model can be a pre-trained image segmentation model, such as the SAM model (Segment Anything Model), the Mask R-CNN (Mask Region-based Convolutional Neural Network) model, the U-Net model, etc. These models can identify different regions in the image and separate the risk control target from the background. Different models may show their own advantages when processing different types of images or targets. For example, the SAM model and its derived segmentation models such as the MobileSAM model and the FastSAM model are trained based on billions of data and have a universal and accurate segmentation effect for any object. Training based on deep learning models such as Mask R-CNN, U-Net, DeepLab, or fine-tuning and expanding the SAM model and its derived models such as computer vision large models and CLIP (Contrastive Language-Image Pre-training) can obtain a segmentation model with better segmentation effect for specific segmentation services. By averaging the separation values ​​of the pending models obtained by multiple preset segmentation models, the accuracy and robustness of the segmentation can be further improved.

[0067] In this optional embodiment, the mean value of each group of corresponding pixels is calculated, and the mean value can be a numerical average or an average after weighting or biasing. For example, if n segmentation models are used, the segmentation result of each model is S i (x, y), where i = 1, 2, · · ·, n, S i (x, y) refers to the classification result of model i on the pixel point (x, y) of the intercepted rectangular box. 1 represents the target and 0 represents the background. The calculation method is: In this formula, if S final(x,y)≥0.5, the corresponding pixel is determined to be the target, otherwise it is the background. In addition, for multiple binary images, the value of each pixel can also be 0 or 255. The value of each pixel can be weighted according to its frequency of occurrence in different segmentation models, or different weights can be assigned to different segmentation models to ensure that in the final segmented image, those areas that are consistently identified in most models have higher weights. This weighted average method can reduce the error caused by the deviation of a single model, thereby improving the overall segmentation quality. In addition, confidence evaluation can be introduced, that is, a confidence score is assigned to the segmentation result of each pixel, which reflects the certainty of the model's classification of the pixel. In this way, the segmentation results can be further optimized to ensure that in the final output image, the high-confidence areas are more accurate, while the low-confidence areas can be manually reviewed or further processed.

[0068] In this optional embodiment, the preset target condition is a threshold for determining which pixels belong to the risk control target. The threshold can be fixed or dynamically adjusted according to the characteristics of the risk control target. For example, if the risk control target has a high contrast or a specific color, a lower threshold can be set to ensure that the target area is accurately segmented. On the contrary, if the contrast between the wind control target and the background is low, a higher threshold may need to be set to avoid misidentifying the background as a risk control target. In this way, it can be ensured that the risk control target segmentation map only contains the target area and does not contain redundant background information, thereby improving the accuracy and efficiency of subsequent image augmentation, and obtaining a risk control target segmentation map composed of pixels that meet the conditions, which clearly shows the outline and position of the risk control target.

[0069] 102. Perform semantic augmentation on the risk control target segmentation map to obtain a risk control target augmented map;

[0070] In this embodiment, semantic augmentation refers to a technology that enhances the target features in the risk control target segmentation map through a series of image processing techniques to generate a new risk control target augmentation map. These technologies include but are not limited to image rotation, scaling, color transformation, texture synthesis, etc., the purpose of which is to increase the diversity of the target image while keeping the semantic information of the target unchanged. For example, the target in the risk control target segmentation map can be slightly rotated or scaled, or its color and brightness can be changed to simulate the image changes under different viewing angles and lighting conditions. In addition, deep learning models such as Generative Adversarial Networks (GAN) can also be used to generate new images with similar semantics but different appearances to the original target. Through these methods, the image data set of the risk control target can be effectively expanded, and the recognition ability of the risk control model for the target can be improved. The obtained risk control target augmentation map is a risk control target with a higher data volume and diversity. Subsequently, by integrating the risk control target into the real business scene image of the risk control, a more diverse and rich risk control image is provided for the downstream risk control model to be trained or fine-tuned.

[0071] Optional, see Figure 4 In one embodiment, the risk control target augmented graph includes a new sample graph within a class and a difficult sample graph between classes. The risk control target segmentation graph is semantically augmented to obtain a risk control target augmented graph, including:

[0072] 1021. Based on the pre-trained first augmentation model, perform intra-class semantic augmentation on the risk control target segmentation map to obtain a new intra-class sample map of the same risk control target category;

[0073] 1022. Based on the pre-trained second augmented model, inter-class semantic augmentation is performed on the risk control target segmentation map to obtain inter-class difficult-class sample maps of different risk control target categories.

[0074] The first augmented model is obtained based on a few-sample fine-tuning of a preset large-scale text graph model, and the second augmented model is obtained based on a coupled preset feature extraction model and a preset diffusion model.

[0075] In this optional embodiment, intra-class semantic augmentation (Intra-class semantic augmentation, IntraCSA) is intended to be used for sample generation augmentation within a certain risk control target category. Because in a scenario with few samples, there is often a lack of image information such as the presentation of the risk control target at different angles, background conditions, and near and far observation distances. For example, when the risk control target is an illegal flag of an illegal organization, the samples obtained are often limited, which makes it extremely difficult to train a risk control model that can detect the illegal flag in a new scene or new angle; and when there are a large number of samples containing the flag in different scenes and angles, the training difficulty of the risk control model is much smaller. Therefore, in the generation stage, in order to controllably generate the specified target, the first augmentation model can be used to perform intra-class semantic augmentation on the risk control target segmentation map to obtain a new intra-class sample map of the same risk control target category, so as to enhance the diversity and richness of the risk control targets in the same category.

[0076] In this optional embodiment, the preset large-scale cultural graph model can generate new images with similar semantics but different appearances by learning a large number of risk control target images in different scenarios. For example, for risk control targets in the flag category, new sample images of flags in different lighting, angles and backgrounds can be generated, thereby increasing the sample diversity during model training and improving the model's recognition ability for specific targets. The preset large-scale cultural graph model can be a Stable Diffusion series model, a DALL-E series model and an Imagen model based on a diffusion model, a Midjourney model based on a Transformer deep learning model, a pre-trained CLIP model, etc. These models have learned a rich correspondence between images and texts through pre-training, can understand complex image content, and generate images that match text descriptions.

[0077] In this optional embodiment, the preset large-scale literary graph model can generate images based on given text prompts. Although it shows strong generation capabilities, its limitations are also very obvious: first, because the model is trained on public data sets collected on the Internet, it can only generate target objects of common, coarse-grained categories on the Internet, and cannot be directly used for risk control target generation; second, in terms of refined controllable generation, it is difficult for the model to accurately generate specified target samples that can match text descriptions. To address the above limitations, this embodiment injects the knowledge of risk control targets into the large-scale literary graph model, and obtains the first augmented model by fine-tuning the generated model with a few samples. Since the risk control target images of a few samples are related to the actual business, the model can better understand the characteristics of specific risk control targets through fine-tuning, thereby generating more accurate new sample images within the class.

[0078] Optionally, in one embodiment, the first augmented model is obtained based on fine-tuning a preset large-scale cultural graph model with a few samples and includes:

[0079] (1) Calculate the loss value between the predicted value and the true value of the training sample image by the preset large-scale cultural graph model based on the preset loss function, where the training sample image includes a fine-tuning sample image and an original generated image, where the original generated image is an image generated by the preset large-scale cultural graph model using the risk control target category as a prompt input before fine-tuning;

[0080] (2) Based on a preset back propagation algorithm, the model parameters of the preset large-scale cultural graph model are updated to iteratively reduce the loss value to obtain a first augmented model.

[0081] Among them, the prediction loss function is: In the formula, represents the prediction loss function, t is the diffusion time step, ∈ is the initial noise of the sample, c is the conditional vector generated by the text encoder and text prompt in the preset large-scale text graph model, α t , σ t and w t To control the parameters of noise strategy and sampling quality, The image generated by the preset large-scale cultural graph model, x (ori,new) is the training sample image, new represents the fine-tuning sample image, and ori represents the original generated image.

[0082] In this optional embodiment, due to the risk control targets belonging to some general and common object categories, the generation model may suffer from the phenomenon of "knowledge forgetting", that is, after fine-tuning with the risk control target sample, the preset large-scale cultural graph model is unable to generate other images under the relevant categories before. To avoid this situation, this optional embodiment adds fine-tuning sample images and original generated images to jointly fine-tune the preset large-scale cultural graph model, wherein the fine-tuning sample images are sample images of a small number of risk control targets originally used for fine-tuning the preset large-scale cultural graph model to represent the new knowledge that the model needs to learn, and the fine-tuning sample images can specifically be newly introduced images including risk control targets, or the aforementioned risk control target images or risk control target screenshots can be used to save data needs, while the original generated images are images generated by the preset large-scale cultural graph model using the risk control target category as a prompt input before fine-tuning, representing the original old knowledge of the model. The number of the two sample images often only needs to be within 100. By jointly fine-tuning the preset text-based image model with the two images, the model's ability to generate general category images can be maintained, while enhancing its ability to recognize and generate specific risk control targets.

[0083] In this optional embodiment, during the fine-tuning process, the preset loss function is used to measure the difference between the model prediction value and the true value, and the preset back propagation algorithm is used to adjust the model parameters according to the loss value to reduce the prediction error. In this way, the first augmented model can generate a new and diverse sample graph within the class that is related to the risk control target category, thereby providing richer and more diverse training data for the risk control model. In the formula of the corresponding loss function, the square error is calculated, t,∈,c,α t , σ t and w t Parameters such as are commonly used in the preset large-scale cultural graph model, and will not be described here. By adjusting these parameters, the style and quality of the generated image can be controlled to meet different risk control requirements. The first augmented model obtained in the above manner can learn the characteristics of the risk control target and generate new images with similar semantics but different appearance to the original risk control target, thereby providing richer and more diverse training data for the risk control model.

[0084] In this optional embodiment, on the other hand, inter-class semantic augmentation (InterCSA) is intended to be used for data generation augmentation between different risk control target categories, generating difficult class sample images between different risk control target categories. Due to the lack of training samples in the few-sample risk control scenario, the risk control model is prone to problems such as overfitting and misidentification. Targets of different categories may have similar appearance features, which makes it difficult for the model to distinguish. For example, some prohibited items may be similar to legal items in shape and color, which makes the model prone to misjudgment. For example, a close-up focus lens image of two fingers together is easily misjudged by the model as exposed legs. The reason is that the risk control model does not clearly distinguish the boundary line between the target categories. For these categories that are easily misidentified, they are also called "difficult categories". Therefore, in the generation stage, in order to improve the generalization ability of the risk control model, the second augmentation model can be used to perform inter-class semantic augmentation on the risk control target segmentation map to obtain inter-class difficult class sample maps of different risk control target categories, and a certain number of difficult class samples can be expanded to supplement the training sample set.

[0085] In this optional embodiment, as mentioned above, many excellent image generation models such as diffusion models can generate images of good quality, but most of these generation models receive text prompts as input, and the generated results are not controllable, which means that it is not feasible to directly use these generation models to generate difficult samples of risk control target classes. The second augmented model of this optional embodiment is obtained based on the coupled preset feature extraction model and the preset diffusion model, which is improved and optimized in two aspects: first, the image of the risk control target class is used as one of the conditional constraints, so that the preset diffusion model generates difficult samples that are similar in structure, texture or color, that is, the preset feature extraction model is used to extract the feature vector of the risk control target image, and then used as the conditional input for the preset diffusion model to decode and generate new samples; second, a screening and filtering mechanism for the generated samples is designed. This process involves mixing different categories of risk control target images to simulate the confusion that may occur in actual scenes, so that the training model can accurately identify and distinguish these subtle differences when facing different categories of targets with similar appearance features, thereby improving the accuracy and robustness of the model in practical applications.

[0086] Optionally, in one embodiment, inter-class semantic augmentation is performed on the risk control target segmentation map based on the pre-trained second augmentation model to obtain inter-class difficult-class sample maps of different risk control target categories, including:

[0087] (1) Extract features from the risk control target segmentation map based on a preset feature extraction model to obtain sample feature data;

[0088] (2) Inputting the sample feature data into a preset diffusion model to obtain a diffusion sample image;

[0089] (3) Screen the diffusion sample images based on the preset screening and filtering mechanism, and perform unsupervised clustering on the retained diffusion sample images to obtain new risk control target categories to update the risk control target categories;

[0090] (4) Input the sample feature data corresponding to the updated risk control target category into the preset diffusion model to obtain the inter-class difficult class sample map.

[0091] In this optional embodiment, based on the risk control target segmentation map, the second augmented model first uses a preset feature extraction model to conduct an in-depth analysis of the image to extract key feature data. These feature data capture key information such as the structure, texture, and color of the risk control target, providing a basis for subsequent image generation. Then, these feature data are input into the preset diffusion model, and a series of diffusion sample images with potential diversity are generated through the diffusion process of the model. Subsequently, a preset screening and filtering mechanism screens these diffusion sample images to ensure that only images that meet specific criteria are retained. The screening process may include multiple evaluations of image clarity, target integrity, and similarity to the risk control target. Through screening, images that are significantly different from the risk control target or of low quality can be removed to ensure the quality of the images used in subsequent steps.

[0092] In this optional embodiment, unsupervised cluster analysis will be performed on the retained diffusion sample images to identify and distinguish different risk control target categories. The clustering process helps to discover natural groupings in the data, thereby updating and refining the categories of risk control targets. In this way, new risk control target categories can be discovered, which may not be clearly identified in the original data set, but are of great significance in practical applications. Finally, the sample feature data corresponding to the updated risk control target category is input into the preset diffusion model again to generate the final inter-class difficult sample map. These images represent challenging samples between different categories, which can effectively expand the training data set of the risk control model and improve the generalization ability and robustness of the model when facing actual business scenarios. Through such inter-class semantic augmentation, the risk control model can better learn and distinguish targets that look similar but belong to different categories, thereby reducing misjudgments and improving recognition accuracy in practical applications.

[0093] In this optional embodiment, a specific embodiment can refer to the following example:

[0094] (1) The sample set G of the risk control target segmentation map is G = {g1, g2, …, g m} Perform feature extraction to obtain a set of feature vectors in Represents sample g i The feature vector of the generated sample is constructed For subsequent similarity calculation and filtering; similarly, for the original risk control target category sample data set T = {t1, t2, ..., t n}Extract features and obtain a set of feature vectors

[0095] (2) Use set F T Each eigenvector in As a query, search the feature library Similar samples in . The similarity function is defined as The similarity function can be cosine similarity or Euclidean distance. For each target sample Find the sample set in the feature library that meets the similarity threshold condition: Here, τ is the similarity threshold, which can be flexibly adjusted according to the actual situation;

[0096] (3) By merging all similar sample sets, we can obtain the generated sample set that needs to be eliminated. And remove similar samples from the feature library to obtain the feature library after removal:

[0097] (4) Generate the remaining sample feature set Perform unsupervised clustering, such as using the K-means clustering algorithm, to obtain N clusters C1, C2, …, C N , where C k Represents the kth cluster:

[0098]

[0099] (5) It is determined that the original risk control target class has S categories. After clustering, the generated samples have added N new categories. These (N+S) categories are used as training sample sets to train the downstream risk control model. More specifically, the corresponding training category set is: C = {C1, C2, …, C N}∪{T1,T2,…,T S}, where T j Represents the jth category in the original risk control target category.

[0100] This optional embodiment implements the inter-class semantic augmentation mechanism through the second augmentation model, which can effectively improve the performance of the risk control model in distinguishing targets with similar appearance but different categories. By introducing inter-class semantic augmentation, the model can learn richer feature representations, thereby reducing misjudgments and improving recognition accuracy in practical applications. In addition, through unsupervised cluster analysis, new risk control target categories can be discovered and distinguished. These categories may not be clearly identified in the original data set, but are of great significance in practical applications. Ultimately, these inter-class difficult-class sample graphs can effectively expand the training data set of the risk control model and improve the generalization ability and robustness of the model when facing actual business scenarios. Through such inter-class semantic augmentation, the risk control model can better learn and distinguish targets that are similar in appearance but belong to different categories, thereby reducing misjudgments and improving recognition accuracy in practical applications.

[0101] 103. The risk control target segmentation map and the risk control target augmented map are integrated into the risk control scene image to obtain a risk control augmented image.

[0102] In this embodiment, the fusion process of the wind control augmented image involves superimposing the wind control target segmentation map and the wind control target augmented image into the actual wind control scene image. This process can use a variety of image processing techniques, such as image superposition, fusion algorithm, etc., to ensure that the augmented image is visually consistent with the original scene image, while ensuring that the features of the wind control target are preserved and enhanced. The fused wind control augmented image not only enriches the diversity of the scene, but also improves the adaptability and accuracy of the wind control model in different environments.

[0103] In this optional embodiment, during the fusion process, the risk control target segmentation map needs to be processed first to ensure that its edge matches the target edge in the risk control target augmented map. This step may require fine-tuning using image processing software or a specific edge processing algorithm to ensure that the target is naturally integrated into the background in the fused image. Next, the target in the risk control target augmented map is placed in the appropriate position of the risk control scene image, and the augmented target is visually coordinated with the rest of the scene image by adjusting parameters such as transparency, contrast, and brightness. In addition, in order to further improve the performance of the risk control model, data enhancement technology can be used to further process the risk control augmented image. For example, through operations such as rotation, scaling, and cropping, more varied risk control scene images can be generated, thereby increasing the sample diversity during model training. These enhanced images can be used as part of the training data to help the model better generalize to new and unseen scenes, obtain rich and diverse risk control augmented images, and enable the risk control model obtained by downstream training to learn more image features of the risk control scene, reduce overfitting of the training data, and improve the generalization ability, accuracy, and robustness of the model to meet a wider range of risk control business needs.

[0104] Optional, see Figure 5 In one embodiment, the risk control target segmentation map and the risk control target augmented map are fused into the risk control scene image to obtain a risk control augmented image, including:

[0105] 1031. Paste the risk control target segmentation map and the risk control target augmented map within the image range of the risk control scene image to obtain an image to be processed;

[0106] 1032. Perform transition smoothing processing on the image to be processed based on a preset fusion algorithm to obtain a wind control augmented image.

[0107] In this optional embodiment, the image range of the wind control scene image is pre-set according to the image size to ensure that the wind control target segmentation map and the wind control target augmentation map can be properly placed and merged, so that the subsequent pasting will not exceed the background image boundary and cause unpredictable problems or errors, so the above constraints are required. For example, suppose the background image set is Each background image Bi Size H B ×W B ; Let the foreground target set be Each foreground image F i The size is H F ×W F , the bounding rectangle of the target area is R j , whose center point is (x j ,y j ). In the background image B i Randomly select a position (x i ,y i ) as the pasting position. Then the constraints of the corresponding pasting position, that is, the image range is: This constraint is used to ensure that the foreground image F i Completely on background image B i Inside.

[0108] In this embodiment, after determining the image range constraint, the foreground image F can be pasted and combined. i The pixel value F j (x,y) covers the background image B i The position (x i ,y i ), and obtain the fused wind control augmented image M ij , whose formula is For each fused image M ij , and can further record the foreground target location information, including the center point (x i ,y i ) and the circumscribed rectangle R j Length and width (W F ,H F ) is used for subsequent risk control model training. In the subsequent risk control model training, there are generally two types: one is image classification, which only requires images and corresponding category labels to form a training set; the other is image detection and positioning, which requires images and the location information and category labels of the corresponding risk control targets in the images to form a training set. Therefore, it is necessary to record the location information here in order to more accurately determine the risk control targets for corresponding classification and detection and labeling.

[0109] In this optional embodiment, the pasting method can be random pasting or pasting based on specific rules. Among them, random pasting is a simple and effective foreground and background fusion method, which can quickly generate endless samples. Although the combined image samples may not look reasonable, practice shows that such a combination method is also useful for improving the accuracy of the wind control model. Pasting based on specific rules, such as identifying and matching the depth of field, proportion and contrast of the background image of the wind control scene to paste the wind control target segmentation map and the wind control target augmented map of the foreground, and analyzing the characteristics of the wind control target to ensure the naturalness and rationality of the pasting position. Although this rule-based pasting method can obtain a more natural and actual scene-compliant wind control augmented image, its processing process is relatively complex and requires more computing resources and time. In practical applications, the appropriate pasting method can be selected according to specific needs and resource conditions. For example, in the case of limited resources, random pasting can be given priority to quickly generate a large number of samples; in situations where there are high requirements for image quality, a pasting method based on specific rules can be adopted to ensure the quality of the generated wind control augmented image and its adaptability to different wind control scenarios.

[0110] In this optional embodiment, after the pasting is completed, there may still be an abrupt transition between the foreground risk control target segmentation map and the risk control target augmented map and the background risk control scene image, which cannot simulate the actual wind control scene well. Therefore, a smooth transition processing can be performed between the foreground and background in the image to optimize the edge effect of the foreground and background of the image to be processed, and obtain a more natural and reliable wind control augmented image.

[0111] In this optional embodiment, a preset fusion algorithm is used to perform transition smoothing on the pasted image. The preset fusion algorithm can be a pixel-based linear interpolation method, or a more complex image content-based fusion technique, such as Poisson fusion algorithm, multi-scale decomposition fusion algorithm, and pyramid fusion algorithm. These algorithms can effectively reduce the visual differences between images, making the transition between the foreground image and the background image more natural, thereby generating a high-quality wind control augmented image. Specifically, the fusion algorithm analyzes the pixel values ​​of the foreground image and the background image in the pasted area, and calculates the new pixel value according to certain fusion rules. For example, a weighted average method can be used to determine the weight of the final pixel value according to the pixel intensity of the foreground image and the background image. In addition, the edge information of the image can also be considered, and the boundary of the foreground and the background can be identified by an edge detection algorithm, and then a more sophisticated fusion strategy is applied in the boundary area to ensure a smooth transition of the edge of the image. After the fusion process, the wind control augmented image will be used to train the wind control model. Through such image fusion and augmentation, the risk control model can be exposed to richer and more diverse scenarios, thus having better generalization ability and accuracy in practical applications. The obtained risk control augmented image helps to improve the overall performance of the downstream model, ensuring that risk targets can be effectively identified and processed in various complex scenarios, greatly improving the recognition accuracy and robustness of the model in practical applications.

[0112] In order to execute the corresponding steps in the above method embodiment and each possible implementation method, the following provides an implementation method of a wind control image augmentation device. Figure 6 , Figure 6 This is a functional module diagram of an embodiment of a wind control image augmentation device in an embodiment of the present invention. In this embodiment, the wind control image augmentation device is applied to an electronic device, and the wind control image augmentation device includes:

[0113] The risk control target extraction module 201 is used to segment the risk control target screenshot to obtain a risk control target segmentation map;

[0114] The risk control target generation module 202 is used to perform semantic augmentation on the risk control target segmentation map to obtain a risk control target augmented map;

[0115] The risk control target fusion module 203 is used to fuse the risk control target segmentation map and the risk control target augmented map into the risk control scene image to obtain a risk control augmented image.

[0116] Optionally, in one embodiment, the risk control target extraction module 201 is further specifically used to: obtain a risk control sample image; determine an external frame of the risk control target in the risk control sample image based on the received target selection instruction; and capture the risk control sample image based on the external frame to obtain a screenshot of the risk control target.

[0117] Optionally, in one embodiment, the risk control target extraction module 201 is specifically used to: perform target segmentation on the risk control target screenshot based on multiple preset segmentation models to obtain multiple pending segmentation maps; calculate the average segmentation value of the corresponding pixel points in the multiple pending segmentation maps corresponding to each pixel point in the risk control target screenshot; and take the pixel points in the risk control target screenshot whose corresponding segmentation value averages meet the target value conditions as risk control target points to obtain a risk control target segmentation map composed of risk control target points.

[0118] Optionally, in one embodiment, the risk control target augmented graph includes an intra-class new sample graph and an inter-class difficult sample graph, and the risk control target generation module 202 is specifically used to: perform intra-class semantic augmentation on the risk control target segmentation graph based on a pre-trained first augmentation model to obtain an intra-class new sample graph of the same risk control target category; perform inter-class semantic augmentation on the risk control target segmentation graph based on a pre-trained second augmentation model to obtain an inter-class difficult sample graph of different risk control target categories; wherein the first augmentation model is obtained based on fine-tuning a preset large-scale literary graph model with a few samples, and the second augmentation model is obtained based on a coupled preset feature extraction model and a preset diffusion model.

[0119] Optionally, in one embodiment, the risk control target generation module 202 is further used to: calculate the loss value between the predicted value and the true value of the training sample image by the preset large-scale Wensheng graph model based on a preset loss function, the training sample image includes a fine-tuning sample image and an original generated image, and the original generated image is an image generated by the preset large-scale Wensheng graph model using the risk control target category as a prompt input before fine-tuning; update the model parameters of the preset large-scale Wensheng graph model based on a preset back propagation algorithm to iteratively reduce the loss value to obtain a first augmented model; wherein the prediction loss function is: In the formula, represents the prediction loss function, t is the diffusion time step, ∈ is the initial noise of the sample, c is the conditional vector generated by the text encoder and text prompt in the preset large-scale text graph model, α t , σ t and w t To control the parameters of noise strategy and sampling quality, The image generated by the preset large-scale cultural graph model, x (ori,new) is the training sample image, new represents the fine-tuning sample image, and ori represents the original generated image.

[0120] Optionally, in one embodiment, the risk control target generation module 202 is specifically used to: extract features from the risk control target segmentation map based on a preset feature extraction model to obtain sample feature data; input the sample feature data into a preset diffusion model to obtain a diffusion sample image; screen the diffusion sample image based on a preset screening and filtering mechanism, and perform unsupervised clustering on the retained diffusion sample image to obtain a new risk control target category to update the risk control target category; input the sample feature data corresponding to the updated risk control target category into the preset diffusion model to obtain an inter-class difficult-to-class sample map.

[0121] Optionally, in one embodiment, the risk control target fusion module 203 is further specifically used to: paste the risk control target segmentation map and the wind control target augmented map within the image range of the wind control scene image to obtain the image to be processed; and perform transition smoothing on the image to be processed based on a preset fusion algorithm to obtain the wind control augmented image.

[0122] Since the embodiments of the device part correspond to the embodiments of the above-mentioned method, please refer to the above-mentioned method embodiments for the introduction of the wind control image augmentation device provided in the embodiments of the present invention. The embodiments of the present invention will not be repeated here, and it has the same beneficial effects as the above-mentioned wind control image augmentation method.

[0123] The present invention also provides a computer device, which includes a memory and a processor. The memory stores computer-readable instructions. When the computer-readable instructions are executed by the processor, the processor executes the wind control image augmentation method as described above. Figure 7 3 is a functional module diagram of a computer device provided by an embodiment of the present invention. The computer device 300 may have relatively large differences due to different configurations or performances, and may include one or more processors (central processing units, CPU) 310 (for example, one or more processors) and a memory 320, and one or more storage media 330 (for example, one or more mass storage devices) storing application programs 333 or data 332. Among them, the memory 320 and the storage medium 330 can be temporary storage or permanent storage. The program stored in the storage medium 330 may include one or more modules (not shown in the figure), and each module may include a series of instruction operations in the computer device 300. Furthermore, the processor 310 can be configured to communicate with the storage medium 330 to execute a series of instruction operations in the storage medium 330 on the computer device 300.

[0124] The computer device 300 may also include one or more power supplies 340, one or more wired or wireless network interfaces 350, one or more input and output interfaces 360, and / or one or more operating systems 331, such as Windows Server, Mac OS X, Unix, Linux, FreeBSD, etc. It will be appreciated by those skilled in the art that Figure 7 The illustrated computer device structure does not constitute a limitation on the computer device, and may include more or fewer components than illustrated, or combine certain components, or arrange the components differently.

[0125] The present invention also provides a computer-readable storage medium, which may be a non-volatile computer-readable storage medium or a volatile computer-readable storage medium. Instructions are stored in the computer-readable storage medium. When the instructions are executed on a computer, the computer executes the wind control image augmentation method as described above.

[0126] Those skilled in the art can clearly understand that, for the convenience and simplicity of description, the specific working process of the device described above can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here. If the integrated module or unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the present invention is essentially or part of the contribution to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk and other media that can store program code.

[0127] As described above, the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features thereof may be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A wind control image augmentation method, characterized in that: include: Perform target segmentation on the risk control target screenshot to obtain a risk control target segmentation map; Performing semantic augmentation on the risk control target segmentation map to obtain a risk control target augmented map; The risk control target segmentation map and the risk control target augmented map are fused into the risk control scene image to obtain a risk control augmented image.

2. The wind control image augmentation method according to claim 1, characterized in that: Before segmenting the risk control target screenshot to obtain the risk control target segmentation map, the method further includes: Obtain risk control sample images; Determining, based on the received target selection instruction, an external bounding box of the risk control target in the risk control sample image; Based on the external frame, the risk control sample image is captured to obtain the risk control target screenshot.

3. The wind control image augmentation method according to claim 1, characterized in that: The target segmentation is performed on the risk control target screenshot to obtain a risk control target segmentation map, including: Performing target segmentation on the risk control target screenshot based on multiple preset segmentation models to obtain multiple pending segmentation graphs; Calculate the average of the segmentation values ​​of the corresponding pixel points in the plurality of pending segmentation images corresponding to each pixel point in the risk control target screenshot; The pixel points in the risk control target screenshot corresponding to the segmentation value mean that meet the preset target value condition are taken as risk control target points to obtain the risk control target segmentation map composed of the risk control target points.

4. The wind control image augmentation method according to claim 1, characterized in that: The risk control target augmented graph includes a new sample graph within a class and a difficult sample graph between classes; The step of performing semantic augmentation on the risk control target segmentation map to obtain a risk control target augmented map includes: Based on the pre-trained first augmentation model, the risk control target segmentation map is semantically augmented within the class to obtain a new sample map within the class of the same risk control target category; Based on the pre-trained second augmentation model, inter-class semantic augmentation is performed on the risk control target segmentation map to obtain the inter-class difficult-class sample maps of different risk control target categories; The first augmented model is obtained based on a few-sample fine-tuning of a preset large-scale cultural graph model, and the second augmented model is obtained based on a coupled preset feature extraction model and a preset diffusion model.

5. The wind control image augmentation method according to claim 4, characterized in that: The first augmented model is obtained by fine-tuning a preset large-scale cultural graph model with a few samples, and includes: Calculating the loss value between the predicted value and the true value of the training sample image by the preset large-scale cultural graph model based on the preset loss function, wherein the training sample image includes a fine-tuning sample image and an original generated image, wherein the original generated image is an image generated by the preset large-scale cultural graph model using the risk control target category as a prompt before fine-tuning; Updating the model parameters of the preset large-scale cultural graph model based on a preset back-propagation algorithm to iteratively reduce the loss value to obtain the first augmented model; Wherein, the prediction loss function is: In the formula, represents the prediction loss function, t is the diffusion time step, ∈ is the initial noise of the sample, c is the conditional vector generated by the text encoder and text prompt in the preset large-scale text graph model, α t , σ t and w t To control the parameters of noise strategy and sampling quality, is the image generated by the preset large-scale cultural graph model, x (ori,new) is the training sample image, new represents the fine-tuning sample image, and ori represents the original generated image.

6. The wind control image augmentation method according to claim 4, characterized in that: The second augmented model based on pre-training performs inter-class semantic augmentation on the risk control target segmentation map to obtain the inter-class difficult class sample maps of different risk control target categories, including: Based on the preset feature extraction model, feature extraction is performed on the risk control target segmentation map to obtain sample feature data; Inputting the sample characteristic data into the preset diffusion model to obtain a diffusion sample image; The diffusion sample images are screened based on a preset screening and filtering mechanism, and unsupervised clustering is performed on the retained diffusion sample images to obtain a new risk control target category to update the risk control target category; The updated sample feature data corresponding to the risk control target category is input into the preset diffusion model to obtain the inter-class difficult class sample map.

7. The wind control image augmentation method according to any one of claims 1 to 6, characterized in that: The step of fusing the risk control target segmentation map and the risk control target augmented map into the risk control scene image to obtain the risk control augmented image includes: Paste the risk control target segmentation map and the risk control target augmented map within the image range of the risk control scene image to obtain an image to be processed; The image to be processed is subjected to transition smoothing processing based on a preset fusion algorithm to obtain the wind control augmented image.

8. A wind control image augmentation device, characterized in that: include: The risk control target extraction module is used to segment the risk control target screenshot to obtain a risk control target segmentation map; A risk control target generation module, used to perform semantic augmentation on the risk control target segmentation map to obtain a risk control target augmented map; The risk control target fusion module is used to fuse the risk control target segmentation map and the risk control target augmented map into the risk control scene image to obtain a risk control augmented image.

9. A computer device, characterized in that: The computer device comprises: a memory and at least one processor, wherein instructions are stored in the memory; The at least one processor calls the instructions in the memory to enable the computer device to execute the wind control image augmentation method according to any one of claims 1 to 7.

10. A computer-readable storage medium having instructions stored thereon, characterized in that: When the instructions are executed by the processor, the wind control image augmentation method according to any one of claims 1 to 7 is implemented.

Images