A multi-scale image data enhancement method, apparatus, device, and storage medium

Abstract

This application discloses a multi-scale image data enhancement method, apparatus, device, and storage medium, relating to the field of image processing technology. The method includes: segmenting an original industrial scene image to obtain a binary mask image of a background region conforming to a preset abnormal target type; acquiring the original binary mask image of the abnormal target and scaling its foreground to generate a scale-adjusted foreground binary mask image of the abnormal target; acquiring a foreground image corresponding to the foreground binary mask image, determining the placement position of the foreground image in the original industrial scene image, pasting the foreground image to the placement position, and performing fusion processing to obtain a fused image; eroding the foreground binary mask image, generating a multi-stage final binary mask image based on the eroded foreground binary mask image and the fused image; and inputting the final binary mask image and the original industrial scene image into an image inpainting model to obtain an enhanced image. This application can generate high-quality, diverse abnormal target images that blend naturally with the background.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to a multi-scale image data enhancement method, apparatus, device, and storage medium. Background Technology

[0002] With the rapid development of artificial intelligence and computer vision technologies, image editing and generation technologies are increasingly being applied in intelligent monitoring of industrial environments (such as power transmission lines, industrial plants, and production equipment monitoring). High-quality training data is indispensable for improving the robustness and generalization ability of image recognition models in industrial scenarios. However, in practical applications, abnormal events in industrial environments (such as fires, smoke, and unauthorized vehicle intrusions) are often sporadic, uncontrollable, and have scarce samples, making it difficult for existing training data to cover various complex real-world situations.

[0003] However, existing data augmentation methods mainly rely on geometric transformations (such as rotation, flipping, and cropping) and color perturbations. While these methods are simple and effective, they cannot generate images of anomalous targets with complex structures and reasonable semantics. Deep learning-driven image editing and generation models (such as Stable Diffusion) offer new approaches to image-level data augmentation. For example, instruction-tuned image editing methods can modify image content based on text instructions, but they lack spatial awareness of the insertion location of anomalous targets. This can lead to problems such as unreasonable insertion positions, anomalous target size that does not match the actual scene, and distortion of background edges after insertion. Consequently, the generated anomalous target images are of poor quality and fail to meet the realism requirements of industrial scenarios for anomalous target simulation. Summary of the Invention

[0004] This application provides a multi-scale image data enhancement method, apparatus, device, and storage medium, which solves the problem of poor quality of abnormal target images generated by existing data enhancement methods. It can efficiently generate high-quality and diverse abnormal target images with reasonable abnormal content location and natural integration with the background, meeting the needs of industrial scenarios for the simulation of the realism of abnormal targets, and providing more diverse and high-quality sample data for the training of image recognition models in industrial scenarios.

[0005] Firstly, this application provides a multi-scale image data augmentation method, including:

[0006] Obtain the original industrial scene image, perform semantic segmentation on the original industrial scene image, and obtain a binary mask image of the background region that conforms to the preset abnormal target type;

[0007] Obtain the original binary mask image of the abnormal target corresponding to the preset abnormal target type, and adjust the scale of the foreground part in the original binary mask image to generate a scale-adjusted foreground binary mask image of the abnormal target.

[0008] Obtain a foreground image corresponding to the foreground binary mask image of the abnormal target; determine the placement position of the foreground image in the original industrial scene image based on the binary mask image of the background region; paste the foreground image to the placement position; and perform fusion processing on the pixel values ​​of the foreground image and the pixel values ​​corresponding to the placement position in the original industrial scene image to obtain a fused image.

[0009] An erosion operation is performed on the foreground binary mask image of the abnormal target. Based on the foreground binary mask image after the erosion operation and the fused image, multiple final binary mask images at different stages are generated.

[0010] The final binary mask images from multiple different stages and the original industrial scene image are input into a preset image restoration model for image restoration to obtain an enhanced image.

[0011] Optionally, the step of semantically segmenting the original industrial scene image to obtain a binary mask image of the background region that conforms to a preset abnormal target type includes:

[0012] The original industrial scene image is input into a first preset segmentation model to obtain the semantic segmentation result of the original industrial scene image, wherein the semantic segmentation result includes a binary mask image corresponding to multiple semantic label regions;

[0013] Select semantic tag regions that match the preset abnormal target type from the multiple semantic tag regions;

[0014] The binary mask image corresponding to the selected adapted semantic label region is used as the binary mask image of the background region.

[0015] Optionally, obtaining the original binary mask image of the abnormal target corresponding to the preset abnormal target type includes:

[0016] Obtain a four-channel brush image dataset of anomalies corresponding to the preset anomaly target type; wherein, the first three channels of the four-channel brush image dataset of the anomaly target represent the visual content of the anomaly target, and the fourth channel represents the regional range of the anomaly target.

[0017] Obtain the fourth channel data from the four-channel brush image dataset of the abnormal target;

[0018] The pixels with pixel values ​​greater than a preset threshold in the fourth channel data are used to form the foreground region of the abnormal target, and the pixels with pixel values ​​less than the preset threshold in the fourth channel data are used to form the background region, so as to obtain the original binary mask image of the abnormal target.

[0019] Optionally, the step of fusing the pixel values ​​of the foreground image with the pixel values ​​corresponding to the placement position in the original industrial scene image to obtain a fused image includes:

[0020] The first pixel value of each pixel in the foreground image and the second pixel value of each pixel corresponding to the placement position in the original industrial scene image are obtained respectively.

[0021] The first pixel value of each pixel in the foreground image and the second pixel value of each corresponding pixel in the original industrial scene image are weighted and summed according to a preset weight to obtain the fused pixel value of each pixel.

[0022] The pixel values ​​of the fused pixels are replaced with the pixel values ​​of the pixels corresponding to the placement positions in the original industrial scene image to obtain the fused image.

[0023] Optionally, the step of generating multiple final binary mask images at different stages based on the foreground binary mask image after the erosion operation and the fused image includes:

[0024] Randomly select a foreground pixel from the foreground binary mask image after the erosion operation, and determine the coordinate position of the foreground pixel;

[0025] The coordinate positions and the fused image are input into a second preset segmentation model. Based on the coordinate positions, the second preset segmentation model performs multi-stage segmentation processing on the fused image to generate multiple final binary mask images at different stages.

[0026] Optionally, the step of inputting the final binary mask images from multiple different stages and the original industrial scene image into a preset image inpainting model for image inpainting to obtain an enhanced image includes:

[0027] Based on the final binary mask image, determine the abnormal target region in the original industrial scene image from which to generate abnormal target content;

[0028] Obtain the contextual semantic information of the non-target anomaly region in the original industrial scene image;

[0029] The preset image restoration model generates abnormal target content within the abnormal target area that is fused with the background of the original industrial scene image, based on the target abnormal region and the contextual semantic information, to obtain an enhanced image.

[0030] Optionally, after generating the scale-adjusted foreground binary mask of the anomalous target, the following steps are also included:

[0031] The initial placement position of the foreground binary mask of the abnormal target is randomly generated within the binary mask image of the background region;

[0032] Determine whether the initial placement position meets preset constraints. If the initial placement position does not meet the preset constraints, regenerate the placement position until the preset constraints are met, thus obtaining the final placement position.

[0033] Paste the foreground binary mask of the abnormal target into the final placement position to obtain the final binary mask.

[0034] Secondly, this application provides a multi-scale image data enhancement apparatus, comprising:

[0035] The semantic segmentation module is used to acquire the original industrial scene image, perform semantic segmentation on the original industrial scene image, and obtain a binary mask image of the background region that conforms to the preset abnormal target type.

[0036] The foreground mask generation module is used to obtain the original binary mask of the abnormal target corresponding to the preset abnormal target type, and to adjust the scale of the foreground part in the original binary mask to generate the scale-adjusted foreground binary mask of the abnormal target.

[0037] The fusion module is used to obtain a foreground image corresponding to the foreground binary mask image of the abnormal target, determine the placement position of the foreground image in the original industrial scene image based on the binary mask image of the background region, paste the foreground image to the placement position, and perform fusion processing on the pixel values ​​of the foreground image and the pixel values ​​corresponding to the placement position in the original industrial scene image to obtain a fused image.

[0038] The final mask image generation module is used to perform an erosion operation on the foreground binary mask image of the abnormal target, and generate multiple final binary mask images at different stages based on the foreground binary mask image after the erosion operation and the fused image.

[0039] The image restoration module is used to input the final binary mask images from multiple different stages and the original industrial scene image into a preset image restoration model to perform image restoration and obtain an enhanced image.

[0040] Thirdly, this application provides a multi-scale image data enhancement device, comprising:

[0041] One or more processors;

[0042] A memory that stores one or more programs that, when executed by one or more processors, cause the one or more processors to implement the multi-scale image data enhancement method as described in the first aspect.

[0043] Fourthly, this application provides a storage medium containing computer-executable instructions, which, when executed by a computer processor, are used to perform the multi-scale image data enhancement method as described in the first aspect.

[0044] In summary, compared with the prior art, the beneficial effects of the technical solution provided in this application include at least the following:

[0045] In this application, an original industrial scene image is acquired, and semantic segmentation is performed on the original industrial scene image to obtain a binary mask image of the background region that conforms to a preset abnormal target type. An original binary mask image of the abnormal target corresponding to the preset abnormal target type is acquired, and the scale of the foreground portion in the original binary mask image is adjusted to generate a scale-adjusted foreground binary mask image of the abnormal target. A foreground image corresponding to the foreground binary mask image of the abnormal target is acquired, and the placement position of the foreground image in the original industrial scene image is determined based on the binary mask image of the background region. The foreground image is pasted to the placement position, and the pixel values ​​of the foreground image are fused with the pixel values ​​corresponding to the placement position in the original industrial scene image to obtain a fused image. An erosion operation is performed on the foreground binary mask image of the abnormal target. Based on the eroded foreground binary mask image and the fused image, multiple final binary masks at different stages are generated. The multiple final binary masks at different stages and the original industrial scene image are input into a preset image inpainting model for image inpainting to obtain an enhanced image. In the aforementioned technical methods, the scale of the foreground portion in the original binary mask image is adjusted to generate a scale-adjusted foreground binary mask image of the anomalous target. By performing multi-scale adjustments on the foreground portion of the anomalous target, anomalous target images of various sizes can be generated, enriching the scale diversity of anomalous samples. After pasting the foreground image to the placement position, the pixel values ​​of the foreground image are fused with the pixel values ​​corresponding to the placement position in the original industrial scene image, which can eliminate obvious traces of pasting edges. Finally, based on the final binary mask image from multiple stages and the contextual semantic information of the original industrial scene image, the image inpainting model efficiently generates high-quality, diverse anomalous target images with reasonable anomalous content positions and natural integration with the background. This meets the need for realistic simulation of anomalous targets in industrial scenes and provides more diverse and high-quality sample data for training industrial scene image recognition models. Attached Figure Description

[0046] Figure 1 This is a flowchart of a multi-scale image data enhancement method provided in an embodiment of this application;

[0047] Figure 2 This is a flowchart of obtaining the original binary mask image of an abnormal target provided in an embodiment of this application;

[0048] Figure 3 This is a flowchart provided in an embodiment of the present application for fusing pixel values ​​of the foreground image with pixel values ​​corresponding to the placement position in the original industrial scene image;

[0049] Figure 4 This is a flowchart illustrating the generation of a final binary mask image at multiple different stages, as provided in an embodiment of this application.

[0050] Figure 5 This is a flowchart provided in this application embodiment of the process of inputting the final binary mask images of multiple different stages and the original industrial scene image into a preset image restoration model for image restoration;

[0051] Figure 6 This is a schematic diagram of the structure of a multi-scale image data enhancement device provided in an embodiment of this application;

[0052] Figure 7 This is a schematic diagram of the structure of a multi-scale image data enhancement device provided in an embodiment of this application. Detailed Implementation

[0053] To make the objectives, technical solutions, and advantages of this application clearer, specific embodiments of this application will be described in further detail below with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are merely for explaining this application and not for limiting it. It should also be noted that, for ease of description, only the parts relevant to this application are shown in the drawings, not all of them. Before discussing exemplary embodiments in more detail, it should be mentioned that some exemplary embodiments are described as processes or methods depicted as flowcharts. Although the flowcharts describe operations (or steps) as sequential processes, many of these operations can be performed in parallel, concurrently, or simultaneously. Furthermore, the order of the operations can be rearranged. A process can be terminated when its operation is completed, but it may also have additional steps not included in the drawings. A process can correspond to a method, function, procedure, subroutine, subprogram, etc.

[0054] The terms "first," "second," etc., used in the specification and claims of this application are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such use of data can be interchanged where appropriate so that embodiments of this application can be implemented in orders other than those illustrated or described herein, and the objects distinguished by "first," "second," etc., are generally of the same class and the number of objects is not limited; for example, a first object can be one or more. Furthermore, in the specification and claims, "and / or" indicates at least one of the connected objects, and the character " / " generally indicates that the preceding and following objects are in an "or" relationship.

[0055] Existing data augmentation methods mainly rely on geometric transformations (such as rotation, flipping, and cropping) and color perturbations. While these methods are simple and effective, they cannot generate images of anomalous targets with complex structures and semantically plausible meanings. Deep learning-driven image editing and generation models (such as Stable Diffusion) offer new approaches to image-level data augmentation. For example, instruction-tuned image editing methods can modify image content based on text instructions, but they lack spatial awareness regarding the insertion location of anomalous targets. This can lead to problems such as unreasonable insertion positions, anomalous target size that does not match the actual scene, and distortion of background edges after insertion. Consequently, the generated anomalous target images are of poor quality and fail to meet the realism requirements of industrial scenarios for anomalous target simulation.

[0056] To address the issue of poor-quality anomalous target images generated by existing data augmentation methods, this application aims to efficiently generate high-quality anomalous target images with reasonable anomalous locations, diverse scales, and natural integration with the background. This meets the demand for realistic simulation of anomalous targets in industrial scenarios and provides more diverse and high-quality sample data for training industrial scene image recognition models. This application provides a multi-scale image data augmentation method. The method adjusts the scale of the foreground portion of the original binary mask image to generate a scale-adjusted foreground binary mask image of the anomalous target. By adjusting the foreground portion of the anomalous target at multiple scales, it can generate anomalous target images of various sizes, enriching the scale diversity of anomalous samples. After pasting the foreground image to the placement location, the pixel values ​​of the foreground image are fused with the pixel values ​​corresponding to the placement location in the original industrial scene image, eliminating obvious traces of pasting edges. Finally, the image inpainting model, based on the multi-stage final binary mask image and the contextual semantic information of the original industrial scene image, efficiently generates high-quality, diverse anomalous target images with reasonable anomalous content locations and natural integration with the background. This meets the demand for realistic simulation of anomalous targets in industrial scenarios and provides more diverse and high-quality sample data for training industrial scene image recognition models.

[0057] The multi-scale image data enhancement method provided in this embodiment can be executed by a multi-scale image data enhancement device, which can be implemented by software and / or hardware. The multi-scale image data enhancement device can be composed of two or more physical entities, or it can be composed of a single physical entity.

[0058] The multi-scale image data augmentation device may have an operating system installed, including but not limited to Android, Linux, and Windows. The device may install at least one application based on the operating system; this application may be a built-in application of the operating system or an application downloaded from a third-party device or server. In this embodiment, the multi-scale image data augmentation device has at least one application capable of executing multi-scale image data augmentation methods.

[0059] The technical solutions provided by the embodiments of this application are described in detail below with reference to the accompanying drawings.

[0060] Figure 1 A flowchart of a multi-scale image data enhancement method provided in an embodiment of this application is given. Please refer to... Figure 1 The multi-scale image data augmentation method specifically includes:

[0061] S110. Obtain the original industrial scene image, perform semantic segmentation on the original industrial scene image, and obtain a binary mask image of the background region that conforms to the preset abnormal target type.

[0062] The original industrial scene image can be understood as an image reflecting the normal state of an industrial environment. Semantic segmentation refers to the process of dividing image pixels into regions with specific semantic meanings and assigning a corresponding category label to each region. In this embodiment, semantic segmentation is performed using the Segment Anything Model 2 (SAM2) segmentation model. Utilizing its pre-trained semantic segmentation capabilities, binary mask images of background regions that match the abnormal target type are selected and extracted based on the abnormal target type. For example, the steps of using the SAM2 segmentation model to perform semantic segmentation on the original industrial scene image to obtain binary mask images of background regions that conform to the preset abnormal target type include S1101-S1103:

[0063] S1101. Input the original industrial scene image into the first preset segmentation model to obtain the semantic segmentation result of the original industrial scene image, wherein the semantic segmentation result includes a binary mask image corresponding to multiple semantic label regions.

[0064] For example, the first preset segmentation model can adopt the SAM2 (Segment Anything Model 2) semantic segmentation model. When the original industrial scene image is input into the SAM2 model, the SAM2 model first performs multi-scale feature extraction on the original image through the encoder to generate a feature map containing global context information. Then, the decoder classifies each pixel in the image based on the semantic information in the feature map, identifies the different semantic label regions (such as ground, road, building, sky, etc.), and generates a binary mask map corresponding to each semantic label region. The binary mask map is stored in the form of a two-dimensional matrix. The value rule of each pixel in the matrix is: if the pixel belongs to the predetermined semantic label region, the matrix value at the corresponding position is 1; if it does not belong to the semantic label region, the matrix value is 0. For example, for the semantic label region of ground, in its corresponding binary mask map, all pixels identified as ground have a value of 1, and other pixels have a value of 0.

[0065] S1102. Select semantic tag regions that are compatible with the preset abnormal target type from the multiple semantic tag regions.

[0066] In this application embodiment, a mapping relationship table between abnormal target types and adapted semantic tags is preset. The mapping relationship table defines the semantic environment in which different types of abnormal targets are most likely to appear, that is, one or more semantic tags adapted to the abnormal target type. For example, when the preset abnormal target type is an abnormal vehicle, the adapted semantic tag area in the mapping relationship table is set to ground or road, etc.

[0067] Specifically, after determining the current preset abnormal target type, the preset abnormal target type is used as the query condition to search in the mapping relationship table and retrieve all the matching semantic tags associated with the preset abnormal target type; then, from the multiple semantic tag regions obtained by semantic segmentation of the original industrial scene image, the regions that match the found matching semantic tags are selected to obtain the semantic tag regions that match the preset abnormal target type.

[0068] S1103. The binary mask image corresponding to the selected adapted semantic label region is used as the binary mask image of the background region.

[0069] The background region's binary mask is a two-dimensional matrix representing the set of all pixel positions in the original industrial scene image where an anomaly of a preset anomaly type can be reasonably placed. For each pixel in the two-dimensional matrix corresponding to the background region's binary mask: a value of 1 indicates that the pixel belongs to the background region adapted to the preset anomaly type in the original industrial image; a value of 0 indicates that the pixel does not belong to the background region adapted to the current preset anomaly type in the original industrial image.

[0070] S120. Obtain the original binary mask image of the abnormal target corresponding to the preset abnormal target type, and adjust the scale of the foreground part in the original binary mask image to generate a scale-adjusted foreground binary mask image of the abnormal target.

[0071] The original binary mask of the anomalous target refers to the binary image data used to characterize the spatial distribution range of the anomalous target in the image after extracting specific channel data from a four-channel brush image dataset containing the anomalous target and performing binarization processing with a preset threshold. In the original binary mask of the anomalous target, each pixel has a value of 0 or 1. The set of pixels with a value of 1 in the original binary mask of the anomalous target defines the foreground region of the anomalous target, and the set of pixels with a value of 0 in the original binary mask of the anomalous target defines the background region, so as to ensure that the mask can accurately capture the outline and spatial distribution of the anomalous target.

[0072] For example, Figure 2 This is a flowchart illustrating the process of obtaining the original binary mask image of an abnormal target, as provided in an embodiment of this application. For example... Figure 2 As shown, the steps for obtaining the original binary mask image of the abnormal target specifically include S1201-S1203:

[0073] S1201. Obtain a four-channel brush image dataset of anomalies corresponding to the preset anomaly target type; wherein, the first three channels of the four-channel brush image dataset of the anomaly target represent the visual content of the anomaly target, and the fourth channel represents the regional range of the anomaly target.

[0074] The four-channel brush image dataset for anomalous targets is a collection of images used to provide visual and spatial information about these targets. Each four-channel brush image dataset contains four independent data channels. The first three channels are the red (R), green (G), and blue (B) primary color channels, which together constitute the color, texture, and morphological details of the anomalous target, as well as the visual features of background scene elements (such as the ground, equipment, and walls in an industrial scene), thus representing the visual content of the anomalous target. The fourth channel refers to the spatial distribution boundary of the anomalous target in the image, i.e., the area of ​​the anomalous target.

[0075] Specifically, based on the current preset abnormal target type, the corresponding abnormal target is determined; and using the determined abnormal target as the query condition, the four-channel brush image dataset of the abnormal target is filtered out from the abnormal target brush image data resource library.

[0076] S1202. Obtain the fourth channel data from the four-channel brush image dataset of the abnormal target.

[0077] S1203. Pixels with pixel values ​​greater than a preset threshold in the fourth channel data are used to form the foreground region of the abnormal target, and pixels with pixel values ​​less than the preset threshold in the fourth channel data are used to form the background region, so as to obtain the original binary mask image of the abnormal target.

[0078] Specifically, the pixel value corresponding to each pixel in the fourth channel data (the pixel value of each pixel is usually 0-255) is traversed. The pixel value corresponding to each pixel is compared with a preset threshold (which can be set to 125). If the pixel value is greater than the preset threshold, it is determined that the pixel belongs to the foreground region corresponding to the abnormal target, and the pixel value of the pixel is assigned to 1. If the pixel value is less than the preset threshold, it is determined that the pixel belongs to the background region corresponding to the abnormal target, and the pixel value of the pixel is assigned to 0. All pixels with pixel values ​​greater than the preset threshold form the complete foreground region of the abnormal target, and all pixels with pixel values ​​less than the preset threshold in the fourth channel data form the complete background region, thereby obtaining the original binary mask image of the abnormal target.

[0079] After obtaining the original binary mask image of the anomalous target, the foreground portion of the original binary mask image needs to be scaled according to a custom scaling factor (such as scaling factors of 1, 1.5, 3, etc.) to obtain foreground binary masks of anomalous targets at different scales. The scaling factor indicates the multiple by which the foreground portion is scaled up proportionally. Specifically, based on each custom scaling factor, the foreground portion of the original binary mask image is scaled up proportionally to obtain foreground binary masks of anomalous targets at multiple scales.

[0080] This application embodiment can simulate the visual presentation effect of abnormal targets at different scales by adjusting the scale of the foreground part in the original binary mask image, effectively expanding the diversity of training data. In actual industrial scenarios, abnormal targets (such as fire, smoke, abnormal vehicles, etc.) may appear at different distances and viewpoints, and appear to be of different sizes. Multi-scale adjustment can better simulate the size changes in real scenes, thereby improving the generalization ability of the recognition model obtained from subsequent training.

[0081] S130. Obtain a foreground image corresponding to the foreground binary mask image of the abnormal target. Determine the placement position of the foreground image in the original industrial scene image based on the binary mask image of the background region. Paste the foreground image to the placement position and perform fusion processing on the pixel values ​​of the foreground image and the pixel values ​​corresponding to the placement position in the original industrial scene image to obtain a fused image.

[0082] The foreground image refers to a color image of the visual information of the abnormal target corresponding to the scale-adjusted binary mask image of the abnormal target's foreground, which contains complete visual appearance information of the abnormal target.

[0083] The binary mask of the background region refers to the set of all pixel positions in the original industrial scene image where the abnormal target corresponding to the preset abnormal target type can be reasonably placed. Therefore, the placement position of the foreground image can be determined based on the binary mask of the background region. To eliminate obvious traces of pasted edges, this embodiment of the application also performs a fusion process on the pixel values ​​of the foreground image and the pixel values ​​corresponding to the placement positions in the original industrial scene image.

[0084] For example, Figure 3 This is a flowchart illustrating the fusion process between pixel values ​​in the foreground image and pixel values ​​corresponding to the placement position in the original industrial scene image, as provided in an embodiment of this application. Figure 3 As shown, the steps for fusing the pixel values ​​of the foreground image with the pixel values ​​corresponding to the placement positions in the original industrial scene image specifically include S1301-S1303:

[0085] S1301. Obtain the first pixel value of each pixel in the foreground image and the second pixel value of each pixel corresponding to the placement position in the original industrial scene image.

[0086] S1302. The first pixel value of each pixel in the foreground image and the second pixel value of each corresponding pixel in the original industrial scene image are weighted and summed according to a preset weight to obtain the pixel value after fusion of each pixel.

[0087] Specifically, in this embodiment of the application, the formula for calculating the fused pixel value of each pixel is as follows:

[0088]

[0089] in, Foreground image at pixel position The first pixel value on, Original industrial scene image at the same location The second pixel value on, The preset fusion weights.

[0090] S1303. Replace the pixel values ​​of the fused pixels with the pixel values ​​of the pixels corresponding to the placement positions in the original industrial scene image to obtain the fused image.

[0091] Image fusion refers to the composite image obtained by pixel-level merging of a foreground image of an abnormal target with an original industrial scene image at a specified placement location. This eliminates obvious edge artifacts that occur when pasting the foreground image into the original industrial scene image. Specifically, based on the determined placement location and the dimensions (height and width) of the foreground image, a corresponding replacement region is identified in the original industrial scene image. Each pixel within this replacement region is traversed, and the merged pixel value is used to replace the pixel value of the corresponding pixel within the replacement region.

[0092] S140. Perform an erosion operation on the foreground binary mask image of the abnormal target, and generate multiple final binary mask images at different stages based on the foreground binary mask image after the erosion operation and the fused image.

[0093] Erosion is a fundamental mathematical morphology image processing algorithm primarily used to reduce the size of foreground objects in an image and eliminate small, isolated noise points. Specifically, a square structuring element is defined, the size of which can be set according to actual needs (e.g., 3×3, 5×5, etc.). The erode function is called, with iterations set to 1-3 times, to erode the foreground binary mask image. The formula is as follows:

[0094]

[0095] in, For the defined square structuring element, The number of iterations ( ), where erode is the erosion function.

[0096] The erosion operation can shrink the edge region of the foreground binary mask, causing the boundary of the foreground binary mask to shrink towards the center; this can effectively reduce the impact of mask edge noise points on subsequent operations, and at the same time make the subsequently selected pixels more biased towards the center of the foreground mask.

[0097] For example, Figure 4 This is a flowchart illustrating the generation of a final binary mask image at multiple different stages, as provided in an embodiment of this application. For example... Figure 4 As shown, the steps for generating the final binary mask image at multiple different stages specifically include S1401-S1402:

[0098] S1401. Randomly select a foreground pixel from the foreground binary mask image after the erosion operation, and determine the coordinate position of the foreground pixel.

[0099] For example, in this embodiment of the application, a foreground pixel coordinate position can be randomly selected using a pseudo-random number generation algorithm. Specifically, this includes: traversing the foreground binary mask image after the erosion operation, collecting the coordinate positions of all foreground pixels, and obtaining a set of foreground pixel coordinates; secondly, obtaining the total number of coordinates in the set of foreground pixels, and using the length of the set as the boundary, generating a random integer in the interval [0, set length - 1] as the target index using a pseudo-random number generation algorithm; finally, directly using the coordinates corresponding to the target index as the position of the selected foreground pixel.

[0100] S1402. Input the coordinate position and the fused image into the second preset segmentation model. The second preset segmentation model performs multi-stage segmentation processing on the fused image based on the coordinate position to generate multiple final binary mask images at different stages.

[0101] The second preset segmentation model is preferably the Segment Anything Model (SAM) segmentation model.

[0102] Specifically, the encoder in the second preset segmentation model extracts features from the fused image based on an attention mechanism to capture global context and local semantic information, resulting in a feature map rich in semantic information. The decoder of the second preset segmentation model first expands the feature map using a region growing algorithm (region growing refers to gradually connecting and merging pixels with similar attributes to form a continuous region) centered on the input coordinates, generating a coarse final binary mask image in the initial stage. This initial final binary mask image identifies the suspected abnormal target region centered on the coordinate position. In each subsequent stage, the decoder performs boundary optimization, i.e., region expansion, on the final binary mask image of the previous stage, gradually refining the contour details of the abnormal target to obtain final binary masks at different stages. The final binary mask image output at each stage represents the segmentation result of the current stage, thus obtaining a multi-stage final binary mask image from coarse to fine.

[0103] By fusing an image with a coordinate position, a diverse set of binary masks representing different segmentation granularities is derived, generating multiple possible segmentation perspectives for the same anomalous scene and enriching the diversity of subsequently generated anomalous target content.

[0104] S150. Input the final binary mask images from multiple different stages and the original industrial scene image into a preset image restoration model to perform image restoration and obtain an enhanced image.

[0105] In this application embodiment, the enhanced image refers to modifying the original industrial scene image, adding or synthesizing abnormal target content (such as fire, smoke, abnormal vehicles, etc.), thereby generating new and diversified training samples.

[0106] For example, Figure 5 This is a flowchart provided in this application embodiment, showing how to input final binary mask images from multiple different stages and the original industrial scene image into a preset image inpainting model for image inpainting. For example... Figure 5 As shown, the steps for image inpainting by inputting the final binary mask images from multiple different stages and the original industrial scene image into a preset image inpainting model specifically include S1501-S1503:

[0107] S1501. Based on the final binary mask image, determine the abnormal target region in the original industrial scene image from which the abnormal target content to be generated is to be determined.

[0108] The final binary mask image is used to indicate the specific location and area range in the original industrial scene image where abnormal target content needs to be generated. Therefore, based on the final binary mask image, the abnormal target area in the original industrial scene image where abnormal target content is to be generated can be determined.

[0109] S1502. Obtain the contextual semantic information of the non-target abnormal region in the original industrial scene image.

[0110] The contextual semantic information of non-target anomaly regions refers to the visual feature information of the scene that can be described from all other regions (i.e., non-target anomaly regions) in the original industrial scene image, except for the target anomaly region. Specifically, the preset image inpainting model extracts semantic information from the non-target anomaly regions in the original industrial scene image to obtain the contextual semantic information of the non-target anomaly regions, i.e., visual feature information, such as color distribution, texture structure, lighting conditions, and scene layout.

[0111] S1503. Based on the target abnormal region and the contextual semantic information, the preset image restoration model generates abnormal target content that is fused with the background of the original industrial scene image within the abnormal target region, thereby obtaining an enhanced image.

[0112] In this embodiment, the abnormal target region defined by the final binary mask image is used as the target abnormal region of the content to be generated. Subsequently, a preset image inpainting model generates initial abnormal target content within the abnormal target region based on learned prior knowledge of the abnormal target (which includes the shape and contour information, visual information, etc. of the abnormal target) and the contextual semantic information of the non-target region. During the generation of the initial abnormal target content, the preset image inpainting model uses a self-attention mechanism to capture the dependencies and semantic associations between all pixels in the non-target region and the pixels of the abnormal target content to be generated in the abnormal target region. Based on these dependencies and semantic associations, the model adjusts the local texture details of the abnormal target content to ensure that the generated initial abnormal target content blends seamlessly with the background of the original industrial scene image. Finally, convolution operations are used to enhance local features and smooth edges of the generated initial abnormal target content to eliminate splicing marks and generate abnormal content that blends naturally with the background, resulting in an enhanced image.

[0113] This application embodiment inputs the final binary mask images of multiple different stages and the original industrial scene image into a preset image restoration model for image restoration. Under the guidance of the final binary mask at different stages, it can obtain enhanced images at each stage, that is, it can generate abnormal targets with different manifestations (e.g., from small flames to large fires with smoke), simulating different stages of the development of abnormal events or different observation angles, thereby providing high-quality and diverse training samples for industrial scene image recognition models.

[0114] In another embodiment of this application, after generating the scale-adjusted foreground binary mask image of the anomalous target, the method further includes:

[0115] The initial placement position of the foreground binary mask of the abnormal target is randomly generated within the binary mask image of the background region; specifically, the initial placement position of the foreground binary mask can be randomly generated within the binary mask image of the background region using a pseudo-random number generation algorithm. This initial placement position is determined by coordinate values. It means that, among them, and These represent the horizontal and vertical coordinates of the foreground binary mask image in the background mask image, respectively.

[0116] Determine whether the initial placement position meets the preset constraints. If the initial placement position does not meet the preset constraints, regenerate the placement position until the preset constraints are met, thus obtaining the final placement position. Specifically, to ensure that the foreground binary mask image is completely within the background area, the preset constraints that must be met are:

[0117]

[0118] in, and These represent the height and width of the scaled foreground binary mask image, respectively. and The height and width of the binary mask image for the background region are defined. If the randomly generated initial placement position does not meet the above preset constraints (i.e., the foreground portion exceeds the background region), the placement position is regenerated until a placement position that meets the conditions is found.

[0119] The foreground binary mask of the anomalous target is pasted to the final placement position to obtain the final binary mask. In this embodiment, by randomly mapping the foreground binary mask of the anomalous target onto the binary mask of the background region, the randomness of the anomalous target's appearance can be simulated through randomized positioning.

[0120] Finally, the final binary mask image and the original industrial scene image can be input into the aforementioned preset image inpainting model for image inpainting. The model generates abnormal target content that is integrated with the background of the original industrial scene image within the abnormal target area, thus obtaining an enhanced image. The steps include: the preset image inpainting model extracts semantic information from the non-target abnormal areas in the original industrial scene image to obtain the contextual semantic information of the non-target abnormal areas, i.e., visual feature information, such as color distribution, texture structure, lighting conditions, and scene layout. The abnormal target region defined by the final binary mask image is used as the target abnormal region for the content to be generated. Subsequently, the preset image inpainting model generates initial abnormal target content within the abnormal target region based on the learned prior knowledge of the abnormal target (including the shape and outline information, visual information, etc. of the abnormal target) and the contextual semantic information of the non-target region. During the generation of the initial abnormal target content, the preset image inpainting model uses a self-attention mechanism to capture the dependencies and semantic associations between all pixels in the non-target region and the pixels of the abnormal target content to be generated in the abnormal target region. Based on these dependencies and semantic associations, the local texture details of the abnormal target content are adjusted to ensure that the generated initial abnormal target content blends seamlessly with the background of the original industrial scene image. Finally, convolution operations are used to enhance the local features and smooth the edges of the generated initial abnormal target content to eliminate splicing marks and generate abnormal content that blends naturally with the background, resulting in an enhanced image.

[0121] In summary, the multi-scale image data enhancement method provided in this application involves: acquiring an original industrial scene image; performing semantic segmentation on the original industrial scene image to obtain a binary mask image of a background region conforming to a preset abnormal target type; acquiring the original binary mask image of the abnormal target corresponding to the preset abnormal target type, and scaling the foreground portion of the original binary mask image to generate a scale-adjusted foreground binary mask image of the abnormal target; acquiring the foreground image corresponding to the foreground binary mask image of the abnormal target, determining the placement position of the foreground image in the original industrial scene image based on the binary mask image of the background region, pasting the foreground image to the placement position, and fusing the pixel values ​​of the foreground image with the pixel values ​​corresponding to the placement position in the original industrial scene image to obtain a fused image; performing an erosion operation on the foreground binary mask image of the abnormal target, and generating multiple final binary masks at different stages based on the eroded foreground binary mask image and the fused image; and inputting the multiple final binary masks at different stages and the original industrial scene image into a preset image inpainting model for image inpainting to obtain an enhanced image. In the aforementioned technical methods, the scale of the foreground portion in the original binary mask image is adjusted to generate a scale-adjusted foreground binary mask image of the anomalous target. By performing multi-scale adjustments on the foreground portion of the anomalous target, anomalous target images of various sizes can be generated, enriching the scale diversity of anomalous samples. After pasting the foreground image to the placement position, the pixel values ​​of the foreground image are fused with the pixel values ​​corresponding to the placement position in the original industrial scene image, which can eliminate obvious traces of pasting edges. Finally, based on the final binary mask image from multiple stages and the contextual semantic information of the original industrial scene image, the image inpainting model efficiently generates high-quality, diverse anomalous target images with reasonable anomalous content positions and natural integration with the background. This meets the need for realistic simulation of anomalous targets in industrial scenes and provides more diverse and high-quality sample data for training industrial scene image recognition models.

[0122] Figure 6 This is a schematic diagram of a multi-scale image data enhancement device provided in an embodiment of this application. (Reference) Figure 6 The multi-scale image data enhancement device provided in this embodiment specifically includes: a semantic segmentation module 21, a foreground mask generation module 22, a fusion module 23, a final mask generation module 24, and an image restoration module 25.

[0123] Semantic segmentation module 21 is used to acquire the original industrial scene image, perform semantic segmentation on the original industrial scene image, and obtain a binary mask image of the background region that conforms to the preset abnormal target type.

[0124] The foreground mask generation module 22 is used to obtain the original binary mask of the abnormal target corresponding to the preset abnormal target type, and to adjust the scale of the foreground part in the original binary mask to generate the scale-adjusted foreground binary mask of the abnormal target.

[0125] The fusion module 23 is used to obtain a foreground image corresponding to the foreground binary mask image of the abnormal target, determine the placement position of the foreground image in the original industrial scene image based on the binary mask image of the background region, paste the foreground image to the placement position, and perform fusion processing on the pixel values ​​of the foreground image and the pixel values ​​corresponding to the placement position in the original industrial scene image to obtain a fused image.

[0126] The final mask image generation module 24 is used to perform an erosion operation on the foreground binary mask image of the abnormal target, and generate multiple final binary mask images at different stages based on the foreground binary mask image after the erosion operation and the fused image.

[0127] The image restoration module 25 is used to input the final binary mask images from multiple different stages and the original industrial scene image into a preset image restoration model to perform image restoration and obtain an enhanced image.

[0128] Based on the above embodiments, the semantic segmentation module 21 includes:

[0129] The segmentation unit is used to input the original industrial scene image into a first preset segmentation model to obtain the semantic segmentation result of the original industrial scene image, wherein the semantic segmentation result includes a binary mask image corresponding to multiple semantic label regions.

[0130] A filtering unit is used to filter out semantic tag regions that are compatible with a preset abnormal target type from multiple semantic tag regions;

[0131] The background region binary mask image determination unit is used to use the binary mask image corresponding to the adapted semantic label region as the background region binary mask image.

[0132] Based on the above embodiments, the foreground mask generation module 22 includes:

[0133] The dataset acquisition unit is used to acquire a four-channel brush image dataset of anomalies corresponding to the preset anomaly target type; wherein, the first three channels of the four-channel brush image dataset of the anomaly target represent the visual content of the anomaly target, and the fourth channel represents the regional range of the anomaly target.

[0134] The fourth channel data acquisition unit is used to acquire fourth channel data from the four-channel brush image dataset of the abnormal target;

[0135] The original binary mask image determination unit is used to form the foreground region of the abnormal target by selecting pixels with pixel values ​​greater than a preset threshold in the fourth channel data, and to form the background region by selecting pixels with pixel values ​​less than the preset threshold in the fourth channel data, so as to obtain the original binary mask image of the abnormal target.

[0136] Based on the above embodiments, the fusion module 23 includes:

[0137] A pixel value acquisition unit is used to acquire the first pixel value of each pixel in the foreground image and the second pixel value of each pixel corresponding to the placement position in the original industrial scene image.

[0138] The pixel value weighted summation unit is used to perform weighted summation calculation on the first pixel value of each pixel in the foreground image and the second pixel value of each corresponding pixel in the original industrial scene image according to a preset weight, so as to obtain the pixel value after fusion of each pixel.

[0139] The fusion unit is used to replace the pixel values ​​of the fused pixels with the pixel values ​​of the pixels corresponding to the placement positions in the original industrial scene image to obtain the fused image.

[0140] Based on the above embodiments, the final mask image generation module 24 includes:

[0141] The coordinate position determination unit is used to randomly select a foreground pixel from the foreground binary mask image after the erosion operation and determine the coordinate position of the foreground pixel.

[0142] The final mask image generation unit is used to input the coordinate positions and the fused image into a second preset segmentation model. The second preset segmentation model performs multi-stage segmentation processing on the fused image based on the coordinate positions to generate multiple final binary mask images at different stages.

[0143] Based on the above embodiments, the image restoration module 25 includes:

[0144] An abnormal target region determination unit is used to determine the abnormal target region in the original industrial scene image to be generated with abnormal target content based on the final binary mask image.

[0145] A semantic information acquisition unit is used to acquire contextual semantic information of non-target abnormal regions in the original industrial scene image;

[0146] An image restoration unit is used to generate abnormal target content that is fused with the background of the original industrial scene image within the abnormal target area based on the target abnormal region and the contextual semantic information, thereby obtaining an enhanced image.

[0147] Based on the above embodiments, after generating the scale-adjusted foreground binary mask image of the anomalous target, the method further includes:

[0148] The initial placement position of the foreground binary mask of the abnormal target is randomly generated within the binary mask image of the background region;

[0149] Determine whether the initial placement position meets preset constraints. If the initial placement position does not meet the preset constraints, regenerate the placement position until the preset constraints are met, thus obtaining the final placement position.

[0150] Paste the foreground binary mask of the abnormal target into the final placement position to obtain the final binary mask.

[0151] In summary, the multi-scale image data enhancement device provided in this application acquires an original industrial scene image, performs semantic segmentation on the original industrial scene image to obtain a binary mask image of a background region that conforms to a preset abnormal target type; acquires an original binary mask image of an abnormal target corresponding to the preset abnormal target type, and performs scale adjustment on the foreground part of the original binary mask image to generate a scale-adjusted foreground binary mask image of the abnormal target; acquires a foreground image corresponding to the foreground binary mask image of the abnormal target, determines the placement position of the foreground image in the original industrial scene image based on the binary mask image of the background region, pastes the foreground image to the placement position, and performs fusion processing on the pixel values ​​of the foreground image and the pixel values ​​corresponding to the placement position in the original industrial scene image to obtain a fused image; performs an erosion operation on the foreground binary mask image of the abnormal target, and generates multiple final binary masks at different stages based on the eroded foreground binary mask image and the fused image; inputs the multiple final binary masks at different stages and the original industrial scene image into a preset image inpainting model for image inpainting to obtain an enhanced image. In the aforementioned technical methods, the scale of the foreground portion in the original binary mask image is adjusted to generate a scale-adjusted foreground binary mask image of the anomalous target. By performing multi-scale adjustments on the foreground portion of the anomalous target, anomalous target images of various sizes can be generated, enriching the scale diversity of anomalous samples. After pasting the foreground image to the placement position, the pixel values ​​of the foreground image are fused with the pixel values ​​corresponding to the placement position in the original industrial scene image, which can eliminate obvious traces of pasting edges. Finally, based on the final binary mask image from multiple stages and the contextual semantic information of the original industrial scene image, the image inpainting model efficiently generates high-quality, diverse anomalous target images with reasonable anomalous content positions and natural integration with the background. This meets the need for realistic simulation of anomalous targets in industrial scenes and provides more diverse and high-quality sample data for training industrial scene image recognition models.

[0152] The multi-scale image data enhancement apparatus provided in this application embodiment can be used to execute the multi-scale image data enhancement method provided in the above embodiment, and has corresponding functions and beneficial effects.

[0153] Figure 7 This is a schematic diagram of the structure of a multi-scale image data enhancement device provided in an embodiment of this application, with reference to... Figure 7 The multi-scale image data enhancement device includes a processor 31, a memory 32, a communication device 33, an input device 34, and an output device 35. The number of processors 31 and the number of memories 32 in the multi-scale image data enhancement device can be one or more. The processor 31, memory 32, communication device 33, input device 34, and output device 35 of the multi-scale image data enhancement device can be connected via a bus or other means.

[0154] The memory 32, as a computer-readable storage medium, can be used to store software programs, computer-executable programs, and modules, such as program instructions / modules corresponding to the multi-scale image data enhancement method in any embodiment of this application (e.g., semantic segmentation module 21, foreground mask generation module 22, fusion module 23, final mask generation module 24, and image restoration module 25 in the multi-scale image data enhancement device). The memory 32 may primarily include a program storage area and a data storage area. The program storage area may store the operating system and at least one application program required for a function; the data storage area may store data created based on the use of the device, etc. Furthermore, the memory 32 may include high-speed random access memory and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other non-volatile solid-state storage device. In some instances, the memory may further include memory remotely located relative to the processor, and these remote memories can be connected to the device via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0155] The communication device 33 is used for data transmission.

[0156] The processor 31 executes various functional applications and data processing of the device by running software programs, instructions and modules stored in the memory 32, thereby realizing the above-mentioned multi-scale image data enhancement method.

[0157] Input device 34 can be used to receive input digital or character information, and to generate key signal inputs related to user settings and function control of the device. Output device 35 may include display devices such as a display screen.

[0158] The multi-scale image data enhancement device provided above can be used to execute the multi-scale image data enhancement method provided in the above embodiments, and has corresponding functions and beneficial effects.

[0159] This application embodiment also provides a storage medium containing computer-executable instructions. When executed by a computer processor, the computer-executable instructions are used to perform a multi-scale image data enhancement method. This multi-scale image data enhancement method includes: acquiring an original industrial scene image; performing semantic segmentation on the original industrial scene image to obtain a binary mask image of a background region conforming to a preset abnormal target type; acquiring an original binary mask image of an abnormal target corresponding to the preset abnormal target type; and scaling the foreground portion of the original binary mask image to generate a scale-adjusted foreground binary mask image of the abnormal target; acquiring the foreground binary mask image of the abnormal target... The foreground image corresponding to the background binary mask is used to determine its placement position in the original industrial scene image. The foreground image is then pasted into the designated position, and its pixel values ​​are fused with those of the corresponding placement position in the original industrial scene image to obtain a fused image. An erosion operation is performed on the foreground binary mask of the abnormal target. Based on the eroded foreground binary mask and the fused image, multiple final binary masks at different stages are generated. These final binary masks and the original industrial scene image are then input into a preset image inpainting model for image inpainting to obtain an enhanced image.

[0160] Storage medium – any type of memory device or storage device. The term “storage medium” is intended to include: mounting media, such as CD-ROM, floppy disk, or magnetic tape devices; computer system memory or random access memory, such as DRAM, DDR RAM, SRAM, EDO RAM, Rambus RAM, etc.; non-volatile memory, such as flash memory, magnetic media (e.g., hard disk or optical storage); registers or other similar types of memory elements, etc. Storage medium may also include other types of memory or combinations thereof. Furthermore, storage medium may reside in a first computer system in which the program is executed, or it may reside in a different second computer system connected to the first computer system via a network (such as the Internet). The second computer system can provide program instructions to the first computer for execution. The term “storage medium” can include two or more storage media residing in different locations (e.g., in different computer systems connected via a network). Storage medium may store program instructions (e.g., specifically implemented as a computer program) executable by one or more processors.

[0161] Of course, the computer-executable instructions provided in the embodiments of this application are not limited to the multi-scale image data enhancement method described above, but can also perform related operations in the multi-scale image data enhancement method provided in any embodiment of this application.

[0162] The multi-scale image data enhancement apparatus, storage medium, and multi-scale image data enhancement device provided in the above embodiments can execute the multi-scale image data enhancement method provided in any embodiment of this application. For technical details not described in detail in the above embodiments, please refer to the multi-scale image data enhancement method provided in any embodiment of this application.

[0163] The above description is merely a preferred embodiment and the technical principles employed in this application. This application is not limited to the specific embodiments described herein, and various obvious changes, readjustments, and substitutions that can be made by those skilled in the art will not depart from the scope of protection of this application. Therefore, although this application has been described in detail through the above embodiments, this application is not limited to the above embodiments, and may include many other equivalent embodiments without departing from the concept of this application. The scope of this application is determined by the scope of the claims.

Claims

1. A multi-scale image data enhancement method, characterized in that, include: Obtain the original industrial scene image, perform semantic segmentation on the original industrial scene image, and obtain a binary mask image of the background region that conforms to the preset abnormal target type; Obtain the original binary mask image of the abnormal target corresponding to the preset abnormal target type, and adjust the scale of the foreground part in the original binary mask image to generate a scale-adjusted foreground binary mask image of the abnormal target. Obtain a foreground image corresponding to the foreground binary mask image of the abnormal target; determine the placement position of the foreground image in the original industrial scene image based on the binary mask image of the background region; paste the foreground image to the placement position; and perform fusion processing on the pixel values ​​of the foreground image and the pixel values ​​corresponding to the placement position in the original industrial scene image to obtain a fused image. An erosion operation is performed on the foreground binary mask image of the abnormal target. Based on the foreground binary mask image after the erosion operation and the fused image, multiple final binary mask images at different stages are generated. The final binary mask images from multiple different stages and the original industrial scene image are input into a preset image restoration model for image restoration to obtain an enhanced image.

2. The multi-scale image data enhancement method according to claim 1, characterized in that, The step of semantically segmenting the original industrial scene image to obtain a binary mask image of the background region that conforms to a preset abnormal target type includes: The original industrial scene image is input into a first preset segmentation model to obtain the semantic segmentation result of the original industrial scene image, wherein the semantic segmentation result includes a binary mask image corresponding to multiple semantic label regions; Select semantic tag regions that match the preset abnormal target type from the multiple semantic tag regions; The binary mask image corresponding to the selected adapted semantic label region is used as the binary mask image of the background region.

3. The multi-scale image data enhancement method according to claim 1, characterized in that, The step of obtaining the original binary mask image of the abnormal target corresponding to the preset abnormal target type includes: Obtain a four-channel brush image dataset of anomalies corresponding to the preset anomaly target type; wherein, the first three channels of the four-channel brush image dataset of the anomaly target represent the visual content of the anomaly target, and the fourth channel represents the regional range of the anomaly target. Obtain the fourth channel data from the four-channel brush image dataset of the abnormal target; The pixels with pixel values ​​greater than a preset threshold in the fourth channel data are used to form the foreground region of the abnormal target, and the pixels with pixel values ​​less than the preset threshold in the fourth channel data are used to form the background region, so as to obtain the original binary mask image of the abnormal target.

4. The multi-scale image data enhancement method according to claim 1, characterized in that, The step of fusing the pixel values ​​of the foreground image with the pixel values ​​corresponding to the placement position in the original industrial scene image to obtain a fused image includes: The first pixel value of each pixel in the foreground image and the second pixel value of each pixel corresponding to the placement position in the original industrial scene image are obtained respectively. The first pixel value of each pixel in the foreground image and the second pixel value of each corresponding pixel in the original industrial scene image are weighted and summed according to a preset weight to obtain the fused pixel value of each pixel. The pixel values ​​of the fused pixels are replaced with the pixel values ​​of the pixels corresponding to the placement positions in the original industrial scene image to obtain the fused image.

5. The multi-scale image data enhancement method according to claim 1, characterized in that, The process of generating multiple final binary mask images at different stages based on the foreground binary mask image after the erosion operation and the fused image includes: Randomly select a foreground pixel from the foreground binary mask image after the erosion operation, and determine the coordinate position of the foreground pixel; The coordinate positions and the fused image are input into a second preset segmentation model. Based on the coordinate positions, the second preset segmentation model performs multi-stage segmentation processing on the fused image to generate multiple final binary mask images at different stages.

6. The multi-scale image data enhancement method according to claim 1, characterized in that, The step of inputting the final binary mask images from multiple different stages and the original industrial scene image into a preset image inpainting model for image inpainting to obtain an enhanced image includes: Based on the final binary mask image, determine the abnormal target region in the original industrial scene image from which to generate abnormal target content; Obtain the contextual semantic information of the non-target anomaly region in the original industrial scene image; The preset image restoration model generates abnormal target content within the abnormal target area that is fused with the background of the original industrial scene image, based on the target abnormal region and the contextual semantic information, to obtain an enhanced image.

7. The multi-scale image data enhancement method according to claim 1, characterized in that, After generating the scale-adjusted foreground binary mask of the anomalous target, the following steps are also included: The initial placement position of the foreground binary mask of the abnormal target is randomly generated within the binary mask image of the background region; Determine whether the initial placement position meets preset constraints. If the initial placement position does not meet the preset constraints, regenerate the placement position until the preset constraints are met, thus obtaining the final placement position. Paste the foreground binary mask of the abnormal target into the final placement position to obtain the final binary mask.

8. A multi-scale image data enhancement device, characterized in that, include: The semantic segmentation module is used to acquire the original industrial scene image, perform semantic segmentation on the original industrial scene image, and obtain a binary mask image of the background region that conforms to the preset abnormal target type. The foreground mask generation module is used to obtain the original binary mask of the abnormal target corresponding to the preset abnormal target type, and to adjust the scale of the foreground part in the original binary mask to generate the scale-adjusted foreground binary mask of the abnormal target. The fusion module is used to obtain a foreground image corresponding to the foreground binary mask image of the abnormal target, determine the placement position of the foreground image in the original industrial scene image based on the binary mask image of the background region, paste the foreground image to the placement position, and perform fusion processing on the pixel values ​​of the foreground image and the pixel values ​​corresponding to the placement position in the original industrial scene image to obtain a fused image. The final mask image generation module is used to perform an erosion operation on the foreground binary mask image of the abnormal target, and generate multiple final binary mask images at different stages based on the foreground binary mask image after the erosion operation and the fused image. The image restoration module is used to input the final binary mask images from multiple different stages and the original industrial scene image into a preset image restoration model to perform image restoration and obtain an enhanced image.

9. A multi-scale image data enhancement device, characterized in that, include: One or more processors; A memory that stores one or more programs that, when executed by one or more processors, enable the one or more processors to implement the multi-scale image data enhancement method as described in any one of claims 1-7.

10. A storage medium containing computer-executable instructions, characterized in that, The computer-executable instructions, when executed by a computer processor, are used to perform the multi-scale image data enhancement method as described in any one of claims 1-7.

Images