Image enhancement method and device, electronic equipment, storage medium and program product

Abstract

The invention provides an image enhancement method and device, electronic equipment, a storage medium and a program product. The method comprises the following steps: dividing an original image to obtain a plurality of image blocks; identifying whether a target defect exists in the image block, wherein the proportion of a long side to a wide side in a minimum bounding rectangular frame corresponding to the target defect is greater than a first preset value; for the image block containing the target defect, marking the target defect on the image block according to the form trend of the target defect to obtain a plurality of marking boxes; the proportion of the long edge to the wide edge of the marking frame is smaller than a second preset value; and combining the adjacent marking frames belonging to the same defect to obtain a plurality of images containing defect marking frames. According to the method, the defect that the length-width ratio is greater than the first preset value is identified, and the defect is segmented into the plurality of balanced labeling boxes of which the length-width ratios are smaller than the second preset value, so that the problem that the length-width ratios of the labeling boxes are unbalanced is fundamentally avoided, and useful features can be extracted from the enhanced image.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and more specifically, to an image enhancement method, apparatus, electronic device, storage medium, and program product. Background Technology

[0002] Industrial defect detection is a crucial step in automated production, directly impacting product quality control and production efficiency. Traditional defect detection methods primarily rely on manual visual inspection, which suffers from inherent drawbacks such as low efficiency, poor stability, and high cost. With the remarkable performance of deep learning technology, especially convolutional neural networks (CNNs) in image recognition, deep learning-based object detection frameworks (such as YOLO and Faster R-CNN) have been widely applied in industrial defect detection.

[0003] The accuracy of deep learning models heavily relies on a large amount of high-quality, labeled training data. However, in industrial practice, defect samples are typically extremely scarce (due to high yield rates on production lines), and the diverse forms of defects lead to severe imbalances in the dataset. To address this issue, existing technologies primarily employ image enhancement methods such as rotation, flipping, and color dithering. For elongated defects, due to the significant difference in length and width, their features are almost nonexistent in the image after scaling, making it impossible to extract useful features. Summary of the Invention

[0004] The purpose of this application is to provide an image enhancement method, apparatus, electronic device, storage medium, and program product to address the annotation of elongated defects, thereby enabling the extraction of useful features from the enhanced image.

[0005] In a first aspect, embodiments of this application provide an image enhancement method, including: The original image is divided into multiple image blocks; Identify whether a target defect exists in the image block, wherein the ratio of the long side to the wide side of the minimum bounding rectangle corresponding to the target defect is greater than a first preset value; For an image block containing the target defect, the target defect on the image block is marked according to the shape and direction of the target defect to obtain multiple annotation boxes; the ratio of the long side to the wide side of the annotation box is less than a second preset value; the second preset value is less than the first preset value; Adjacent bounding boxes belonging to the same defect are merged to obtain multiple images containing defect bounding boxes, with each image containing defect bounding boxes of different sizes.

[0006] This application embodiment identifies defects with aspect ratios greater than a first preset value and divides them into multiple balanced annotation boxes with aspect ratios less than a second preset value, fundamentally avoiding the problem of unbalanced aspect ratios of the annotation boxes, thereby enabling the extraction of useful features from the enhanced image.

[0007] In one possible implementation of the first aspect, the step of annotating the target defect on the image patch according to the morphological orientation of the target defect includes: The image patch is input into the defect recognition model to obtain the defect mask image corresponding to the target defect; The defect mask image is thinned by iteratively removing boundary pixels until its width is only 1 pixel, thereby obtaining the skeleton line of the target defect. The division length is determined based on the preset target aspect ratio and the maximum width of the target defect; Based on the specified division length, starting from the beginning of the skeleton line, a dividing point is set at intervals of the specified length to obtain multiple skeleton segments; Obtain the pixels of the defect mask image corresponding to the skeleton segment, and use PCA to calculate the directional principal components of the pixel segment; wherein, the direction corresponding to the maximum value of the principal component is the defect direction; Calculate the smallest bounding rectangle that is perpendicular to the defect direction and can enclose the pixels of the skeleton segment, and obtain the annotation box corresponding to the skeleton segment.

[0008] This application embodiment extracts the skeleton segment of the defect, determines the defect direction based on the skeleton segment, and then marks the skeleton segment based on the defect direction. The skeleton segment can accurately describe the main line of the defect, laying the foundation for subsequent reasonable segmentation.

[0009] In one possible implementation of the first aspect, merging adjacent bounding boxes belonging to the same defect to obtain multiple images containing defect bounding boxes includes: The coordinates of the bounding boxes in all image blocks are converted into global coordinates based on the position of the image blocks in the original image. The distance between the nearest edges of the two bounding boxes is calculated based on the global coordinates of the image patch. If the distance is less than a preset threshold, the defects in the two bounding boxes are determined to be the same defect. Based on the spatial location and morphological continuity of the target defect, the bounding boxes of the same defect are merged to reconstruct images containing defect bounding boxes at multiple scales.

[0010] This application embodiment merges bounding boxes belonging to the same defect to obtain images at multiple scales, and the merging is based on global coordinates, which improves the accuracy and reliability of the merging.

[0011] In one possible implementation of the first aspect, the method further includes: Identify micro-defects in the image block whose defect size is smaller than a preset size; The smallest bounding rectangle of the micro-defect whose spatial distance is less than a preset distance is used as the annotation box.

[0012] This application embodiment combines multiple small defects that are spatially close into a larger bounding box, thereby increasing the target size and improving the model's sensitivity and recall to small defects, while avoiding missed detections.

[0013] In one possible implementation of the first aspect, the method further includes: Based on the physical causes of the defects contained in the image patch, the morphological transformation is used to simulate the appearance of the defects under different surface materials and lighting conditions to obtain an enhanced image patch of the defects. Based on the texture features of the defects contained in the image patch, a multi-scale transformation is performed to obtain an enhanced image of the defects; Based on changes in lighting, viewing angle shifts, and surface deformations in the industrial environment, an enhanced image corresponding to the image block containing the defects is generated.

[0014] The embodiments of this application expand the samples used for model training by performing image enhancement based on factors such as defect morphology, texture, and lighting.

[0015] In one possible implementation of the first aspect, after obtaining multiple images containing defect bounding boxes, the method further includes: Obtain the image containing the defect annotation boxes and their corresponding location information in the original image; For each preset cropping size, the cropping matrix information corresponding to the image containing the defect annotation box is calculated based on the position information; the cropping matrix information includes the coordinates of the top left corner of the cropping matrix in the original image; The corresponding cropped image block is cropped from the original image based on the coordinates of the upper left corner and the preset cropping size.

[0016] The embodiments of this application can crop out images containing defects at multiple scales based on the original image, thereby reducing the loss of defect information.

[0017] In one possible implementation of the first aspect, the method further includes: The cropped image patch is scaled to the size required by the model.

[0018] The embodiments of this application scale the cropped images at multiple scales to the size required by the model, thereby satisfying the requirements for model training.

[0019] In one possible implementation of the first aspect, the method further includes: The enhanced images are used as training samples to train the industrial defect detection model.

[0020] Secondly, embodiments of this application provide an image enhancement apparatus, comprising: The image segmentation module is used to divide the original image into multiple image blocks; The defect identification module is used to identify whether there is a target defect in the image block, wherein the ratio of the long side to the wide side of the minimum bounding rectangle corresponding to the target defect is greater than a first preset value; The defect annotation module is used to annotate the target defect on the image block containing the target defect according to the shape and direction of the target defect, and obtain multiple annotation boxes; the ratio of the long side to the wide side of the annotation box is less than a second preset value; the second preset value is less than the first preset value; The annotation box merging module is used to merge adjacent annotation boxes belonging to the same defect to obtain multiple enhanced images containing defect annotation boxes, and the size of the defect annotation boxes contained in each enhanced image is different.

[0021] Thirdly, embodiments of this application provide an electronic device, including: a processor, a memory, and a bus, wherein: The processor and the memory communicate with each other via the bus; The memory stores program instructions that can be executed by the processor, and the processor can execute the method of the first aspect by calling the program instructions.

[0022] Fourthly, embodiments of this application provide a non-transitory computer-readable storage medium, comprising: The non-transitory computer-readable storage medium stores computer instructions that cause the computer to perform the methods in various possible implementations of the first aspect.

[0023] Fifthly, embodiments of this application provide a computer program product, including computer program instructions, which, when read and executed by a processor, perform the methods in various possible implementations of the first aspect.

[0024] Other features and advantages of this application will be set forth in the following description and will be apparent in part from the description or may be learned by practicing embodiments of this application. The objectives and other advantages of this application may be realized and obtained by means of the structures particularly pointed out in the written description, claims, and drawings. Attached Figure Description

[0025] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments of this application will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0026] Figure 1 This is a schematic flowchart of an image enhancement method provided in an embodiment of this application; Figure 2 A schematic diagram of defect annotation provided for an embodiment of this application; Figure 3 This is a schematic diagram of another image enhancement method provided in an embodiment of this application; Figure 4 This is a schematic diagram of an image enhancement device structure provided in an embodiment of this application; Figure 5 This is a schematic diagram of the physical structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0027] The embodiments of the technical solution of this application will now be described in detail with reference to the accompanying drawings. These embodiments are only used to more clearly illustrate the technical solution of this application and are therefore merely examples, and should not be used to limit the scope of protection of this application.

[0028] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains; the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the application; the terms “comprising” and “having”, and any variations thereof, in the specification, claims, and foregoing description of the drawings are intended to cover non-exclusive inclusion.

[0029] In the description of the embodiments of this application, technical terms such as "first" and "second" are used only to distinguish different objects and should not be construed as indicating or implying relative importance or implicitly specifying the number, specific order, or primary and secondary relationship of the indicated technical features. In the description of the embodiments of this application, "multiple" means two or more, unless otherwise explicitly defined.

[0030] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0031] In the description of the embodiments in this application, the term "and / or" is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. Additionally, the character " / " in this document generally indicates that the preceding and following related objects have an "or" relationship.

[0032] In the description of the embodiments of this application, the term "multiple" refers to two or more (including two), similarly, "multiple sets" refers to two or more (including two sets), and "multiple pieces" refers to two or more (including two pieces).

[0033] In the description of the embodiments of this application, the technical terms "center," "longitudinal," "lateral," "length," "width," "thickness," "upper," "lower," "front," "rear," "left," "right," "vertical," "horizontal," "top," "bottom," "inner," "outer," "clockwise," "counterclockwise," "axial," "radial," and "circumferential" indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are only for the convenience of describing the embodiments of this application and simplifying the description, and are not intended to indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on the embodiments of this application.

[0034] In the description of the embodiments of this application, unless otherwise expressly specified and limited, technical terms such as "installation," "connection," "joining," and "fixing" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral part; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; they can refer to the internal communication of two components or the interaction between two components. For those skilled in the art, the specific meaning of the above terms in the embodiments of this application can be understood according to the specific circumstances.

[0035] Industrial defect detection is a crucial step in automated production, aiming to ensure product quality and prevent defective products from entering the market. After acquiring images of industrial products, the resulting images have extremely high resolution. Directly inputting these ultra-high-resolution images into deep learning models would impose a significant computational burden. Therefore, a sliding window approach is typically used to crop the ultra-high-resolution image into image patches, and multiple image patches are inferred in parallel using batch input. In real-world industrial scenarios, a single ultra-high-resolution image often contains only one defect. After cropping into image patches, the defect may only be distributed across a single patch, resulting in a significant difference in data volume compared to the data required by deep learning models. Therefore, to address the problem of insufficient data, researchers have proposed various data augmentation methods to increase the diversity and quantity of training data, mainly including: 1. Traditional data augmentation methods: By performing operations such as rotating, flipping horizontally or vertically, and randomly cropping images, more diverse training data can be generated; the brightness, contrast, saturation, etc. of images can be adjusted to simulate images under different lighting conditions.

[0036] 2. Data generation based on Generative Adversarial Networks (GANs): Generative adversarial networks (GANs) are used to generate new image data. Adversarial training makes the generated images more realistic and increases the diversity of the dataset.

[0037] 3. Synthetic data: By artificially adding defects to normal images, synthetic defective images are generated to increase the amount of training data.

[0038] On the other hand, cropping image patches can prevent deep learning models from extracting sufficient contextual information, thus hindering accurate defect identification. Furthermore, industrial products often have complex backgrounds; for example, display panels typically feature various circuit patterns. The dimensions of different defects can also vary significantly; for instance, a crack might only be 2-3 pixels wide in an image, posing a significant challenge to accurate detection. In summary, defect detection in real-world industrial scenarios faces challenges such as limited defect samples, weak defect features, complex backgrounds, and diverse defect types, resulting in poor accuracy and robustness. To address these issues, multi-scale methods are widely used in defect detection. Common multi-scale methods include: Image pyramid: A technique for representing images at multiple scales. By progressively reducing the resolution of images, a sequence of images is formed, with each image being a scaled version of the previous one.

[0039] Feature Pyramid: A structure used in deep learning models, especially in Convolutional Neural Networks (CNNs). It extracts features from different layers of the network to form feature maps of different scales, making it easier to capture objects of different sizes.

[0040] Multi-scale Convolution: In convolutional neural networks, convolutional kernels of different sizes can be used to capture features at different scales. This approach allows the network to learn both local and global features simultaneously within a single convolutional layer.

[0041] Existing deep learning solutions typically rely on large amounts of labeled data for model training. However, data acquisition is difficult in industrial settings, and traditional data augmentation methods, such as flipping and rotating, cannot provide sufficient data diversity, especially in terms of data morphology and background complexity. While data synthesis methods directly paste defects onto images with different backgrounds, solving the background complexity problem, the defects and background cannot be well integrated, preventing the model from learning true image features. GAN-based data generation methods, on the other hand, are limited by computational resources and the difficulty of GAN training, significantly increasing training costs.

[0042] In current mainstream object detection frameworks, multi-scale methods are often used to extract effective features from images, enabling the model to identify defects at different scales and thus be more robust. However, multi-scale methods primarily focus on feature extraction and do not consider the multi-scale nature of the input and output. Image pyramids, on the other hand, can lead to the loss of some image information, especially for fine-grained objects such as cracks, which occupy very few pixels. After image scaling, their features are almost non-existent in the image, and the model naturally cannot extract the corresponding features.

[0043] To address the aforementioned technical problems, this application provides an image enhancement method. It is understood that the method provided in this application can be applied to electronic devices, including terminals and servers. Specifically, the terminal can be a smartphone, tablet computer, computer, personal digital assistant (PDA), etc.; the server can be an application server or a web server.

[0044] Figure 1 This is a schematic diagram of an image enhancement method provided in an embodiment of this application, such as... Figure 1 As shown, the method includes: Step 101: Divide the original image into multiple image blocks; Step 102: Identify whether there is a target defect in the image block, wherein the ratio of the long side to the wide side of the minimum bounding rectangle corresponding to the target defect is greater than a first preset value; Step 103: For an image block containing the target defect, mark the target defect on the image block according to the shape and direction of the target defect to obtain multiple annotation boxes; the ratio of the long side to the wide side of the annotation box is less than a second preset value; the second preset value is less than the first preset value; Step 104: Merge adjacent bounding boxes belonging to the same defect to obtain multiple enhanced images containing defect bounding boxes, and the size of the defect bounding boxes in each enhanced image is different.

[0045] In step 101, images acquired by industrial cameras are typically ultra-high resolution (e.g., tens of millions of pixels), and directly inputting them into a deep learning model would impose a huge computational burden. This step reduces the large image to smaller ones, making subsequent processing feasible. Specifically, a sliding window with a fixed step size is used to divide the original image into uniformly sized image blocks (e.g., 640×640 pixels). The step size is usually set to be smaller than the window size (e.g., 512 pixels), allowing overlapping areas between adjacent blocks and ensuring the complete capture of defects across blocks. In industrial scenarios, a single ultra-high resolution image typically contains only 1-2 defects, and after cropping, the vast majority of image blocks are negative samples (defect-free). To balance the dataset, a dynamic sampling strategy is adopted: sampling weights are increased for defective regions, while defect-free regions are randomly and sparsely sampled. A bidirectional mapping table is established between the image block coordinate system and the original image's global coordinate system, recording the absolute coordinates (x0, y0) of the top-left corner of each image block in the original image, providing a spatial reference for subsequent cross-block annotation and merging.

[0046] In step 102, a pre-trained, lightweight defect recognition model (such as a binary classification CNN) is used to perform preliminary screening on each image patch to obtain initial bounding boxes containing defects within the image patch. The minimum bounding rectangle (MBR) of the labeled defect region within the image patch is calculated, obtaining the pixel values ​​of the longer side L and the wider side W. A first preset value is set based on industrial experience and the physical characteristics of the defect; this first preset value is typically T1 ∈ [10:1, 30:1]. For example: monitor crack: T1 = 15:1 (150 pixels long, 10 pixels wide); metal surface scratch: T1 = 20:1 (200 pixels long, 10 pixels wide). If the ratio of the longer side to the wider side within the minimum bounding rectangle is greater than the first preset value, it indicates that the defect is a slender defect, and this type of defect is identified as the target defect.

[0047] In step 103, since the target defect is elongated, multiple small-sized annotation boxes can be used to annotate the target defect, such as... Figure 2 As shown, multiple locally bounding boxes with relatively balanced aspect ratios are used to annotate the target defect. During the annotation process, annotations can be made according to the shape and orientation of the target defect. The ratio of the long side to the wide side of the annotation box is less than a second preset value, and the second preset value is less than a first preset value.

[0048] In step 104, the local bounding boxes in all image patches are mapped back to the global coordinate system of the original high-resolution image based on their location within the patch. Then, the spatial proximity of defects within different boxes is calculated (e.g., the distance between the nearest edges of the bounding boxes). If the distance is less than a threshold, they are determined to belong to the same defect. During merging, two adjacent bounding boxes, three adjacent bounding boxes, or four adjacent bounding boxes can be merged, etc. It should be understood that merging bounding boxes also means merging the corresponding image patches. After merging, multiple enhanced images containing defect bounding boxes can be obtained, and the sizes of these enhanced images can vary.

[0049] This application embodiment identifies defects with aspect ratios greater than a first preset value and divides them into multiple balanced annotation boxes with aspect ratios less than a second preset value, fundamentally avoiding the problem of unbalanced aspect ratios of the annotation boxes, thereby enabling the extraction of useful features from the enhanced image.

[0050] Building upon the above embodiments, the image patch is input into a pre-trained defect recognition model. This model can be an instance segmentation model, such as Mask R-CNN, YOLOv8-SEG, or a Transformer-based model (such as the Segment Anything Model, an industrially fine-tuned version of SAM). The model's output is a defect mask image. This is a binary image where white pixels (typically with a value of 1) represent regions identified as defects by the model, and black pixels (with a value of 0) represent the background.

[0051] The obtained defect mask image is then thinned. This can be achieved using classic image morphology algorithms, such as the Zhang-Suen thinning algorithm. This algorithm iteratively removes boundary pixels layer by layer without disrupting connectivity until the width of the region is only one pixel, ultimately obtaining the skeleton line of the defect. The preset target aspect ratio (i.e., the second preset value in the scheme, such as 3:1) and the maximum width of the target defect estimated from the original mask image are used.

[0052] Based on the principle that segment length ≈ target aspect ratio × maximum defect width, an approximate segment length can be calculated. For example, if the defect width is approximately 5 pixels and the target aspect ratio is 4:1, the segment length can be set to 20 pixels. This ensures that the generated rectangles are not too long and thin.

[0053] Starting from one end of the skeleton line, treat it as a parametric curve. Along the curve, set a segmentation point at intervals of one arc length. The portion between these segmentation points constitutes a skeleton segment. Locate all pixels in the original defect mask region corresponding to this skeleton segment. Use the set of coordinates of these pixels as input for principal component analysis.

[0054] PCA calculates the directions of two principal components and their variances. The direction of the first principal component with the largest variance represents the main direction of the defect pixel distribution in that segment, i.e., the local defect orientation. The calculated defect orientation is used as the orientation angle of the bounding box. The smallest bounding rectangle perpendicular to this direction (i.e., along the minor axis) that exactly encloses all the defect pixels corresponding to the skeleton segment is calculated. This rectangle is the final, orientation-optimized local bounding box.

[0055] This application embodiment extracts the skeleton segment of the defect, determines the defect direction based on the skeleton segment, and then marks the skeleton segment based on the defect direction. The skeleton segment can accurately describe the main line of the defect, laying the foundation for subsequent reasonable segmentation.

[0056] Based on the above embodiments, merging adjacent bounding boxes belonging to the same defect to obtain multiple images containing defect bounding boxes includes: The coordinates of the bounding boxes in all image blocks are converted into global coordinates based on the position of the image blocks in the original image. The distance between the nearest edges of the two bounding boxes is calculated based on the global coordinates of the image patch. If the distance is less than a preset threshold, the defects in the two bounding boxes are determined to be the same defect. Based on the spatial location and morphological continuity of the target defect, the bounding boxes of the same defect are merged to reconstruct images containing defect bounding boxes at multiple scales.

[0057] In the specific implementation process, the coordinates of all image blocks and their corresponding local bounding boxes (usually the coordinates relative to the top left corner of the image block (x_min_local, y_min_local, x_max_local, y_max_local)), as well as the global coordinates of the top left corner of each image block in the original image (x_offset, y_offset).

[0058] For each bounding box, convert its coordinates to global coordinates relative to the top-left corner of the original image. The calculation formula is: x_min_global = x_offset + x_min_local y_min_global = y_offset + y_min_local x_max_global = x_offset + x_max_local y_max_global = y_offset + y_max_local Where (x_min_global, y_max_global) are the coordinates of the top-left corner of the image patch, (x_max_global, y_min_global) are the coordinates of the bottom-right corner of the image patch, and (x_offset, y_offset) are the global coordinates of the top-left corner of the image patch in the original image.

[0059] For any two bounding boxes (Box A and Box B) in the global coordinate system, calculate the Euclidean distance between their two nearest edges. This is more suitable than the traditional Intersection over Union (IoU) for handling slender defect boxes that are connected end-to-end but do not overlap.

[0060] Set a preset distance threshold (e.g., 10-20 pixels). If the calculated nearest edge distance is less than this threshold, and the defect category labels (i.e., containing defects or not containing defects) of the two boxes are the same, then the defects in the two boxes are determined to belong to the same defect.

[0061] Based on the pairwise judgment conditions mentioned above, a clustering algorithm is further employed to automatically group all bounding boxes. Specifically, the global coordinates of the center point of each bounding box are treated as a data point, and the density-based DBSCAN clustering algorithm is used, with the neighborhood radius parameter set to the preset distance threshold. This algorithm can automatically group all spatially connected bounding box points into the same cluster, thereby efficiently and accurately identifying all local bounding boxes belonging to the same complete defect as an "association group," laying the foundation for subsequent merging operations.

[0062] By traversing all box pairs and using clustering algorithms (such as DBSCAN) or disjoint-set data structures, all boxes that meet the criteria are classified into different "association groups", each group representing a complete defect entity.

[0063] For each "associated group" (i.e., a complete defect), calculate the minimum bounding rectangle. Then, gather the coordinates of all local bounding boxes within the group and calculate a new minimum bounding rectangle that encloses all points. This new rectangle spatially covers most of the defect's area, reconstructing its overall scale. For curved defects, curve fitting can be performed first, followed by calculating the bounding rectangle, making the merged result more closely match the shape.

[0064] For this target defect, annotation information at multiple scales can be obtained, namely the original multiple local small boxes and the merged one or more overall large boxes.

[0065] These bounding boxes of different scales are mapped back to image patches of different sizes. For example, small local bounding boxes are mapped to a base image patch of 640x640, while the merged large bounding boxes are mapped to a larger image (e.g., 1280x640) cropped from the original image and scaled down to 640x640.

[0066] Ultimately, multiple training images can be obtained. These training images are derived from different regions or cropped at different sizes of the original image, but all contain bounding boxes of the same defect at different scales.

[0067] This application embodiment merges bounding boxes belonging to the same defect to obtain images at multiple scales, and the merging is based on global coordinates, which improves the accuracy and reliability of the merging.

[0068] Based on the above embodiments, the method further includes: Identify micro-defects in the image block whose defect size is smaller than a preset size; The smallest bounding rectangle of the micro-defect whose spatial distance is less than a preset distance is used as the annotation box.

[0069] In practice, the algorithm iterates through all labeled defect bounding boxes in a given image patch or the entire dataset. For each bounding box, its physical dimensions are calculated. Typically, the area (Area = Width × Height) or diagonal length of the bounding box is used as the metric. For example, the bounding box area of ​​a pinhole defect might only be 10-20 square pixels. A preset size threshold is set (e.g., area < 100 square pixels, or width / height < 10 pixels). Any defect smaller than this threshold is identified and classified as a "micro-defect."

[0070] Calculate the spatial distance between any two bounding boxes of micro-defects. A common method is to calculate the Euclidean distance between the center points of the two boxes. Another method is to calculate the distance between the nearest edges of the two boxes. Set a preset distance threshold (e.g., center point distance < 30 pixels). If the distance between two micro-defects is less than this threshold, they are considered spatially adjacent. Use a clustering algorithm (such as DBSCAN or simple connected component analysis) to group all micro-defects that meet the distance condition together.

[0071] For a group of micro-defects, collect the coordinates of the four vertices of the original bounding boxes of all micro-defects within the group. Based on this set of vertex coordinates, calculate the minimum bounding rectangle that can enclose all vertices. This newly generated rectangle is the final merged annotation box used for training. Its size is much larger than any single individual micro-defect box.

[0072] This application embodiment combines multiple small defects that are spatially close into a larger bounding box, thereby increasing the target size and improving the model's sensitivity and recall to small defects, while avoiding missed detections.

[0073] Based on the above embodiments, the method further includes: Based on the physical causes of the defects contained in the image patch, the morphological transformation is used to simulate the appearance of the defects under different surface materials and lighting conditions to obtain an enhanced image patch of the defects. Based on the texture features of the defects contained in the image patch, a multi-scale transformation is performed to obtain an enhanced image of the defects; Based on changes in lighting, viewing angle shifts, and surface deformations in the industrial environment, an enhanced image corresponding to the image block containing the defects is generated.

[0074] In the specific implementation process, firstly, a prior knowledge base can be established to associate defect types (such as scratches, dents, and stains) with their physical causes. For example: a scratch is a linear groove formed on a surface by a sharp object. A dent is a recessed area formed by a localized impact. A stain is foreign matter adhering to a surface.

[0075] Based on the aforementioned physical model, defects in image patches are simulated and transformed. Specifically, for materials, the same scratch will appear as a high-reflectivity stripe on a metal surface, while appearing as a dull line on a frosted plastic surface. Generative enhancement can be implemented as follows: For morphological simulation, a deformation field is generated and image distortion is applied to simulate the three-dimensional shape of defects such as dents, and the visual representation of defects such as scratches is adjusted based on a material reflection characteristic lookup table. For texture synthesis, texture features are extracted from real defects and transferred to normal images at different scales and blending degrees using a texture synthesis algorithm. Simultaneously, complex lighting conditions and viewpoint changes are simulated by applying lighting models and random affine transformations. All enhancement processes ensure that the original annotation information is transformed synchronously with the image to generate effective training sample pairs. For different lighting conditions, the shadow effects of defects under different lighting angles and intensities are simulated. For example, directional filtering or rendering techniques are used to change the shadow length and depth of dents to simulate the shape when light shines from different directions. For texture extraction and representation, texture features are extracted from the original defect region (e.g., using Gabor filters, local binary mode LBP, or deep learning features) to characterize the microstructure of the defect.

[0076] When scaling image patches, scaling can be done not only at the image level but also at the texture level, allowing for multi-scale compositing. For example, in addition to generating normal-sized stain textures, it is possible to generate sparser or denser stain texture variants.

[0077] When performing texture blending, extracted or generated defect textures can be naturally integrated into a normal background image. This differs from simple "pasting" and instead considers edge blending and brightness adaptation, making the synthesized defects appear to be "grown" on the product surface rather than floating on the surface.

[0078] When simulating changes in lighting, it can simulate common lighting instability conditions in industrial settings, such as uneven lighting, localized overexposure / underexposure, and flash. This can be achieved by randomly adjusting the Gamma value and contrast of image patches, or by adding simulated lighting gradient maps.

[0079] When simulating perspective shift, it is possible to simulate minute angular changes between the camera and the inspected product caused by installation or production line vibrations. This can be achieved by performing minute rotations, stretching, and distortions on the image block using affine or perspective transformations.

[0080] When simulating surface deformation, for non-rigid products or flexible materials, the effect of natural undulations or slight deformations on the appearance of defects can be simulated. This can be achieved through elastic deformation algorithms.

[0081] The embodiments of this application expand the samples used for model training by performing image enhancement based on factors such as defect morphology, texture, and lighting.

[0082] Based on the above embodiments, after obtaining multiple images containing defect annotation boxes, the method further includes: Obtain the image containing the defect annotation boxes and their corresponding location information in the original image; For each preset cropping size, the cropping matrix information corresponding to the image containing the defect annotation box is calculated based on the position information; the cropping matrix information includes the coordinates of the top left corner of the cropping matrix in the original image; The corresponding cropped image block is cropped from the original image based on the coordinates of the upper left corner and the preset cropping size.

[0083] In the specific implementation process, after obtaining multiple images containing defect-labeled bounding boxes, in order to further expand the diversity of training data and introduce richer contextual information, this application embodiment uses the global location information of the labeled defects in the original image to intelligently and purposefully crop out image boxes of various sizes from the original image, thereby providing the model with multi-scale learning samples.

[0084] A set of cutting dimensions can be preset according to actual needs, such as: 640×640, 640×1280, 1280×640, 1280×1920. These dimensions represent different aspect ratios and field of view.

[0085] The cropping is not arbitrary, but centered around key points. Typically, this center point is chosen as the global coordinates (cx, cy) of the center point of the defect annotation box on the original image. This ensures that cropping at different scales revolves around the defect.

[0086] The selection of the center point is determined based on the distribution range of the defect: for defects entirely located within a single image patch, the center point of the smallest bounding rectangle formed by all its local bounding boxes is used as the cropping center; for defects spanning multiple image patches, the complete bounding box of the defect is first reconstructed through a bounding box merging mechanism, and its center point in the global coordinate system is used as the unified cropping center. This strategy ensures that regardless of the defect distribution range, multi-scale cropping can completely preserve the defect's main body and contextual information based on its geometric center. For each preset size (W, H), the global coordinates (X, Y) of the upper left corner of the cropping area are calculated with reference to the center point (cx, cy). The calculation formula is: X = cx - (W / 2), Y = cy - (H / 2). For example, if a 1280x640 image patch is cropped with the center point (1600, 2240), then the coordinates of the upper left corner are (1600-640, 2240-320) = (960, 1920).

[0087] The four parameters (X, Y, W, H) collectively define a cropping matrix, which crops a rectangular region of width W and height H starting from the (X, Y) point of the original image. Using image processing libraries (such as OpenCV's array slicing or PIL's Image.crop() function), the corresponding image region is cropped from the original ultra-high-resolution image based on the cropping matrix information (X, Y, W, H). When the calculated cropping region exceeds the original image boundary (i.e., X or Y is negative, or X+W is greater than the original image width, or Y+H is greater than the original image height), processing is necessary to ensure consistent output size. A common strategy is padding, which involves first cropping the effective portion and then filling the missing areas with specified pixel values ​​(such as 0, mean, or edge pixels) to ensure the final image patch size remains W×H. Therefore, a set of image patches of varying sizes (e.g., 640x640, 640x1280, etc.) can be obtained, each containing the defective region of interest and varying degrees of surrounding background information.

[0088] The embodiments of this application can crop out images containing defects at multiple scales based on the original image, thereby reducing the loss of defect information.

[0089] Based on the above embodiments, after obtaining enhanced images at multiple scales, these enhanced images are converted into a fixed input size required by the model through a scaling algorithm, such as 640×640 pixels required by the YOLOv8 model.

[0090] Scaling algorithms include: nearest neighbor interpolation, bilinear interpolation, bicubic interpolation, and region interpolation.

[0091] To scale a 1280×640 image patch to 640×640, the process is as follows: Step 1: Determine the target size. The model requires a fixed input size, such as target_size = (640, 640).

[0092] Step 2: Calculate the scaling ratio. Calculate the scaling ratio for both width and height separately: scale_x = target_width / original_width = 640 / 1280 = 0.5 scale_y = target_height / original_height = 640 / 640 = 1.0 In this example, only the width direction needs to be scaled down, while the height remains unchanged.

[0093] Step 3: Perform interpolation transformation. Use the selected interpolation algorithm (such as cv2.resize(image,(640,640), interpolation=cv2.INTER_LINEAR) in OpenCV) to perform geometric transformation and generate a new 640×640 image.

[0094] Step 4: Aspect Ratio Adaptive Filling. For image blocks with aspect ratios that differ significantly from the target size (such as extremely narrow strips), direct scaling may result in severe distortion. A better strategy is to maintain the original aspect ratio while scaling, ensuring the longer side matches the target size (e.g., 640 pixels). Then, fill the shorter sides (e.g., by filling both sides with grayscale values) to achieve a final image size of 640×640. This method better preserves the original shape of the defects.

[0095] The embodiments of this application scale the cropped images at multiple scales to the size required by the model, thereby satisfying the requirements for model training.

[0096] Based on the above embodiments, after image augmentation according to the above embodiments, the obtained enhanced images can be used as training samples to train the industrial defect detection model.

[0097] Figure 3 This is a schematic diagram of another image enhancement method provided in an embodiment of this application, as shown below. Figure 3 As shown, the method includes: Step 301: Image acquisition; acquire the original image obtained by image acquisition of the industrial product, which is the original resolution image.

[0098] Step 302: Cut into image blocks of fixed size; divide the original image into multiple image blocks according to the fixed size.

[0099] Step 303: Image block defect annotation; perform defect annotation on the image blocks, which can be done manually to identify target defects.

[0100] Step 304: Crop image blocks with different aspect ratios; Crop according to different aspect ratios.

[0101] Step 305: Scale to a fixed size; after obtaining image patches of multiple sizes, scale them down to the size required for the model.

[0102] Step 306: Merge annotation boxes; merge adjacent annotation boxes belonging to the same defect to obtain multiple images containing defect annotation boxes.

[0103] Step 307: Model training; Input the images obtained in steps 305 and 306 into the model for training.

[0104] It should be noted that the specific implementation methods of each of the above steps can be found in the above embodiments, and the embodiments of this application will not be repeated here.

[0105] Figure 4 This is a schematic diagram of an image enhancement device provided in an embodiment of this application. The device can be a module, program segment, or code on an electronic device. It should be understood that this device is similar to the one described above. Figure 1 The method implementation corresponds to this and can be executed. Figure 1 The specific functions of the device involved in each step of the method embodiment can be found in the description above; to avoid repetition, detailed descriptions are omitted here. The device includes: an image segmentation module 401, a defect recognition module 402, a defect annotation module 403, and an annotation box merging module 404, wherein: The image segmentation module 401 is used to segment the original image to obtain multiple image blocks; The defect identification module 402 is used to identify whether there is a target defect in the image block, wherein the ratio of the long side to the wide side of the minimum bounding rectangle corresponding to the target defect is greater than a first preset value; The defect annotation module 403 is used to annotate the target defect on the image block containing the target defect according to the shape and direction of the target defect, and obtain multiple annotation boxes; the ratio of the long side to the wide side of the annotation box is less than a second preset value; the second preset value is less than the first preset value; The annotation box merging module 404 is used to merge adjacent annotation boxes belonging to the same defect to obtain multiple enhanced images containing defect annotation boxes, and the size of the defect annotation boxes contained in each enhanced image is different.

[0106] Based on the above embodiments, the defect labeling module 403 is specifically used for: The image patch is input into the defect recognition model to obtain the defect mask image corresponding to the target defect; The defect mask image is thinned by iteratively removing boundary pixels until its width is only 1 pixel, thereby obtaining the skeleton line of the target defect. The division length is determined based on the preset target aspect ratio and the maximum width of the target defect; Based on the specified division length, starting from the beginning of the skeleton line, a dividing point is set at intervals of the specified length to obtain multiple skeleton segments; Obtain the pixels of the defect mask image corresponding to the skeleton segment, and use PCA to calculate the directional principal components of the pixel segment; wherein, the direction corresponding to the maximum value of the principal component is the defect direction; Calculate the smallest bounding rectangle that is perpendicular to the defect direction and can enclose the pixels of the skeleton segment, and obtain the annotation box corresponding to the skeleton segment.

[0107] Based on the above embodiments, the annotation box merging module 404 is specifically used for: The coordinates of the bounding boxes in all image blocks are converted into global coordinates based on the position of the image blocks in the original image. The distance between the nearest edges of the two bounding boxes is calculated based on the global coordinates of the image patch. If the distance is less than a preset threshold, the defects in the two bounding boxes are determined to be the same defect. Based on the spatial location and morphological continuity of the target defect, the bounding boxes of the same defect are merged to reconstruct images containing defect bounding boxes at multiple scales.

[0108] Based on the above embodiments, the device further includes a micro-defect merging module, used for: Identify micro-defects in the image block whose defect size is smaller than a preset size; The smallest bounding rectangle of the micro-defect whose spatial distance is less than a preset distance is used as the annotation box.

[0109] Based on the above embodiments, the device further includes an enhancement module for: Based on the physical causes of the defects contained in the image patch, the morphological transformation is used to simulate the appearance of the defects under different surface materials and lighting conditions to obtain an enhanced image patch of the defects. Based on the texture features of the defects contained in the image patch, a multi-scale transformation is performed to obtain an enhanced image of the defects; Based on changes in lighting, viewing angle shifts, and surface deformations in the industrial environment, an enhanced image corresponding to the image block containing the defects is generated.

[0110] Based on the above embodiments, the device further includes a cutting module, used for: Obtain the image containing the defect annotation boxes and their corresponding location information in the original image; For each preset cropping size, the cropping matrix information corresponding to the image containing the defect annotation box is calculated based on the position information; the cropping matrix information includes the coordinates of the top left corner of the cropping matrix in the original image; The corresponding cropped image block is cropped from the original image based on the coordinates of the upper left corner and the preset cropping size.

[0111] Based on the above embodiments, the device further includes an image scaling module, used for: The cropped image patch is scaled to the size required by the model.

[0112] Based on the above embodiments, the device further includes a model training module, used for: The enhanced images are used as training samples to train the industrial defect detection model.

[0113] Figure 5 This is a schematic diagram of the physical structure of the electronic device provided in the embodiments of this application, such as... Figure 5 As shown, the electronic device includes: a processor 501, a memory 502, and a bus 503; wherein: The processor 501 and the memory 502 communicate with each other through the bus 503; The processor 501 is used to call program instructions in the memory 502 to execute the methods provided in the above-described method embodiments, including, for example, dividing the original image to obtain multiple image blocks; identifying whether there is a target defect in the image block, wherein the ratio of the long side to the wide side of the minimum bounding rectangle corresponding to the target defect is greater than a first preset value; for the image block containing the target defect, marking the target defect on the image block according to the shape and direction of the target defect to obtain multiple annotation boxes; wherein the ratio of the long side to the wide side of the annotation box is less than a second preset value; the second preset value is less than the first preset value; merging adjacent annotation boxes belonging to the same defect to obtain multiple enhanced images containing defect annotation boxes, wherein the size of the defect annotation boxes contained in each enhanced image is different.

[0114] Processor 501 can be an integrated circuit chip with signal processing capabilities. The processor 501 can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the various methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor can be a microprocessor or any conventional processor.

[0115] The memory 502 may include, but is not limited to, random access memory (RAM), read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), etc.

[0116] This embodiment discloses a computer program product, which includes a computer program stored on a non-transitory computer-readable storage medium. The computer program includes program instructions, and when the program instructions are executed by a computer, the computer can execute the methods provided in the above-described method embodiments, such as: dividing the original image to obtain multiple image blocks; identifying whether there is a target defect in the image blocks, wherein the ratio of the long side to the wide side of the minimum bounding rectangle corresponding to the target defect is greater than a first preset value; for the image blocks containing the target defect, marking the target defect on the image blocks according to the shape and direction of the target defect to obtain multiple annotation boxes; wherein the ratio of the long side to the wide side of the annotation box is less than a second preset value; the second preset value is less than the first preset value; merging adjacent annotation boxes belonging to the same defect to obtain multiple enhanced images containing defect annotation boxes, and the size of the defect annotation boxes contained in each enhanced image is different.

[0117] This embodiment provides a non-transitory computer-readable storage medium storing computer instructions that cause the computer to execute the methods provided in the above-described method embodiments. These instructions include, for example,: dividing the original image to obtain multiple image blocks; identifying whether a target defect exists in the image blocks, wherein the ratio of the long side to the wide side of the minimum bounding rectangle corresponding to the target defect is greater than a first preset value; for the image blocks containing the target defect, annotating the target defect on the image blocks according to the shape and orientation of the target defect to obtain multiple annotation boxes; wherein the ratio of the long side to the wide side of the annotation box is less than a second preset value; the second preset value is less than the first preset value; merging adjacent annotation boxes belonging to the same defect to obtain multiple enhanced images containing defect annotation boxes, wherein the size of the defect annotation boxes contained in each enhanced image is different.

[0118] In the embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. The apparatus embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. Furthermore, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Additionally, the displayed or discussed mutual couplings, direct couplings, or communication connections may be through some communication interfaces; indirect couplings or communication connections between devices or units may be electrical, mechanical, or other forms.

[0119] Furthermore, the units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0120] Furthermore, the functional modules in the various embodiments of this application can be integrated together to form an independent part, or each module can exist independently, or two or more modules can be integrated to form an independent part.

[0121] In this document, relational terms such as first and second are used only to distinguish one entity or operation from another entity or operation, without necessarily requiring or implying any such actual relationship or order between these entities or operations.

[0122] The above description is merely an embodiment of this application and is not intended to limit the scope of protection of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. An image enhancement method, characterized in that, include: The original image is divided into multiple image blocks; Identify whether a target defect exists in the image block, wherein the ratio of the long side to the wide side of the minimum bounding rectangle corresponding to the target defect is greater than a first preset value; For an image block containing the target defect, the target defect on the image block is marked according to the shape and direction of the target defect to obtain multiple annotation boxes; the ratio of the long side to the wide side of the annotation box is less than a second preset value; the second preset value is less than the first preset value; Adjacent bounding boxes belonging to the same defect are merged to obtain multiple enhanced images containing defect bounding boxes, and the size of the defect bounding boxes in each enhanced image is different.

2. The method according to claim 1, characterized in that, The step of annotating the target defect on the image patch according to the shape and orientation of the target defect includes: The image patch is input into the defect recognition model to obtain the defect mask image corresponding to the target defect; The defect mask image is thinned by iteratively removing boundary pixels until its width is only 1 pixel, thereby obtaining the skeleton line of the target defect. The division length is determined based on the preset target aspect ratio and the maximum width of the target defect; Based on the specified division length, starting from the beginning of the skeleton line, a dividing point is set at intervals of the specified length to obtain multiple skeleton segments; Obtain the pixels of the defect mask image corresponding to the skeleton segment, and use PCA to calculate the directional principal components of the pixel segment; wherein, the direction corresponding to the maximum value of the principal component is the defect direction; Calculate the smallest bounding rectangle that is perpendicular to the defect direction and can enclose the pixels of the skeleton segment, and obtain the annotation box corresponding to the skeleton segment.

3. The method according to claim 1, characterized in that, The step of merging adjacent bounding boxes belonging to the same defect to obtain multiple images containing defect bounding boxes includes: The coordinates of the bounding boxes in all image blocks are converted into global coordinates based on the position of the image blocks in the original image. The distance between the nearest edges of the two bounding boxes is calculated based on the global coordinates of the image patch. If the distance is less than a preset threshold, the defects in the two bounding boxes are determined to be the same defect. Based on the spatial location and morphological continuity of the target defect, the bounding boxes of the same defect are merged to reconstruct images containing defect bounding boxes at multiple scales.

4. The method according to claim 1, characterized in that, The method further includes: Identify micro-defects in the image block whose defect size is smaller than a preset size; The smallest bounding rectangle of the micro-defect whose spatial distance is less than a preset distance is used as the annotation box.

5. The method according to claim 1, characterized in that, The method further includes: Based on the physical causes of the defects contained in the image patch, the morphological transformation is used to simulate the appearance of the defects under different surface materials and lighting conditions to obtain an enhanced image patch of the defects. Based on the texture features of the defects contained in the image patch, a multi-scale transformation is performed to obtain an enhanced image of the defects; Based on changes in lighting, viewing angle shifts, and surface deformations in the industrial environment, an enhanced image corresponding to the image block containing the target defect is generated.

6. The method according to claim 1, characterized in that, After obtaining multiple images containing defect annotation boxes, the method further includes: Obtain the image containing the defect annotation boxes and their corresponding location information in the original image; For each preset cropping size, the cropping matrix information corresponding to the image containing the defect annotation box is calculated based on the position information; the cropping matrix information includes the coordinates of the top left corner of the cropping matrix in the original image; The corresponding cropped image block is cropped from the original image based on the coordinates of the upper left corner and the preset cropping size.

7. The method according to claim 6, characterized in that, The method further includes: The cropped image patch is scaled to the size required by the model.

8. The method according to any one of claims 1-7, characterized in that, The method further includes: The enhanced images are used as training samples to train the industrial defect detection model.

9. An image enhancement device, characterized in that, include: The image segmentation module is used to divide the original image into multiple image blocks; The defect identification module is used to identify whether there is a target defect in the image block, wherein the ratio of the long side to the wide side of the minimum bounding rectangle corresponding to the target defect is greater than a first preset value; The defect annotation module is used to annotate the target defect on the image block containing the target defect according to the shape and direction of the target defect, and obtain multiple annotation boxes; the ratio of the long side to the wide side of the annotation box is less than a second preset value; the second preset value is less than the first preset value; The annotation box merging module is used to merge adjacent annotation boxes belonging to the same defect to obtain multiple enhanced images containing defect annotation boxes, and the size of the defect annotation boxes contained in each enhanced image is different.

10. An electronic device, characterized in that, include: Processor, memory, and bus, among which: The processor and the memory communicate with each other via the bus; The memory stores program instructions that can be executed by the processor, and the processor can execute the method as described in any one of claims 1-8 by calling the program instructions.

11. A non-transitory computer-readable storage medium, characterized in that, The non-transitory computer-readable storage medium stores computer instructions that, when executed by a computer, cause the computer to perform the method as described in any one of claims 1-8.

12. A computer program product, characterized in that, It includes computer program instructions, which, when read and executed by a processor, perform the method as described in any one of claims 1-8.

Images