Image data enhancement processing method and system for defect inspection

Abstract

This invention provides an image data enhancement processing method and system combined with defect inspection, relating to the fields of image processing and defect inspection technology. First, an initial inspection image set corresponding to the defect inspection task is obtained, where each inspection image unit carries acquisition time and spatial location information. Next, multi-level differential features are constructed from the initial inspection image set to generate a hierarchical differential feature map set. Based on this set, defect regions are segmented to obtain a set of binary masks for defect candidate regions. Finally, the binary mask set and the initial inspection image units are combined through defect morphology transfer synthesis to generate an enhanced inspection image set containing multiple defect types. This effectively solves the problem of scarce defect samples and improves the performance of the defect detection model.

Description

Technical Field

[0001] This invention relates to the field of image processing and defect inspection technology, and more specifically, to an image data enhancement processing method and system that combines defect inspection. Background Technology

[0002] In numerous fields such as industrial production and infrastructure maintenance, defect inspection is a crucial step in ensuring the quality and safety of equipment or products. Currently, image-based defect inspection methods are widely used, the core of which lies in identifying defects through the analysis of inspection images. However, in practical applications, the scarcity of defect samples poses a significant challenge to training defect detection models.

[0003] Traditional image data augmentation methods, such as simple geometric transformations (rotation, flipping, etc.) and color adjustments (brightness, contrast changes, etc.), can increase the diversity of image data to some extent, but these methods do not specifically consider the characteristics of defects. They only operate on the image as a whole and cannot highlight the morphological and textural information of defects, thus having limited effect on improving the ability of defect detection models to identify specific defect types. Especially when faced with some rare or complex defect morphologies, the augmented images generated by traditional methods often cannot effectively simulate real defect scenarios, resulting in insufficient generalization ability of the model when facing actual defects, and easy occurrences of missed or false detections. Summary of the Invention

[0004] In view of the aforementioned problems, and in conjunction with the first aspect of the present invention, embodiments of the present invention provide an image data enhancement processing method combined with defect inspection, the method comprising:

[0005] Acquire the initial inspection image set corresponding to the defect inspection task, wherein the initial inspection image set contains inspection image units with acquisition time information and spatial location information;

[0006] The initial inspection image set is subjected to multi-level differential feature construction processing. The inspection image units are decomposed layer by layer using the pixel response differences corresponding to different defect morphologies to generate a hierarchical differential feature map set containing defect morphology sensitive features and background suppression features.

[0007] Based on the hierarchical differential feature map set, defect region segmentation processing is performed. The boundary of potential defect region is determined according to the response distribution of defect morphology sensitive features in the hierarchical differential feature map, and a binary mask set of defect candidate regions is generated.

[0008] The binary mask set of the defect candidate region is combined with the inspection image unit of the initial inspection image set to perform defect morphology migration and synthesis processing. While keeping the background of the inspection image unit unchanged, the texture distribution of the defect candidate region is replaced with the defect texture pattern in the preset defect morphology library to generate an enhanced inspection image set containing multiple defect types.

[0009] Furthermore, embodiments of the present invention also provide an image data enhancement processing system combined with defect inspection, characterized in that it includes:

[0010] A processor; a machine-readable storage medium for storing machine-executable instructions of the processor; wherein the processor is configured to perform the above-described image data enhancement processing method incorporating defect inspection by executing the machine-executable instructions.

[0011] In another aspect, embodiments of the present invention also provide a computer program product, the computer program product including machine-executable instructions, the machine-executable instructions being stored in a computer-readable storage medium, a processor of a computer device reading the machine-executable instructions from the computer-readable storage medium, the processor executing the machine-executable instructions, causing the computer device to perform the above-described image data enhancement processing method combined with defect inspection.

[0012] Based on the above, by acquiring an initial inspection image set containing acquisition time sequence information and spatial location information, multi-level differential feature construction processing is performed on the initial inspection image set. By decomposing the inspection image units layer by layer using the pixel response differences corresponding to different defect morphologies, a hierarchical differential feature map set containing defect morphology sensitive features and background suppression features is generated. Key features of defects are extracted and background interference is suppressed. Defect region segmentation processing is performed based on the hierarchical differential feature map set. The boundary of the potential defect region is determined according to the response distribution of defect morphology sensitive features, and a binary mask set of defect candidate regions is generated, realizing accurate positioning of the defect region. Finally, the binary mask set of defect candidate regions and the inspection image units of the initial inspection image set are subjected to defect morphology transfer synthesis processing. Under the premise of keeping the background unchanged, the texture distribution of the defect candidate region is replaced with the defect texture pattern in the preset defect morphology library, generating an enhanced inspection image set containing multiple defect types. This effectively increases the diversity of defect samples, improves the defect detection model's ability to identify and generalize different defect types, and thus improves the accuracy and reliability of the entire defect inspection system. Attached Figure Description

[0013] Figure 1 This is a schematic diagram of the execution flow of the image data enhancement processing method combined with defect inspection provided in an embodiment of the present invention.

[0014] Figure 2This is a schematic diagram of exemplary hardware and software components of the image data enhancement processing system combined with defect inspection provided in an embodiment of the present invention. Detailed Implementation

[0015] Figure 1 This is a flowchart illustrating an image data enhancement processing method combined with defect inspection provided in one embodiment of the present invention, which will be described in detail below.

[0016] The image data enhancement processing method combined with defect inspection provided in this embodiment can be applied to image inspection scenarios for various infrastructures, industrial facilities, or natural resources. For example, this embodiment uses drone image inspection of a mining area as a unified application scenario. In this scenario, a drone equipped with a visible light camera flies along a preset route, periodically acquiring images of key areas such as mine slopes, mining steps, spoil heaps, and transportation roads, obtaining massive amounts of inspection image data. Because real mine defects (such as slope cracks, rock corrosion, step deformation, and protective netting detachment) are scarce and diverse in form, directly using the acquired raw images to train a defect inspection recognition model is difficult to achieve ideal recognition accuracy and generalization ability. Therefore, the method in this embodiment performs a series of data enhancement processes on the raw inspection images to generate enhanced inspection images containing diverse defect forms and with high background fidelity, which are used to expand the training sample set of the defect inspection recognition model. The above data collection process strictly follows relevant privacy protection and data security regulations. The images collected by the drones are all physical images of the mine surface and facilities, and do not involve the collection of personal information. The transmission and storage of image data are encrypted and access permissions are controlled to ensure the legality and compliance of data use.

[0017] Step S110: Obtain the initial inspection image set corresponding to the defect inspection task. The initial inspection image set contains inspection image units with acquisition timing information and spatial location information.

[0018] In this embodiment, the visible light image data collected by the UAV after each mine inspection flight is aggregated. Each frame of image constitutes an inspection image unit. The initial inspection image set consists of image units accumulated from multiple consecutive inspection missions. Each inspection image unit is bound to acquisition timing information and spatial location information. The acquisition timing information records the acquisition timestamp of the image unit, accurate to the second, and stored in Coordinated Universal Time (UTC) format. The spatial location information records the three-dimensional coordinates of the camera optical center and the azimuth and pitch angles of the camera optical axis when the image unit was acquired. The three-dimensional coordinates of the camera optical center are obtained through a real-time dynamic differential positioning device onboard the UAV, with the coordinate system being the geodetic coordinate system. The azimuth and pitch angles of the camera optical axis are obtained through an inertial measurement unit onboard the UAV, with the azimuth angle based on true north and the pitch angle based on the horizontal plane. For each batch of newly acquired image data, before aggregating it into the initial inspection image set, the acquisition authorization mark is verified to confirm that the UAV flight plan filing number corresponding to each frame of image is valid and that the acquisition period is within the time window of the filing permit. This ensures that all data comes from legal acquisition of authorized flights and complies with the regulations for geographic information data acquisition management. After completing the authorization verification, each inspection image unit, along with its acquisition time sequence information and spatial location information, is stored in the initial inspection image set.

[0019] Step S120: Perform multi-level differential feature construction processing on the initial inspection image set, and decompose the inspection image units layer by layer using the pixel response differences corresponding to different defect morphologies to generate a hierarchical differential feature map set containing defect morphology sensitive features and background suppression features.

[0020] In this embodiment, for each inspection image unit in the initial inspection image set, instead of directly performing global feature extraction, a multi-level differential feature construction process is used to amplify the pixel response differences of different defect morphologies while suppressing the interference of regular textures in the mine background. This processing is based on the following technical principle: defects such as slope cracks, rock corrosion, and step deformation in mine inspection images manifest as abrupt changes in local grayscale or color distribution at the pixel scale, and the abrupt change patterns of different defect types are different; while background areas such as normal mine slopes and step planes exhibit relatively regular spatial self-similar textures. By calculating the difference between pixels and their surrounding environment under a multi-scale neighborhood structure and matching it with a preset defect morphology pixel response template, the features of the above-mentioned differential responses can be extracted to form a hierarchical differential feature map set.

[0021] Step S121: Construct a multi-scale pixel neighborhood structure for each inspection image unit in the initial inspection image set. The multi-scale pixel neighborhood structure takes each pixel in the inspection image unit as the center and establishes a multi-layer nested annular sampling region according to a preset neighborhood expansion coefficient. The pixel sampling point distribution density of each annular sampling region is different.

[0022] In this embodiment, for any inspected image unit, it is first converted from the red-green-blue color space to a grayscale image space to obtain a single-channel grayscale image. Then, each pixel in the grayscale image is traversed, and a multi-scale pixel neighborhood structure is constructed for each pixel. Centered on the current pixel, multiple levels of annular sampling regions are established according to a preset set of neighborhood expansion coefficients. The set of neighborhood expansion coefficients defines the radius and number of sampling points of each level of annular sampling region. For example, the neighborhood expansion coefficients increase layer by layer from the inner layer to the outer layer, so that the inner annular sampling region is close to the center pixel and the sampling points are more densely distributed, which is used to capture subtle local pixel changes; the outer annular sampling region has a larger radius and the sampling points are relatively sparsely distributed, which is used to capture a larger range of pixel change trends. The pixel sampling points of each annular sampling region are distributed at equal angular intervals along the ring or at equal pixel intervals along the square ring. The sampling point coordinates are obtained by bilinear interpolation to obtain sub-pixel level pixel values, ensuring the continuity and accuracy of sampling. Thus, each pixel corresponds to a multi-scale pixel neighborhood structure composed of multiple layers of annular sampling regions.

[0023] Step S122: Perform pixel value encoding processing on the multi-layer nested annular sampling regions respectively, calculate the pixel value deviation between the pixel sampling point and the center pixel point in each annular sampling region, and generate pixel deviation response vectors in neighborhoods of different scales. The dimension of the pixel deviation response vector corresponds to the number of pixel sampling points in the annular sampling region of that layer.

[0024] In this embodiment, pixel value encoding is performed layer by layer for the multi-scale pixel neighborhood structure constructed for each pixel. For a certain layer of annular sampling region, the pixel values ​​of all pixel sampling points in that layer are obtained, and the pixel value of each pixel sampling point is subtracted from the pixel value of the center pixel to obtain the pixel value deviation corresponding to that sampling point. All pixel sampling points in that layer are traversed, and all pixel value deviations are arranged in the spatial order of the sampling points to form a pixel deviation response vector. The dimension of this pixel deviation response vector is consistent with the number of pixel sampling points in the annular sampling region of that layer, and the index position of each element in the vector implies the orientation information of the sampling point relative to the center pixel. For multi-layer annular sampling regions, the above pixel value encoding process is repeated layer by layer to generate a set of pixel deviation response vectors under different scale neighborhoods. The pixel deviation response vector under each scale neighborhood represents the difference pattern between the center pixel and its local environment at that observation scale. The intensity and distribution of this difference pattern are the key basis for subsequently distinguishing defective pixels from background pixels. For gray-scale gradient defects (such as the edge of an eroded area), the inner pixel deviation response vector exhibits a smooth gradient characteristic; for gray-scale abrupt change defects (such as the edge of a crack), the inner pixel deviation response vector exhibits a local sharp jump characteristic.

[0025] Step S123: Based on the preset defect morphology pixel response template library, select crack-type defect response template, corrosion-type defect response template, and deformation-type defect response template from the defect morphology pixel response template library, and perform layer-by-layer convolution matching processing on the pixel deviation response vectors of different scale neighborhoods with each defect response template to generate defect morphology matching response intensity at each scale.

[0026] In this embodiment, a preset defect morphology pixel response template library stores ideal response templates corresponding to various typical defect morphologies in the pixel deviation response vector space. For the mine inspection scenario, crack-type defect response templates, corrosion-type defect response templates, and deformation-type defect response templates are selected from this template library. The crack-type defect response template characterizes the directionality and sharpness of linear fracture structures on the pixel deviation response vector; that is, the pixel deviation value along the crack direction is small, while the pixel deviation value perpendicular to the crack direction is large. The corrosion-type defect response template characterizes the dispersion characteristics of patchy structures formed by surface material loss on the pixel deviation response vector; that is, the pixel deviation value is relatively uniformly distributed in all directions but with moderate amplitude. The deformation-type defect response template characterizes the curvature response characteristics of the arc-shaped gradient changes formed by plastic bending of the structural surface on the pixel deviation response vector; that is, the pixel deviation value exhibits a gradual bending pattern with changes in azimuth angle. The pixel deviation response vectors at different scales are convolved with each defect response template for matching. This convolution matching process calculates the normalized value of the dot product between the pixel deviation response vector and the defect response template. The larger the normalized value of the dot product, the better the local difference pattern of the pixel matches the defect morphology template. By traversing all scales and all combinations of defect response templates, the defect morphology matching response intensity at each scale is generated. Each scale corresponds to three response intensity values: crack, corrosion, and deformation.

[0027] Step S124: Perform inter-scale feature fusion processing on the defect morphology matching response intensity at each scale through cross-scale response aggregation operation, and cascade the defect morphology matching response intensity of adjacent scales along the scale dimension to generate a multi-scale defect morphology sensitive feature vector. Each element in the multi-scale defect morphology sensitive feature vector corresponds to the matching degree of a defect morphology response template at a certain scale.

[0028] In this embodiment, after obtaining the defect morphology matching response intensity at each independent scale, information from different observation scales is integrated through cross-scale response aggregation. For the current pixel, the crack-type defect morphology matching response intensity, corrosion-type defect morphology matching response intensity, and deformation-type defect morphology matching response intensity at each scale in the scale sequence are arranged in ascending order of scale. The arrangement method is as follows: first, the three response intensity values ​​at the same scale are combined into a triplet in a fixed order of crack, corrosion, and deformation; then, the triplets of different scales are cascaded along the scale dimension. For example, if there are K scales, the cascaded multi-scale defect morphology sensitive feature vector contains 3 times K elements, each element clearly corresponding to the matching degree of a defect morphology response template at a specific scale. This multi-scale defect morphology sensitive feature vector integrates defect response information from local details and contextual scope, enabling both fine crack recognition and macroscopic corrosion area identification capabilities.

[0029] Step S125: Perform background suppression filtering on the multi-scale defect morphology sensitive feature vector, calculate the contribution uniformity of each element in the multi-scale defect morphology sensitive feature vector within the spatial neighborhood, extract feature elements with contribution uniformity lower than a preset differentiation threshold as defect morphology sensitive features, and perform zeroing processing on feature elements with contribution uniformity not lower than the preset differentiation threshold to obtain background suppression features.

[0030] In this embodiment, to eliminate the interference of regular background textures (such as rolling marks, uniform turf, and textured slope protection) in mine inspection images on subsequent defect segmentation, background suppression filtering is performed on the multi-scale defect morphology sensitive feature vector. For each element in the multi-scale defect morphology sensitive feature vector, a local spatial neighborhood window is defined with the current pixel as the center. The response values ​​of all pixels within this window at the same feature element position are counted, and the contribution uniformity of these response values ​​is calculated. The contribution uniformity is calculated by dividing the mean of the response values ​​of the element within the local spatial neighborhood window by its standard deviation. The quotient is used as the contribution uniformity measure. The smaller the mean and the larger the standard deviation, the more uneven the response distribution of the feature element at the current spatial position, the more localized the differences, and the more likely it is to correspond to the real defect structure. A differentiation threshold is set. Feature elements with a contribution uniformity lower than the differentiation threshold are retained and marked as defect morphology sensitive features. Feature elements with a contribution uniformity not lower than the differentiation threshold are zeroed out so that they no longer participate in subsequent feature representation. After this processing, the low-differentiation features that were originally present in the background region in the multi-scale defect morphology sensitive feature vector are filtered out, and only the feature elements that show significant differences in the local space are retained, forming background suppression features.

[0031] Step S126: Stack and recombine the defect morphology sensitive features and the background suppression features element by element according to their original spatial positions to generate differential feature layers of different levels. Each differential feature layer corresponds to a matching response spatial distribution of a defect morphology response template.

[0032] In this embodiment, after background suppression filtering, the defect morphology-sensitive features and background-suppressed features are stacked and recombined element by element according to their original spatial positions. Specifically, for each pixel, all retained elements and all zeroed elements in the feature vector after background suppression filtering are grouped according to their original template type and scale attributes. Feature elements of all scales belonging to the same defect morphology response template type (e.g., cracks) are stacked into a differential feature layer; feature elements of all scales belonging to another defect morphology response template type (e.g., corrosion) are stacked into another differential feature layer, and so on. Each differential feature layer is a two-dimensional spatial distribution map, where the rows and columns correspond to the row and column coordinates of the original inspection image unit, respectively. The number of channels in the layer equals the number of corresponding scales, and each channel stores the feature response value after background suppression filtering at that scale. This stacking and recombining process maintains a one-to-one correspondence between feature elements and their original pixel spatial positions, without any spatial downsampling or translation operations, enabling subsequent defect region boundary localization to be based on pixel-level accuracy. Finally, each inspection image unit generates multiple differential feature layers corresponding to the morphological response templates of crack, corrosion, and deformation defects.

[0033] Step S127: Based on the feature complementarity relationship between the differential feature layers corresponding to different defect morphology response templates within the same inspection image unit, calculate the feature mutual information metric between each differential feature layer, and compile the differential feature layers with feature mutual information metrics higher than the preset complementarity threshold into the same hierarchical differential feature map group.

[0034] In this embodiment, there may be complementary features between the differential feature layers corresponding to different defect morphology response templates. That is, two differential feature layers may produce high responses in adjacent parts of the same defect region, and their combined use can more completely characterize the defect morphology. To capture the above complementary relationship, the feature mutual information metric is calculated between each pair of differential feature layers. Specifically, for any two differential feature layers, their feature response values ​​at all pixel positions are extracted to form two response value sequences. The numerical range of each response value sequence is divided into several equidistant intervals, and the joint probability distribution and the individual edge probability distribution of the two response value sequences in each interval combination are statistically analyzed. The feature mutual information metric between the two differential feature layers is calculated using the joint probability distribution and the edge probability distribution. The larger the metric, the more information the two differential feature layers share and the stronger the feature complementarity. A complementarity threshold is set. When the feature mutual information metric between the two differential feature layers is higher than the complementarity threshold, the two differential feature layers are incorporated into the same hierarchical differential feature map group. If the mutual information metric between a certain difference feature layer and all other difference feature layers is lower than the complementarity threshold, then that difference feature layer forms a hierarchical difference feature map group on its own.

[0035] Step S128: Perform layer fusion processing on the differential feature layers within the same hierarchical differential feature map group to generate a fused differential feature map that characterizes the comprehensive response intensity of the corresponding defect morphology. Combine the fused differential feature maps generated by different hierarchical differential feature map groups along the hierarchical dimension to obtain a set of hierarchical differential feature maps.

[0036] In this embodiment, the layer fusion processing performs pixel-wise maximum pooling along the channel dimension on multiple differential feature layers within each hierarchical differential feature map group to generate a fused differential feature map. The advantage of this pixel-wise maximum pooling operation is that for each spatial pixel location, if any differential feature layer within the group exhibits a high response at that location, this high response will be retained in the fused differential feature map, thereby achieving integrated representation of complementary features. The size of the fused differential feature map is consistent with the spatial size of the original inspection image unit, with one channel. Each pixel value represents the comprehensive response intensity of the defect morphology type corresponding to that hierarchical differential feature map group at that spatial location. The fused differential feature maps generated from different hierarchical differential feature map groups are combined along the hierarchical dimension, i.e., arranged into a sequence according to the generation order, to obtain a hierarchical differential feature map set. Each layer in this set is a fused differential feature map, and different layers correspond to different defect morphology types or their combinations.

[0037] Step S130: Perform defect region segmentation processing based on the hierarchical differential feature map set, determine the boundary of the potential defect region according to the response distribution of the defect morphology sensitive features in the hierarchical differential feature map, and generate a binary mask set of defect candidate regions.

[0038] In this embodiment, the hierarchical differential feature map set has effectively separated the defect morphology-sensitive features from the background suppression features, and different defect types are distributed in different feature layers. Next, based on these feature layers, precise spatial boundary division of the defect region is needed to segment continuous high-response regions from the background, forming a binary mask to provide accurate region localization information for subsequent defect texture transfer. The core challenge of defect region segmentation is that the boundaries of high-response regions in the fused differential feature maps are often blurred and gradual, requiring adaptive region growing and boundary optimization methods to strike a balance between avoiding undersegmentation and oversegmentation.

[0039] Step S131: Perform feature response extreme point search processing on each fused differential feature map in the hierarchical differential feature map set, locate the coordinates of all local maxima points in the spatial dimension of the fused differential feature map, and obtain the candidate defect core point coordinate set. Each coordinate point in the candidate defect core point coordinate set is accompanied by its corresponding defect morphology comprehensive response intensity value.

[0040] In this embodiment, for each fused differential feature map in the hierarchical differential feature map set, a feature response extremum point search process is first performed. This search process employs a two-dimensional non-maximum suppression method, performing dual comparisons in both the row and column directions of the fused differential feature map. For each pixel in the fused differential feature map, its comprehensive response intensity value is compared with the comprehensive response intensity values ​​of all pixels within its local neighborhood window. If the comprehensive response intensity value of the pixel is greater than the comprehensive response intensity values ​​of all other pixels within the local neighborhood window, the pixel is determined as a local maximum point, and its two-dimensional coordinates and the corresponding defect morphology comprehensive response intensity value are recorded. The size of the local neighborhood window is set according to the defect type feature scale corresponding to the level to which the fused differential feature map belongs: for the level corresponding to crack-type defects, the local neighborhood window is smaller to capture the fine orientation of linear cracks; for the level corresponding to corrosion-type defects, the local neighborhood window is larger to accommodate the aggregation characteristics of sheet corrosion. All fused differential feature maps are traversed, and all located local maximum points, their coordinates, and response intensity values ​​are summarized to form a set of candidate defect core point coordinates.

[0041] Step S132: Using each candidate defect core point in the set of candidate defect core point coordinates as a region growth seed point, perform a continuous neighborhood pixel traversal search along eight spatial directions in the fused differential feature map to determine the maximum extension distance of each growth seed point where the comprehensive response intensity value of the defect morphology along each spatial direction is continuously higher than the average response value of the adjacent pixels.

[0042] In this embodiment, each candidate defect core point in the set of candidate defect core point coordinates is used as a seed point for region growth, and region growth is performed in the corresponding fused differential feature map. The direction of region growth is set to eight spatial directions: horizontal to the right, horizontal to the left, vertical upward, vertical downward, upper right diagonal, upper left diagonal, lower right diagonal, and lower left diagonal. For each growth seed point, adjacent pixels are traversed outward in each spatial direction. In each traversal step, the difference between the response intensity value of the previous pixel and the response intensity value of the next pixel is calculated and compared with a preset response attenuation tolerance. If the decrease in the response intensity value of the current pixel relative to the response intensity value of the previous pixel is less than the response attenuation tolerance, growth continues in that direction; if the decrease exceeds the response attenuation tolerance or the traversal reaches the image boundary, growth in that direction is stopped, and the number of pixel steps traversed from the growth seed point to the stop point along that direction is recorded as the maximum extension distance. The response attenuation tolerance is dynamically determined based on the overall response intensity distribution of the fused differential feature map, taking a fixed multiple of the standard deviation of the response intensity values ​​of all pixels in the fused differential feature map. Ultimately, each growth seed point obtains a maximum extension distance in each of the eight spatial directions.

[0043] Step S133: Take the maximum extension distance of each growth seed point in eight spatial directions as the growth boundary distance parameter in that spatial direction, and perform closed connection processing on the endpoints of the eight growth boundary distance parameters through the convex hull connection algorithm to generate the initial defect region polygon boundary corresponding to the candidate defect core point.

[0044] In this embodiment, for each growth seed point, the maximum extension distance obtained in eight spatial directions is converted into the coordinates of eight endpoints in a polar coordinate system with the growth seed point as the origin. The polar angles corresponding to the eight spatial directions are 0 radians, π / 4 radians, π / 2 radians, 3π / 4 radians, π radians, 5π / 4 radians, 3π / 2 radians, and 7π / 4 radians, respectively, and the polar radii are the maximum extension distances in each direction. These eight endpoints are converted from the polar coordinate system back to the image pixel coordinate system to obtain eight pixel coordinate points. Then, the convex hull connection algorithm is called, using these eight pixel coordinate points as the input point set, to calculate the minimum convex polygon of the point set. All vertices of this minimum convex polygon are arranged in counterclockwise order to form the initial defect region polygon boundary corresponding to the candidate defect core point. This initial defect region polygon boundary is a preliminary estimate of the defect region range. Its shape is a convex polygon, which can quickly outline the approximate range of the defect with fewer vertices, but may have the problem of over-inclusion of concave defect regions.

[0045] Step S134: Extract the initial defect region polygon boundaries corresponding to all candidate defect core points, perform region merging judgment processing on the initial defect region polygon boundaries that overlap in spatial location, calculate the ratio of the overlapping area to the area of ​​each region, and merge regions with an overlap ratio higher than the fusion threshold into a unified defect region polygon boundary.

[0046] In this embodiment, since multiple candidate defect core points may exist within the same defect region, the polygonal boundaries of different initial defect regions may overlap spatially. First, all initial defect region polygonal boundaries are traversed, and the overlap area between any two initial defect region polygonal boundaries is calculated. The overlap area is calculated using a polygon clipping algorithm to obtain the area of ​​the intersecting region of the two polygons. Then, the ratio of the overlap area to the area of ​​each of the two initial defect regions is calculated. A fusion threshold is set; if the ratio of the overlap area to the area of ​​any one of the initial defect regions is higher than the fusion threshold, the two initial defect regions are determined to belong to the same defect entity and are merged. The merging operation gathers the vertex coordinates of all the polygonal boundaries of the two initial defect regions, recalculates the minimum convex polygon or circumscribed polygon of the merged point set, and generates a unified defect region polygonal boundary. The merged unified defect region polygonal boundary participates in subsequent further merging judgments until all pairs of regions meeting the merging conditions have been merged.

[0047] Step S135: Extract the vertex coordinate sequence of the boundary polygon on the polygon boundary of the unified defect area, perform curvature analysis on the vertex coordinate sequence of the boundary polygon, and identify the vertex with abrupt change in boundary curvature as the key point for boundary shape adjustment. The vertex with abrupt change in boundary curvature is the vertex whose boundary curvature value is higher than the average curvature value of the adjacent vertices.

[0048] In this embodiment, although the unified defect region polygon boundary has completed region merging, its boundary is generated by the convex hull algorithm, which may be too smooth and fail to conform to the actual concave or convex shape of the defect. Therefore, the boundary needs to be refined. First, the vertex coordinate sequence of the unified defect region polygon boundary is extracted and arranged in counterclockwise order to form an ordered vertex list. For each vertex, the boundary curvature value at that vertex is calculated using its preceding and following vertices. The boundary curvature value is calculated as follows: take the current vertex and its preceding and following vertices to form a local triangle, and calculate the reciprocal of the radius of the circumcircle of the triangle as the approximate boundary curvature value at that vertex. After traversing the vertex sequence and calculating the boundary curvature value of each vertex, a curvature mutation threshold is set. This threshold is the mean of the boundary curvature values ​​of all vertices plus a fixed multiple of the standard deviation. Vertices with boundary curvature values ​​higher than this curvature mutation threshold are identified as boundary curvature mutation vertices. These vertices are usually located at the inflection points where the defect boundary undergoes a sharp change in direction, and are key control points for fine adjustment of the boundary shape.

[0049] Step S136: Perform boundary smoothing redrawing between adjacent boundary curvature abrupt change vertices. Use the adaptive distance weighting method to resample the original boundary line segments between adjacent boundary curvature abrupt change vertices to generate smoothly transitioned replacement boundary curve segments. Patch the replacement boundary curve segments with the unadjusted boundary line segments to form the adjusted precise boundary of the defect area.

[0050] In this embodiment, adjacent boundary curvature abrupt change vertices are used as the start and end anchor points of the boundary segment, and the original boundary line segment between these two anchor points is smoothed and redrawn. This smoothing and redrawing process does not directly retain the polyline vertices on the original boundary line segment, but rather performs adaptive resampling based on the distance between adjacent anchor points and the distribution of the comprehensive response intensity of the local defect morphology. First, several resampling candidate points are inserted equidistantly along the straight line between the two anchor points. Then, for each resampling candidate point, the pixel position with the highest comprehensive response intensity value in the fused differential feature map on the vertical line is used as the new boundary point position, replacing the resampling candidate point. A series of new boundary points after the above adjustment are connected with a smooth curve to generate a replacement boundary curve segment. The replacement boundary curve segment maintains first-order continuity with the adjacent curve segments at the start and end anchor points. All replacement boundary curve segments are sequentially spliced ​​with the boundary line segments that were not adjusted (i.e., line segments that do not contain boundary curvature abrupt change vertices) along the boundary direction to form the adjusted precise boundary of the defect region.

[0051] Step S137: Perform region filling processing on the interior of the precise boundary of the defect region, set all pixels located inside the precise boundary of the defect region as foreground marker values, and set pixels located outside the precise boundary of the defect region as background marker values, thereby generating a binary mask unit. The foreground region of the binary mask unit corresponds to the potential defect region in the inspection image unit.

[0052] In this embodiment, the precise boundary of the adjusted defect region is a closed two-dimensional polygon. A scan-line filling algorithm is used to fill the interior of this polygon. Scanning from the smallest row coordinate to the largest row coordinate, all intersection points of the scan line with the polygon boundary are calculated for each row. These intersection points are then sorted by column coordinate and paired for filling. All pixels falling inside the polygon are set as foreground marker values ​​in the binary mask unit, with the foreground marker value being logically true; all pixels falling outside the polygon are set as background marker values, with the background marker value being logically false. This generates a binary mask unit with the same spatial resolution as the inspection image unit, containing only foreground and background states. The foreground region of this binary mask unit corresponds to the potential defect region in the inspection image unit.

[0053] Step S138: Traverse all fused differential feature maps in the hierarchical differential feature map set, perform spatial position consistency verification on the binary mask unit generated by each fused differential feature map, compare the consistency of binary labels of different hierarchical fused differential feature maps at the same spatial position, merge the foreground regions with consistent labels, and generate a hierarchical unified defect candidate region binary mask.

[0054] In this embodiment, since the hierarchical differential feature map set contains multiple levels, each level may generate its own binary mask units, and these binary mask units may have labeling differences at the same spatial location. To eliminate labeling conflicts between levels, a spatial location consistency check is performed on the binary mask units generated by each fused differential feature map. The binary mask units of the fused differential feature maps from different levels are compared pixel by pixel. If a spatial location is labeled as foreground in more than half of the level's binary mask units, then that spatial location is labeled as foreground in the final binary mask; otherwise, it is labeled as background. This voting mechanism ensures that the final retained defect candidate regions receive consistent support from multiple levels of features. After consistency check, a binary mask for defect candidate regions with unified hierarchical structure is generated.

[0055] Step S139: Bind the hierarchically unified binary mask of the defect candidate region to the spatial location information of the corresponding inspection image unit in the initial inspection image set to generate a binary mask entry of the defect candidate region carrying spatial location markers. All binary mask entries of the defect candidate region carrying spatial location markers are integrated into a set of binary masks of the defect candidate region.

[0056] In this embodiment, the hierarchically unified binary mask for defect candidate regions only contains spatial marker information and has not yet been bound to its source inspection image unit. The binary mask for defect candidate regions is bound to the spatial location information of the inspection image unit that generated it. Here, the spatial location information is the geographical boundary coordinates or center coordinates and coverage radius of the geographical area covered by the inspection image unit. After binding, a binary mask entry for defect candidate regions is generated. This entry includes a binary mask data matrix, spatial location markers, and a source inspection image unit identifier. All inspection image units in the initial inspection image set are traversed, and steps S131 to S139 are performed for each inspection image unit to generate their corresponding binary mask entries for defect candidate regions. All these entries are then aggregated to form a set of binary masks for defect candidate regions.

[0057] Step S140: Perform defect morphology migration and synthesis processing on the binary mask set of the defect candidate region and the inspection image unit of the initial inspection image set. Under the premise of keeping the background of the inspection image unit unchanged, replace the texture distribution of the defect candidate region with the defect texture mode in the preset defect morphology library to generate an enhanced inspection image set containing multiple defect types.

[0058] In this embodiment, after obtaining a precise set of binary masks for defect candidate regions and the original inspection image units, defect morphology transfer synthesis can be performed. The core objective of this transfer synthesis process is to replace the texture distribution within the defect candidate regions with realistic defect texture patterns while keeping the pixels in the non-defect background regions of the inspection image units completely unchanged, thereby synthesizing a naturally shaped and realistically textured defect region on the original mine background. The enhanced inspection image generated in this way retains the background information of the original image and adds new defect instances, greatly enriching the defect types and manifestations of the training samples.

[0059] Step S141: Read the defect texture pattern sample of the specified defect type from the preset defect morphology library. The preset defect morphology library includes a set of crack texture pattern samples, a set of corrosion texture pattern samples, a set of deformation texture pattern samples, and a set of peeling texture pattern samples. Each defect texture pattern sample has a unique texture pattern identifier.

[0060] In this embodiment, the preset defect morphology library is a pre-constructed database of typical mine defect textures. The crack texture pattern sample set in this database contains various crack texture patches extracted from real mine slope crack images, categorized and stored according to attributes such as crack thickness, extension direction, and number of branches. The corrosion texture pattern sample set contains honeycomb, powdery, and water-stained texture patches formed by weathering and corrosion of rock surfaces. The deformation texture pattern sample set contains surface wrinkles, bulges, and depressions caused by mine pressure or slope sliding. The spalling texture pattern sample set contains texture patches such as damaged protective netting and exposed rebar due to concrete spalling. Each defect texture pattern sample is a fixed-size square texture image block, bound to a unique texture pattern identifier. The identifier encodes the defect type and attribute information. During defect morphology transfer synthesis, the corresponding defect texture pattern sample is read from this library using the texture pattern identifier according to the actual defect type requiring enhancement.

[0061] Step S142: Perform texture feature extraction processing on the defective texture pattern sample. Decompose the defective texture pattern sample into primitives using texture primitive analysis method, extract the texture primitive shape features and texture primitive spatial arrangement rules of the defective texture pattern sample, and generate texture primitive feature description vector and texture spatial distribution rule encoding.

[0062] In this embodiment, after reading the defect texture pattern sample, it is not directly pasted into the defect candidate region. Instead, it is first decomposed and analyzed at the texture primitive level to adapt to defect regions of different shapes and sizes. The texture primitive analysis method considers the defect texture pattern sample as being composed of several visual primitives arranged repeatedly according to certain rules. First, edge segments in the defect texture pattern sample are extracted using an edge detection algorithm. The edge segments are clustered according to connectivity and directional similarity, and each cluster constitutes a texture primitive. The shape features of each texture primitive are extracted, including the aspect ratio of the bounding rectangle, duty cycle, orientation angle, and normalized Fourier descriptor. All shape features are encoded into a texture primitive feature description vector. Then, the arrangement pattern of all texture primitives in the defect texture pattern sample space is analyzed, and a spatial adjacency graph between texture primitives is constructed. Parameters such as repetition period, arrangement direction, and interval distribution are extracted from the graph and encoded into a texture space distribution rule code. This texture primitive feature description vector and the texture space distribution rule code together constitute the transferable texture representation of the defect texture pattern sample.

[0063] Step S143: Perform geometric morphological feature extraction processing on the binary mask entries of the defect candidate region in the binary mask set, calculate the area of ​​the foreground region, the direction angle of the major axis of the region and the perimeter of the region of the binary mask of the defect candidate region, and generate region shape constraint parameters. The region shape constraint parameters are used to guide the placement of texture primitives in the defect candidate region.

[0064] In this embodiment, for each binary mask entry in the defect candidate region binary mask set, the geometric features of its foreground region are extracted. First, the total number of pixels contained in the foreground region is counted through connected component analysis to obtain the region area. Then, the covariance matrix of the pixel coordinates of the foreground region is calculated, and eigenvalue decomposition is performed on this covariance matrix. The direction angle of the eigenvector corresponding to the largest eigenvalue is taken as the major axis direction angle of the region, which reflects the main extension direction of the defect candidate region. Finally, the number of boundary pixels of the foreground region is calculated using a boundary tracking algorithm to obtain the region perimeter. The region area, the major axis direction angle, and the region perimeter are combined to form the region shape constraint parameter. This region shape constraint parameter will play a role in the adaptive deformation processing of texture primitives, ensuring that the implanted texture primitives are consistent with the geometry of the defect candidate region.

[0065] Step S144: Based on the region shape constraint parameters of the foreground region of the binary mask of the defect candidate region and the texture primitive feature description vector, perform adaptive deformation processing of the texture primitive, rotate and adjust the direction of the texture primitive feature description vector to align the direction of the texture primitive with the direction angle of the region's major axis, and perform size scaling adjustment to match the size of the texture primitive with the area of ​​the region, thereby obtaining the adapted texture primitive.

[0066] In this embodiment, adaptive deformation processing of texture primitives completes the geometric adaptation of texture primitives from the source texture sample to the target defect candidate region. For each texture primitive, based on the region's major axis direction angle in the region shape constraint parameters, the angle difference between the current direction angle of the texture primitive and the region's major axis direction angle is calculated. All edge fragment coordinates of the texture primitive are rotated around the primitive center by the angle difference, aligning the texture primitive's direction with the main extension direction of the defect candidate region. Then, a scaling factor is calculated based on the region area, which is proportional to the square root of the region area, ensuring that the relative size of the texture primitive within the target region remains unchanged. All edge fragment coordinates of the texture primitive are scaled proportionally. The texture primitive after orientation rotation and size scaling is the adapted texture primitive, which not only retains the microscopic morphological features of the original defect texture but also coordinates with the target defect candidate region in terms of direction and scale.

[0067] Step S145: Using the Poisson image fusion method, the adapted texture primitives are encoded and implanted into the foreground region marked by the binary mask of the defect candidate region of the corresponding inspection image unit in the initial inspection image set according to the texture space distribution rules. During the implantation process, the background region pixel values ​​marked by the binary mask of the defect candidate region remain unchanged, and the texture implantation only acts on the foreground region pixels, thus obtaining a single-type defect enhanced inspection image unit.

[0068] In this embodiment, texture implantation employs a Poisson image fusion method. This method can implant adapted texture primitives into the target region while maintaining the continuity of pixel gradients at the implantation boundary, eliminating manual stitching traces. First, based on the texture spatial distribution rule encoding, the placement positions of each adapted texture primitive are determined within the foreground region marked by the binary mask of the defect candidate region. The placement positions conform to the repetition period and arrangement defined by the spatial distribution rule encoding. For each placement position, the pixel value of the adapted texture primitive overwrites the original pixel value at that position. Then, using the foreground region boundary as the fusion boundary condition, the pixel values ​​on the fusion boundary are fixed as the original background pixel values. The Poisson equation is solved within the foreground region to ensure that the Laplacian operator of the synthesized pixels within the foreground region is as close as possible to the Laplacian operator of the adapted texture primitive. The resulting foreground region pixel values ​​maintain texture pattern characteristics while naturally transitioning with the background pixel values ​​at the boundary. Throughout the implantation process, the pixel values ​​of the background region marked by the binary mask of the defect candidate region are completely preserved without any modification. After implantation, a single-type defect enhancement inspection image unit is obtained, which contains an enhanced synthesis effect of a specific defect type (crack, corrosion, deformation or peeling).

[0069] Step S146: Perform texture transition smoothing processing on the single-type defect enhancement inspection image unit, extract the boundary contour line between the foreground region and the background region, perform pixel value gradient smoothing optimization along the transition zone with a preset width range on both sides of the boundary contour line, eliminate the texture boundary jump caused by defect texture implantation, and generate a single-type defect enhancement inspection image unit with continuous boundary.

[0070] In this embodiment, although Poisson image fusion can largely guarantee boundary continuity, slight boundary jumps may still exist in some areas with high texture complexity. Therefore, further texture transition smoothing processing is performed. The boundary contour line between the foreground and background regions is extracted, and buffer areas of a preset pixel width are extended outwards and inwards along this contour line to form a transition zone. Within the transition zone, the pixel value of each pixel is weighted according to its distance from the boundary contour line. Pixels closer to the boundary contour line have a more even mixing ratio of background and foreground components in their pixel values; pixels further from the boundary contour line into the background region have their original background pixel values ​​gradually increased to full weight; pixels further from the boundary contour line into the foreground region have their implanted texture pixel values ​​gradually increased to full weight. The mixing process uses a linear or cosine gradient weighting function. After this texture transition smoothing processing, the texture boundary jumps caused by defect texture implantation are eliminated, generating a single-type defect-enhanced inspection image unit with coherent boundaries.

[0071] Step S147: Traverse the texture pattern identifiers of all defect texture pattern samples in the preset defect morphology library, and repeatedly perform texture primitive adaptive deformation processing and Poisson image fusion processing for each texture pattern identifier to generate multiple single-type defect enhanced inspection image unit variants corresponding to different defect texture pattern identifiers for the same inspection image unit.

[0072] In this embodiment, for the same inspection image unit, to maximize its defect type coverage, the texture pattern identifiers of all defect texture pattern samples in the preset defect morphology library are traversed. For each texture pattern identifier, the texture primitive feature extraction in step S142, the texture primitive adaptive deformation processing in step S144, and the Poisson image fusion processing in step S145 are repeatedly executed. The processing of each texture pattern identifier utilizes the texture features and spatial arrangement rules of the defect texture pattern sample corresponding to that identifier to generate synthetic defects with different visual effects within the same defect candidate region. Thus, for the same inspection image unit, multiple single-type defect enhanced inspection image unit variants will be generated, each variant carrying different defect types and texture representations.

[0073] Step S148: Combine the multiple single-type defect enhancement inspection image unit variants generated by the same inspection image unit with the original version of the inspection image unit to form a defect enhancement image group for the inspection image unit. Each image unit in the defect enhancement image group carries a defect texture pattern identifier and a spatial location marker of the corresponding defect candidate region binary mask.

[0074] In this embodiment, all processing results corresponding to the same inspection image unit are summarized. The processing results include the original version of the inspection image unit (without any synthetic defects) and various single-type defect-enhanced inspection image unit variants. These image units are packaged into a defect-enhanced image group. Each image unit in this defect-enhanced image group carries two sets of metadata: the first set is the texture pattern identifier of the synthetic defect contained in the image unit; if it is the original version, the texture pattern identifier is empty; the second set is the spatial location marker of the binary mask of the defect candidate region, indicating the regional range of the synthetic defect in the coordinate system of this image unit. This organizational structure facilitates sample selection and loss function weighting in the subsequent training phase.

[0075] Step S149: Perform defect enhancement image group generation processing on all inspection image units, and collect the defect enhancement image groups corresponding to all inspection image units to generate an enhanced inspection image set containing multiple temporal phases and multiple defect types. The enhanced inspection image set is used to expand the defect type coverage of the training samples of the defect inspection and recognition model.

[0076] In this embodiment, all inspection image units in the initial inspection image set are traversed, and steps S141 to S148 are performed on each inspection image unit to generate its own defect enhancement image group. During processing, the acquisition time sequence information is considered. For inspection image units at the same spatial location but different time phases, their defect candidate regions may change over time (crack propagation, corrosion aggravation). When synthesizing defect morphology, a defect texture pattern sample matching the degree of defect development at that time phase can be selected. After processing all inspection image units, all image units in all defect enhancement image groups are aggregated to construct an enhanced inspection image set containing multiple time phases and multiple defect types. In this enhanced inspection image set, the defect types cover common mine defects such as cracks, corrosion, deformation, and spalling. Furthermore, image units at the same spatial location but different time phases can reflect the dynamic evolution process of defects, effectively expanding the defect type coverage and time span coverage of the training samples of the defect inspection and recognition model.

[0077] Step S210: Perform scene adaptive fusion processing on the enhanced inspection image set, extract the acquisition scene features of the inspection image units in the initial inspection image set, and perform illumination consistency adjustment and texture naturalization processing on the synthetic defect areas in the enhanced inspection image set according to the acquisition scene features.

[0078] In this embodiment, although the enhanced inspection image set has generated enhanced images containing various defect types, the texture of the synthesized defect region is implanted under the illumination conditions of the original inspection image unit. When the inspection image units in the initial inspection image set are acquired at different times and under different weather conditions, the illumination environment varies. Direct merging may cause the synthesized defect to exhibit inconsistent illumination in different images. Therefore, scene adaptive fusion processing is introduced to ensure that the illumination and texture representation of the synthesized defect is consistent with the actual acquisition scene of the image unit.

[0079] Step S211: Extract scene lighting features from the inspection image units of the initial inspection image set. The scene lighting features include the lighting direction vector, the lighting intensity distribution, and the color temperature value.

[0080] In this embodiment, for each inspection image unit in the initial inspection image set, its scene lighting features are extracted. The estimation of the lighting direction vector is based on the brightness variation of obvious geometric structures (such as step edges and rock ridges) in the image. By analyzing the statistical distribution of image gradients in multiple directions, the direction with the largest gradient amplitude is taken as the projection direction of the lighting direction. The lighting direction vector is calculated by combining the known camera optical axis azimuth and pitch angles. The lighting intensity distribution is obtained by statistically analyzing the mean and variance of image pixel values. The image is divided into blocks, and the average pixel value of each block is calculated to form a spatial distribution map of lighting intensity. The color temperature value is estimated by performing color constancy analysis on near-white areas in the image, calculating the ratio of the red, green, and blue channels, and estimating the color temperature of the scene light source. These three features constitute the scene lighting features, characterizing the lighting environment during the acquisition of the inspection image unit.

[0081] Step S212: Perform illumination response modeling on the synthetic defect region marked by the binary mask of the defect candidate region of each enhanced inspection image unit in the enhanced inspection image set. Calculate the pixel response of the synthetic defect region under different illumination conditions based on the illumination direction vector, illumination intensity distribution, and color temperature value, and generate an illumination response prediction map of the synthetic defect region.

[0082] In this embodiment, illumination response modeling is performed on the synthetic defect region of each enhanced inspection image unit in the enhanced inspection image set. First, based on the illumination direction vector, the cosine of the angle between the surface normal vector and the illumination direction vector at each pixel location of the synthetic defect region is calculated using the Lambertian volume reflection model. This cosine value is used as the diffuse reflection coefficient at that pixel. Then, the diffuse reflection coefficient is multiplied by the illumination intensity value at that pixel location in the illumination intensity distribution to obtain the initial illumination response value. Next, the gain coefficients of the red, green, and blue channels are adjusted according to the color temperature value so that the color shift of the initial illumination response value is consistent with the scene color temperature. Finally, an illumination response prediction map of the same size as the synthetic defect region is generated. This prediction map reflects the brightness and color performance that the synthetic defect texture should present under the illumination conditions acquired by this inspection image unit.

[0083] Step S213: Perform illumination inversion correction processing on the illumination response prediction map of the synthesized defect region, compare the difference in illumination direction consistency between the synthesized defect region and the background region of the inspection image unit, adjust the pixel values ​​in the synthesized defect region to make the illumination direction of the synthesized defect region consistent with that of the background region, and obtain the illumination-consistent synthesized defect region.

[0084] In this embodiment, due to the diversity of texture pattern sample sources, the original illumination response of the texture within the synthesized defect area may be inconsistent with the illumination direction of the background area of ​​the inspected image unit, resulting in a visual inconsistency. Illumination inversion correction processing is used to eliminate this inconsistency. First, the illumination direction representation of the background area surrounding the synthesized defect area is extracted, that is, the pixel gradient pattern in the background area arranged according to the illumination direction vector direction is calculated. Then, the corresponding gradient pattern is extracted from the illumination response prediction map of the synthesized defect area. The difference in gradient directions between the two is compared. If the difference exceeds a preset direction tolerance, the pixel values ​​in the synthesized defect area are translated and adjusted in the illumination direction dimension. The translation amount is the gradient direction difference multiplied by a gain coefficient, so that the gradient direction pattern of the synthesized defect area after adjustment is consistent with the illumination direction of the background area. After this processing, the illumination-consistent synthesized defect area blends seamlessly with the surrounding background in the illumination direction.

[0085] Step S214: Perform texture naturalization processing on the illumination consistency synthesis defect area, extract the texture roughness difference between the illumination consistency synthesis defect area and the surrounding background area in the texture transition zone, generate a texture gradient modulation signal, use the texture gradient modulation signal to fine-tune the texture density of the transition zone pixels, and generate a texture naturalization synthesis defect area.

[0086] In this embodiment, texture pattern samples from different sources may have different texture roughness, meaning the statistical characteristics of texture primitive size and spacing do not match the texture roughness of the surrounding background structure. Texture naturalization is used to bridge this difference. Texture roughness measures are extracted between the illumination consistency-synthesized defect region and the surrounding background region on both sides of the transition zone. Texture roughness is characterized by the ratio of local pixel value variance to local pixel gradient. The difference in texture roughness on both sides is calculated to generate a texture gradient modulation signal that smoothly transitions from the center of the defect region to the outer edge of the background region based on spatial location. This modulation signal is assigned a texture roughness difference value inside the foreground region and a value of zero outside the background region, exhibiting a linear transition within the transition zone. Using this texture gradient modulation signal, the spacing between texture primitives within the synthesized defect region is fine-tuned. The modulation method involves stretching or compressing the spatial arrangement density of texture primitives according to the sign and amplitude of the modulation signal, resulting in a smooth gradient of texture roughness from the defect region to the background region within the transition zone. The modulated texture naturalization synthesizes a defect region that achieves a natural transition in texture roughness with the surrounding background.

[0087] Step S215: Replace the texture naturalization synthesis defect area back into the original enhanced inspection image unit, perform multi-scale feathering fusion processing on the replacement boundary, and generate scene-adaptive enhanced inspection image units. All scene-adaptive enhanced inspection image units constitute a scene-adaptive enhanced inspection image set.

[0088] In this embodiment, the processed texture naturalization synthesis defect region is replaced back to the corresponding spatial position of the original enhanced inspection image unit. During replacement, a binary mask of the defect candidate region is used as the basic replacement template. To eliminate potential seams introduced by the replacement operation, multi-scale feathering fusion processing is performed on the replacement boundary. Multi-scale feathering fusion processing constructs a Gaussian pyramid, weighting the pixels on both sides of the replacement boundary at different scale levels. At higher scale levels, the fusion range is large to ensure color continuity, while at lower scale levels, the fusion range is small to preserve texture details. The fusion results at each level are reconstructed back to the original resolution to obtain a seamless replacement effect. The generated scene-adaptive enhanced inspection image unit simultaneously possesses lighting characteristics and texture naturalness matching the scene acquired by the inspection image unit. After traversing all enhanced inspection image units in the enhanced inspection image set and performing scene-adaptive fusion processing, all generated scene-adaptive enhanced image units are aggregated to form a scene-adaptive enhanced image set.

[0089] Step S216: Merge the scene adaptive enhanced inspection image set with the initial inspection image set to form a defect inspection comprehensive training sample set. The defect inspection comprehensive training sample set is used to train the defect inspection recognition model to improve the model's defect recognition generalization ability under different collection scenarios.

[0090] In this embodiment, the scene-adaptive enhanced inspection image set provides a large number of scene-aware and visually natural synthetic defect samples, while the initial inspection image set provides real original inspection image samples. The data from the two sets are merged to form a comprehensive defect inspection training sample set. In this training sample set, each sample is labeled with a defect type label, defect region location information, and collection scene feature label. This training sample set is used to train a defect inspection recognition model (such as a convolutional neural network-based object detection model or semantic segmentation model). During training, the model can learn invariant expressions of defect features under different lighting and climate collection scenarios, thus possessing stronger recognition generalization capabilities in the face of various complex environmental challenges during actual deployment.

[0091] Step S310: Perform morphological continuity constraint processing on the defect regions in the enhanced inspection image set, and adjust the morphological continuity of the defect regions at the same spatial location but different acquisition time phases in the enhanced inspection image set according to the preset defect growth and evolution law.

[0092] In this embodiment, the defect synthesis process in the enhanced inspection image set typically processes each inspection image unit independently, without considering the continuity of defect morphology at the same spatial location along the time axis. In the real physical world, the evolution of mine defects (such as cracks) is a gradual process, and the defect morphology of adjacent time phases should follow a certain continuous change pattern. By introducing morphological continuity constraint processing and using preset defect growth and evolution laws to coordinate the morphology of the synthesized defects in the time dimension, the generated enhanced inspection image sequence is made more consistent with physical reality.

[0093] Step S311: Select inspection image units with the same spatial location markers but different acquisition time phases from the initial inspection image set, and construct a temporal inspection image sequence corresponding to the spatial location. The temporal inspection image sequence is arranged in the order of acquisition time.

[0094] In this embodiment, the initial set of inspection images is traversed. Based on the spatial location information of each inspection image unit, all inspection image units with the same spatial location marker (i.e., ground coverage overlap exceeding a preset threshold) are grouped together. All inspection image units are selected from each group and arranged in chronological order according to their respective acquisition timestamps to construct a temporal inspection image sequence corresponding to a spatial location. This sequence reflects the surface state evolution of the same geographical area at multiple observation time points. For a fixed monitoring area of ​​a mine slope, the inspection image units in the sequence may be acquired at different times, such as the beginning, middle, and end of the year.

[0095] Step S312: Perform morphological comparison processing on the binary mask of the defect candidate region corresponding to each two adjacent time phases of the inspection image sequence, and extract the area change and boundary offset vector of the defect candidate region between adjacent time phases respectively.

[0096] In this embodiment, for a temporal inspection image sequence, each pair of adjacent temporal inspection image units is processed sequentially. The binary mask of the defect candidate region corresponding to the inspection image unit at the previous moment and the binary mask of the defect candidate region corresponding to the inspection image unit at the next moment are extracted. The area change is calculated, which is the difference between the foreground area of ​​the binary mask at the next moment and the foreground area of ​​the binary mask at the previous moment. The boundary offset vector is calculated by sampling several boundary points at equal intervals on the boundary of the binary mask at the previous moment, projecting along the normal direction of each boundary point onto the boundary of the binary mask at the next moment, and recording the projection distance and direction to form a set of boundary offset vectors. This area change and boundary offset vector quantitatively describe the direction and rate of morphological evolution of the defect region between adjacent temporal phases.

[0097] Step S313: Using a pre-trained defect growth and evolution prediction network, perform evolution trend analysis based on the area change and boundary offset vector to predict the morphological transition state of the defect candidate region between adjacent time phases, and generate a defect morphological intermediate state prediction mask. The defect growth and evolution prediction network is trained under the guidance of physical constraint rules of defect morphological evolution.

[0098] In this embodiment, a pre-trained defect growth and evolution prediction network is used to predict the morphological transition states. This network is a sequence prediction model employing a gated recurrent unit network structure. Its input is an encoded sequence of area changes and boundary offset vectors between several consecutive adjacent time phases in a temporal inspection image sequence. The output is the defect morphology parameters for several intermediate transition states between adjacent time phases. The network's training data consists of defect evolution sample pairs generated based on physical simulation models (such as fracture mechanics finite element simulation and rock weathering rate models). By adding a physical constraint penalty term (such as the consistency error between the predicted intermediate state boundary offset vector direction and the simulation result direction) to the loss function, the network's prediction results are ensured to conform to physical laws. In the application phase, the area changes and boundary offset vectors extracted in step S312 are input into the trained defect growth and evolution prediction network. The network outputs a defect morphology intermediate state prediction mask, which provides the expected morphology of the defect region at the intermediate transition moments between adjacent time phases.

[0099] Step S314: Compare the consistency of the defect morphology intermediate state prediction mask with the corresponding enhanced inspection image unit, and locate the region in the enhanced inspection image unit where the defect morphology and defect growth and evolution law are inconsistent as the morphological abnormal region.

[0100] In this embodiment, each enhanced inspection image unit corresponds to a specific acquisition time in the temporal sequence. The intermediate mask corresponding to that time in the predicted defect morphology intermediate state prediction mask between two actual observation times before and after that time is compared pixel-by-pixel with the enhanced inspection image unit's own defect candidate region binary mask. The intersection-over-union ratio (IoU) is calculated, and local regions with IoU below a preset threshold are marked. The marked regions are the morphological anomaly regions in the enhanced inspection image unit where the synthesized defect morphology does not match the morphology predicted based on evolutionary laws.

[0101] Step S315: Perform morphological correction processing on the binary mask of the defect candidate region in the morphologically abnormal region. Adjust the boundary of the morphologically abnormal region according to the boundary morphology of the defect morphology intermediate state prediction mask to make it conform to the defect growth and evolution law, and generate a morphologically continuous modified binary mask of the defect candidate region.

[0102] In this embodiment, for the located morphologically abnormal regions, the boundary morphology at the corresponding position in the defect morphology intermediate state prediction mask is used as a reference to perform morphological correction processing on these regions. The morphological correction processing is implemented through a morphological deformation algorithm, progressively moving the boundary control points of the morphologically abnormal region towards the boundary control points of the defect morphology intermediate state prediction mask. The movement step size is determined by the distance between the boundary control points and the co-threshold. After adjustment, the boundary of the morphologically abnormal region maintains its main shape features while converging towards the morphology predicted by the defect growth and evolution law, generating a binary mask of the defect candidate region with morphological continuity correction.

[0103] Step S316: Re-execute the defect morphology migration synthesis process using the binary mask of the defect candidate region after morphological continuity correction to generate enhanced inspection image units that are consistent with the defect growth and evolution law. All regenerated enhanced inspection image units are integrated into a set of morphological continuity enhanced inspection images.

[0104] In this embodiment, the binary mask of the defect candidate region after morphological continuity correction replaces the original binary mask with inconsistent morphology, and the defect morphological migration synthesis process in step S140 is re-executed. The defect texture pattern sample used during re-synthesis remains consistent with the previous one, with only the boundary of the defect candidate region changing. Therefore, the regenerated enhanced inspection image unit is similar to the previous one in texture representation, and its morphological contour is more consistent with the defect growth and evolution law. All the enhanced inspection image units regenerated after morphological continuity correction are integrated with the unaffected enhanced inspection image units in the original enhanced inspection image set to form a morphological continuity enhanced inspection image set. The image sequences in this set have a coordinated and consistent evolutionary trend in the time dimension.

[0105] Step S320: Perform local defect feature weakening and cyclic enhancement processing on the initial inspection image set, and gradually reduce the interference of background structure on defect feature extraction in the inspection image unit through iterative method.

[0106] In this embodiment, the mine inspection images contain a large number of regular background structures, such as protective netting arranged at fixed intervals, rut textures formed by rolling, and regular cross-sectional textures formed by directional blasting. These background structures exhibit strong periodic responses in the frequency and spatial domains, causing spectral aliasing with the local responses of certain defects (such as microcracks), which can easily lead to false alarms or missed detections during the defect segmentation stage. The local defect feature weakening and enhancement processing adopts an iterative strategy of alternating background attenuation and defect enhancement. Through multiple rounds of iteration, the interference of background structures is gradually stripped away, and the contrast of the weak defect signal is amplified round by round in the iteration.

[0107] Step S321: Select inspection image units with background structure complexity higher than a preset threshold from the initial inspection image set as inspection image units to be enhanced, perform background structure noise separation processing on the inspection image units to be enhanced, and use a multi-directional strip structure filter to extract regular texture structure information in the inspection image units and generate a background structure feature layer.

[0108] In this embodiment, the background structure complexity of all inspected image units is first quantitatively evaluated. Background structure complexity is measured by the weighted sum of the contrast and correlation statistics of the image's gray-level co-occurrence matrix; a higher value indicates more significant regular textures and stronger directionality in the image. A screening threshold is set, and inspected image units with background structure complexity exceeding this threshold are marked as image units to be enhanced. For each image unit to be enhanced, background structure noise separation processing is performed. This processing uses a set of strip-shaped filters covering different directions, each filter being a directional filtering kernel of the second-order partial derivative of a two-dimensional Gaussian function in a specific direction. The filter bank covers multiple directions uniformly sampled from 0 degrees to 180 degrees. The image unit to be enhanced is convolved with the directional filtering kernel corresponding to each direction to obtain the structure response map for that direction. The maximum response value and its corresponding direction are extracted pixel-by-pixel from the structure response maps of all directions to form a background structure feature layer. The background structural feature layer contains the orientation and intensity information of the regular texture structure in the inspection image unit, while the defect area has a very weak response in this layer because it does not have strip-shaped structural features with consistent orientation.

[0109] Step S322: Calculate the periodic arrangement pattern of the background structure in space based on the background structure feature layer, and generate a background structure periodic map, which records the repetition pattern of the background texture cells.

[0110] In this embodiment, after obtaining the background structure feature layer, the periodic arrangement pattern of the background structure in space is extracted through autocorrelation analysis. A two-dimensional autocorrelation operation is performed on the background structure feature layer. The autocorrelation function exhibits local peaks when the hysteresis coordinates are integer multiples of the background structure repetition period. The main peak positions of the two-dimensional autocorrelation function, excluding zero points, are detected. The hysteresis coordinates corresponding to these peak positions constitute a set of basis vectors, which describe the repetition period of the background texture cells in the row and column directions. The parallelogram spanned by this set of basis vectors is used as the periodic unit of the background texture cells, and the size of the periodic unit, the direction of the basis vectors, and the length are encoded into a background structure periodic map. This background structure periodic map accurately records the repetition patterns of background texture cells at different spatial locations within the inspected image unit. For background regions with non-uniform periods, a sliding window approach can be used to calculate the local periodic map block by block.

[0111] Step S323: Based on the background structure periodic spectrum, perform background structure pattern suppression processing on the image unit to be enhanced, target the frequency components corresponding to the background structure in the frequency domain space, retain the non-periodic frequency components corresponding to the defect area, and generate the enhanced intermediate image unit after background suppression.

[0112] In this embodiment, the background structure periodicity map obtained in the previous step is used to perform targeted attenuation of the background structure in the frequency domain. First, the image unit to be enhanced is transformed to the frequency domain using a two-dimensional discrete Fourier transform to obtain the amplitude spectrum and phase spectrum. Based on the basis vectors defined in the background structure periodicity map, the frequency coordinates corresponding to each basis vector in the frequency domain are calculated. Since regular periodic textures appear as discrete peaks in the frequency domain, the positions of these peaks are determined by the reciprocal relationship of the basis vectors in the frequency domain. The values ​​in the amplitude spectrum located at the above-mentioned frequency coordinates and their integer multiples of harmonic frequencies are attenuated by multiplying the amplitude value at the corresponding position by an attenuation factor less than 1. After the amplitude spectrum attenuation is completed, the modified amplitude spectrum is combined with the original phase spectrum and transformed back to the spatial domain using a two-dimensional discrete Fourier inverse transform to obtain the enhanced intermediate image unit after background suppression. In this intermediate image unit, the periodic background structure is significantly weakened, while non-periodic defect areas such as cracks and corrosion patches are basically unaffected due to the diffuse spectral distribution.

[0113] Step S324: Perform defect edge sharpening and reconstruction processing on the enhanced inspection intermediate image unit after background suppression, extract the gradient magnitude distribution information of the defect region in the enhanced inspection intermediate image unit, perform gradient enhancement on pixels with gradient magnitude lower than the edge preservation threshold, and generate an inspection image unit with enhanced defect region boundary.

[0114] In this embodiment, while background suppression eliminates background structures, it may also cause slight loss of edge gradients in the defect region, resulting in blurred defect contours. Defect edge sharpening and reconstruction is used to restore and enhance defect edges. The gradient magnitude distribution of the enhanced inspection intermediate image unit is calculated. The gradient magnitude is obtained by calculating the gradient components in the horizontal and vertical directions using the Sobel operator and then taking the modulus. The cumulative distribution of gradient magnitudes across the entire image is statistically analyzed, and the gradient magnitude corresponding to the lowest percentile in the cumulative distribution is set as the edge preservation threshold. For pixels with gradient magnitudes lower than this edge preservation threshold, their original pixel value is added to the gradient magnitude multiplied by a gain coefficient, thus enhancing the gradient in weak edge regions. For pixels with gradient magnitudes not lower than the edge preservation threshold, their original pixel values ​​remain unchanged. The resulting image unit with enhanced defect region boundaries removes background structure interference while maintaining clear defect edges.

[0115] Step S325: Use the inspection image unit with enhanced defect region boundary as the input for the next round of background structure noise separation processing, and iteratively execute background structure noise separation processing, background structure pattern suppression processing, and defect edge sharpening and reconstruction processing until the iteration end condition is met, and obtain a set of inspection image units enhanced by multiple rounds of iteration.

[0116] In this embodiment, a single-round background suppression process may not be able to completely remove complex background structures, especially for multi-layered nested textures (such as rust texture superimposed on a protective mesh). Therefore, an iterative enhancement strategy is adopted. The inspection image unit with enhanced defect region boundaries generated in step S324 is used as new input to replace the original inspection image unit to be enhanced. The background structure noise separation processing in step S321, the background structure periodic map construction in step S322, the background structure pattern suppression processing in step S323, and the defect edge sharpening and reconstruction processing in step S324 are re-executed. Before each iteration, the background structure complexity of the current image unit is re-evaluated. When the background structure complexity drops below a preset termination threshold, or the decrease in background structure complexity between two adjacent iterations is less than a preset convergence tolerance, the iteration termination condition is met, and the iteration stops. During the iteration process, the inspection image unit with enhanced defect region boundaries generated in each round is retained, and the outputs of all rounds together constitute a set of inspection image units enhanced by multiple rounds of iterative enhancement. In this set, the image units in later rounds have cleaner backgrounds and sharper defect edges; while the image units in earlier rounds retain more of the original lighting and texture details.

[0117] Step S326: Merge the set of inspection image units enhanced by multiple rounds of iteration with the enhanced inspection image set to generate a hybrid enhanced inspection image set containing multi-morphological enhancement and multiple rounds of iteration enhancement.

[0118] In this embodiment, the set of inspection image units enhanced through multiple iterations obtained in step S325 represents enhanced samples obtained by iteratively stripping background structures and enhancing defects in the original image, while the set of enhanced inspection images obtained in steps S130 to S140 represents enhanced samples obtained through defect morphology transfer synthesis. The two types of enhanced samples are complementary in terms of enhancement strategies, defect morphology sources, and background preservation methods. Merging the two sets and removing duplicates generates a hybrid enhanced inspection image set containing both multi-morphology enhancement and multi-iterative enhancement. This hybrid enhanced inspection image set can provide a richer distribution of training samples for the defect inspection and recognition model, covering a wide range of variations from original defect morphology to synthetic defect morphology, and from samples with strong background interference to samples with weak background interference.

[0119] Step S330: Perform separation enhancement processing on the overlapping regions of multiple defects in the enhanced inspection image set to solve the problem of mutual interference of enhancement synthesis effects when multiple defect types are spatially overlapping in the inspection image unit.

[0120] In this embodiment, mine inspection images commonly show multiple defects such as cracks, corrosion, and spalling overlapping in the same spatial area. In previous enhancement and compositing processes, directly implanting a defect texture into this overlapping area would destroy other existing defect information; if multiple textures were mixed and implanted, texture conflicts and artificial traces might occur. The separation enhancement process for overlapping areas with multiple defects solves this problem by processing the overlapping areas layer by layer according to defect type and compositing them layer by layer.

[0121] Step S331: Identify complex defect inspection image units with overlapping distribution of multiple defect types from the initial inspection image set; perform spatial location analysis on the overlapping areas of different defect types in the complex defect inspection image units; and determine the layering order of defect types based on the response intensity ranking of each defect response template in the overlapping area in the hierarchical differential feature map set.

[0122] In this embodiment, the initial inspection image set is first traversed, and a hierarchical differential feature map set is used to locate spatial regions in each inspection image unit that have high responses to two or more defect types. If a spatial location has two or more comprehensive response intensity values ​​higher than the response threshold in the differential feature maps of crack, corrosion, and deformation, then that location is determined to be a multi-defect overlapping region. Inspection image units containing multi-defect overlapping regions are marked as complex defect inspection image units. For each complex defect inspection image unit, the comprehensive response intensity value of each defect response template within its multi-defect overlapping region is extracted and sorted from high to low. This sorting determines the processing order of each defect type in the layered processing: the defect type with the highest response intensity is processed first as the bottom layer (most prominent layer), and the defect type with the lowest response intensity is processed last as the top layer, to ensure that the texture implantation of each defect type does not interfere with each other.

[0123] Step S332: Perform layer-by-layer defect morphology separation processing on the complex defect inspection image unit according to the defect type layer order. In each layer processing, only the binary mask of the defect candidate region corresponding to the current defect type is retained, and the pixels of the overlapping regions of other defect types are marked as regions to be filled.

[0124] In this embodiment, processing is performed layer by layer starting from the first layer (the layer with the highest response intensity) according to the layering order determined in step S331. In the processing of the current layer, only the binary mask of the defect candidate region corresponding to the defect type is extracted, and the foreground region in the mask is used as the processing range of the current layer. For the portion within the processing range of the current layer that overlaps with other defect types—that is, the foreground region in the binary mask of the defect candidate region of other defect types—its pixel value is marked as a region to be filled. This mark of the region to be filled is recorded in memory. This region is not written to pixels when the defect texture is implanted in the current layer, but is left to be filled in subsequent background texture completion processing. Defect regions in the current layer that do not overlap with other defect types are preserved normally. This layer-by-layer peeling process ensures that each layer processes only the region unique to the current defect type.

[0125] Step S333: Perform background texture completion processing on the area to be filled, and use the background texture information of the surrounding non-overlapping areas to interpolate and fill the pixel values ​​of the area to be filled, generating a pure defect enhancement base layer that removes interference from other defect types outside the current layer.

[0126] In this embodiment, pixels marked as areas to be filled are considered as areas with missing information. Background texture completion processing uses the background texture of non-overlapping areas surrounding the area to be filled to interpolate and fill the missing area. The interpolation process uses a sample block-based image inpainting algorithm. Starting from the boundary of the area to be filled, using the boundary pixel block as a template, the algorithm searches for the optimal match in the background pixel values ​​of the surrounding known background areas. The optimally matched pixel block is copied and filled to the corresponding boundary position of the area to be filled. Then, the boundary of the area to be filled is gradually shrunk inward with a pixel step size, and the matching and filling are repeated until all pixels in the area to be filled are filled. After filling, the pixel values ​​in this area only reflect the background texture and no longer contain the texture information of other defect types previously marked. The current layer obtains a clean defect-enhanced base layer that removes interference from other defect types.

[0127] Step S334: Perform defect morphology migration compositing processing on the defect candidate region of the current layer corresponding to the defect type in the pure defect enhancement base layer to obtain a single-type enhancement layer of the current defect type. The single-type enhancement layer only contains the enhancement compositing effect of the current layer defect type.

[0128] In this embodiment, for the clean defect enhancement base layer, the defect morphology migration compositing process in step S140 is invoked. The candidate defect region of the current layer's defect type is used as the compositing target region. Defect texture pattern samples matching the current layer's defect type are selected from a preset defect morphology library for adaptive texture primitive deformation and Poisson image fusion processing. Since the area to be filled has been completed by the background texture, the texture implantation during the defect morphology migration compositing process will not be interfered with by other defect types. The final generated single-type enhancement layer only contains the enhancement compositing effect of the current layer's defect type, while other defect types will be processed separately in subsequent layers.

[0129] Step S335: Obtain the single-type enhancement layer corresponding to all defect types, perform layer overlay and synthesis processing on multiple single-type enhancement layers in the order of layering, use the binary mask of the defect candidate region of each defect type as the layer transparency control template, and perform transparency weighted overlay on the defect region and background region in each single-type enhancement layer to generate a multi-defect overlapping and separated enhanced inspection image unit.

[0130] In this embodiment, after all defect types in the layering sequence have completed steps S332 to S334, multiple single-type enhancement layers are obtained. These single-type enhancement layers are then layered and composited according to the layering sequence determined in step S331. During layering, the original inspection image unit is used as the base image, and layers are stacked upwards from the bottom layer of the layering sequence. Each layer uses a binary mask of the defect candidate region for that defect type as a transparency control template: at pixel positions where the binary mask is the foreground, the pixels of that single-type enhancement layer participate in the blending with full transparency weight; at pixel positions where the binary mask is the background, the transparency weight of the pixels of that single-type enhancement layer is zero and they do not participate in the blending. After the multiple single-type enhancement layers are stacked, the resulting inspection image unit has different defect types occupying their respective defect candidate regions. The boundaries between regions transition naturally due to the layering, and there are no texture conflicts in the overlapping areas, resulting in an inspection image unit with multi-defect overlap and separation enhancement.

[0131] Step S336: Merge all the inspection image units with multi-defect overlap separation enhancement with the enhanced inspection image set to obtain an expanded enhanced inspection image set containing the multi-defect overlap separation enhancement effect.

[0132] In this embodiment, the inspection image units enhanced by multi-defect overlap separation generated in step S335 are merged into the previously constructed enhanced inspection image set. During the merging process, duplicate image units are removed. If both a normal enhanced version and a multi-defect overlap separation enhanced version exist for the same inspection image unit, both versions are retained and labeled for differentiation. The final result is an expanded enhanced inspection image set that includes the multi-defect overlap separation enhancement effect.

[0133] Step S340: Perform cross-modal defect feature mapping enhancement processing on the initial inspection image set, and enhance the image by utilizing the defect sensitivity characteristics in the non-visible light modal image corresponding to the spatial position of the inspection image unit.

[0134] In this embodiment, in addition to visible light cameras, thermal infrared cameras and ultrasonic scanning equipment are often used to simultaneously collect data during mine inspections. Thermal infrared images are sensitive to surface temperature anomalies caused by internal cavities or moisture infiltration, while ultrasonic scanning images are sensitive to subsurface cracks and delamination layers. Abnormal areas in these non-visible light modal images often indicate potential defects that are not easily detected or have not yet been revealed in the visible light images. Cross-modal defect feature mapping enhancement processing utilizes the defect sensitivity characteristics of the aforementioned non-visible light modes to guide the enhancement of visible light images.

[0135] Step S341: Obtain a set of non-visible light inspection images that have the same spatial location markers as the inspection image units in the initial inspection image set. The set of non-visible light inspection images includes thermal infrared inspection image units and ultrasonic scanning inspection image units.

[0136] In this embodiment, during the data acquisition phase, a UAV or ground inspection device simultaneously carries a visible light camera, a thermal infrared camera, and an ultrasonic scanning probe to collect data on the same mining area. The acquisition time of the three devices is synchronized, and the spatial coverage is registered at the pixel level through multi-sensor calibration. After acquisition, the visible light inspection image units, thermal infrared inspection image units, and ultrasonic scanning inspection image units are associated according to spatial location markers, ensuring that each visible light inspection image unit can find a corresponding registered image within the same spatial range in the non-visible light modal data. If a spatial location marker only acquires visible light data and not thermal infrared or ultrasonic data, the inspection image unit at that location skips cross-modal enhancement processing. After the above pairing and filtering, the qualified non-visible light modal image units are summarized into a non-visible light inspection image set.

[0137] Step S342: Perform temperature anomaly region extraction processing on the thermal infrared inspection image unit, identify temperature anomaly connected regions based on the temperature difference between the pixel values ​​in the thermal infrared inspection image unit and the surrounding area, and generate a thermal infrared anomaly region mask. The thermal infrared anomaly region mask marks the corresponding positions of possible internal defects in the visible light inspection image unit.

[0138] In this embodiment, the value of each pixel in the thermal infrared inspection image unit represents the thermal radiation temperature value corresponding to that spatial location. The temperature anomaly region extraction process first performs Gaussian filtering on the thermal infrared inspection image unit to eliminate isolated temperature jumps caused by sensor noise. Then, a sliding window-based temperature background estimation method is used. With each pixel as the center, the mean and standard deviation of the temperature values ​​within the neighborhood window are calculated. The current pixel temperature value is subtracted from the window temperature mean and then divided by the window temperature standard deviation to obtain the temperature anomaly value for that pixel. A temperature anomaly value threshold is set, and pixels with temperature anomaly values ​​higher than this threshold are marked as temperature anomaly pixels. Connectivity analysis is performed on all temperature anomaly pixels, aggregating spatially adjacent temperature anomaly pixels into several temperature anomaly connected regions. The outer boundary polygon of each temperature anomaly connected region is extracted and filled to generate a thermal infrared anomaly region mask. The foreground region of this mask corresponds to areas in the thermal infrared image with abnormal temperature performance. These areas may indicate the surface temperature characterization location of internal defects such as cavities and seepage channels within loose rock mass.

[0139] Step S343: Extract the acoustic impedance change region of the ultrasonic scanning inspection image unit, identify the boundary of the region where the acoustic impedance value changes abruptly in the ultrasonic scanning inspection image unit, and generate an acoustic impedance abnormal region mask. The acoustic impedance abnormal region mask marks the potential distribution range of subsurface defects in the visible light inspection image unit.

[0140] In this embodiment, the ultrasonic scanning inspection image unit records the amplitude and propagation time of the reflected echoes when ultrasonic waves propagate inside the rock mass. Interfaces of media with different densities can cause abrupt changes in acoustic impedance values, resulting in significant reflected echoes. The acoustic impedance variation region extraction process first extracts the echo amplitude sequence column by column along the depth direction from the ultrasonic scanning inspection image unit. For each echo amplitude sequence, its first-order difference is calculated; the difference value reflects the rate of change of acoustic impedance values ​​between adjacent depth sampling points. Depth locations where the absolute value of the first-order difference exceeds the acoustic impedance change rate threshold are marked as acoustic impedance abrupt change points. The spatial locations of all columns of acoustic impedance abrupt change points are mapped back to two-dimensional spatial coordinates, forming an acoustic impedance abrupt change point matrix. Spatial clustering and boundary tracing are performed on the acoustic impedance abrupt change point matrix to obtain the closed boundaries of the regions where acoustic impedance values ​​change abruptly. After region filling, an acoustic impedance anomaly region mask is generated. The foreground region of this mask marks the potential distribution range of defects such as cracks, voids, or abrupt changes in medium density beneath the surface.

[0141] Step S344: Spatially superimpose the thermal infrared anomaly region mask and the acoustic impedance anomaly region mask to complete the cross-modal anomaly mask fusion process. Assign a high confidence mark value to the part of the two anomaly region masks that overlap in spatial position, and assign a low confidence mark value to the anomaly region that appears only in a single mode, thereby generating a cross-modal defect confidence distribution map.

[0142] In this embodiment, the thermal infrared anomaly region mask generated in step S342 and the acoustic impedance anomaly region mask generated in step S343 are transformed into the same spatial coordinate system for layer overlay. The combination of marker values ​​from the two masks at a given location is analyzed pixel-by-pixel. If a pixel is marked as foreground in both the thermal infrared anomaly region mask and the acoustic impedance anomaly region mask, it indicates that the location receives support from both modes simultaneously, resulting in the highest probability of defect existence, and is assigned a high-confidence marker value. If a pixel is marked as foreground only in either the thermal infrared anomaly region mask or the acoustic impedance anomaly region mask, it indicates that the location receives support from only a single mode, resulting in a relatively low confidence level of defect existence, and is assigned a low-confidence marker value. These two marker values ​​correspond to different defect screening priorities. The confidence marker values ​​for all pixel locations are summarized to form a cross-modal defect confidence distribution map, which indicates the strength of the probability of defect existence inferred from non-visible light modal information at each spatial location.

[0143] Step S345: Based on the spatial distribution of different confidence marker values ​​in the cross-modal defect confidence distribution map, perform defect feature enhancement processing on the corresponding pixels of the high confidence marker area in the visible light inspection image unit to enhance the visual contrast between the high confidence marker area and the surrounding normal area, and generate a cross-modal enhanced inspection image unit.

[0144] In this embodiment, a cross-modal defect confidence distribution map is used as guiding information to perform targeted feature enhancement on visible light inspection image units. During processing, only pixels within high-confidence marked regions are enhanced, while low-confidence marked regions and regions with zero confidence remain unchanged. The specific operation of defect feature enhancement is as follows: within the high-confidence marked region, the grayscale or color value of the corresponding pixel in the visible light inspection image unit is extracted. Based on the sign and magnitude of the difference between this pixel value and the average value of its surrounding neighboring pixels, the magnitude of this difference is enhanced, i.e., the contrast difference between the background average and the region pixels is increased. The enhanced pixel value is then written back to the corresponding position of the visible light inspection image unit. After this processing, the high-confidence marked region becomes more prominent in the visible light image, and its visual difference from the surrounding normal region is actively amplified, which is beneficial for subsequent defect candidate region segmentation or manual interpretation. The generated image unit is denoted as a cross-modal enhanced inspection image unit.

[0145] Step S346: The region with enhanced defect features in the cross-modal enhanced inspection image unit is taken as the defect candidate region. The binary mask set of the defect candidate region and the enhanced inspection image set are used to perform defect morphology transfer synthesis processing to generate an enhanced inspection image unit set containing cross-modal defect feature guidance.

[0146] In this embodiment, the regions of defects with enhanced contrast in the cross-modal enhanced inspection image units are processed by defect region segmentation in step S130 to generate a binary mask of defect candidate regions with higher recall. This binary mask, guided by cross-modal features, marks potential defect regions that were not obvious in the visible light image but are supported by thermal infrared or ultrasonic evidence. The binary mask of defect candidate regions obtained through cross-modal guidance is paired one-to-one with the inspection image units in the initial inspection image set, and the defect morphology transfer and synthesis processing in step S140 is invoked to generate new enhanced inspection image units. The defect location and extent in these new enhanced inspection image units are determined by non-visible light information, and the synthesized defect texture originates from the visible light defect texture pattern, thus possessing both multimodal defect perception sensitivity and high-fidelity visual performance. All these newly generated enhanced inspection image units are summarized to generate a set of enhanced inspection image units guided by cross-modal defect features.

[0147] For example, step S360: perform defect evolution trajectory constraint enhancement processing on the enhanced inspection image set, and generate continuous transition state image units of defect evolution by utilizing the changing trend of defect morphology with the acquisition time sequence in the inspection image unit at the same spatial location.

[0148] In this embodiment, mine inspection is a periodic, repetitive observation process, with image acquisition conducted at regular intervals for deformation-sensitive areas such as slopes and spoil heaps. The acquired data from multiple time phases implicitly contain spatiotemporal trajectory information of defect growth. Defect evolution trajectory constraint enhancement processing models the spatial displacement and morphological change trends of defects over time, interpolating between adjacent time phases to generate enhanced images of intermediate transitional states. This ensures that the final training dataset includes not only discrete time-phase defect snapshots but also continuously evolving time-series samples.

[0149] Step S361: Extract all inspection image units with the same spatial location marker from the initial inspection image set, and construct a spatially aligned temporal inspection image sequence according to the acquisition time sequence information. The temporal inspection image sequence contains inspection image units arranged in chronological order, and each inspection image unit has an acquisition timestamp.

[0150] In this embodiment, spatial location grouping is performed on the initial inspection image set, grouping inspection image units with the same spatial location markers or whose spatial coverage overlaps with a preset confidence threshold into the same group. For each group, the inspection image units are sorted from earliest to latest according to their acquisition timestamps, constructing a spatially aligned temporal inspection image sequence. The acquisition intervals of the inspection image units in the sequence may be unequal, but the acquisition timestamps are all fully preserved.

[0151] Step S362: Perform spatiotemporal trajectory modeling processing on the time-series inspection image sequence for defect regions. Map the binary mask of the defect candidate region of each inspection image unit to a unified spatial coordinate system. Extract the centroid coordinate sequence of the defect candidate region in continuous time phases and generate the spatial displacement trajectory line of the defect. At the same time, extract the contour boundary sequence of the defect candidate region in continuous time phases to generate the defect morphology evolution trajectory.

[0152] In this embodiment, to analyze defect evolution under the same reference frame, the binary masks of defect candidate regions for all inspection image units in the time-series inspection image sequence are mapped to a unified spatial coordinate system through image registration transformation. This unified spatial coordinate system is selected as the image coordinate system of the first frame inspection image unit in the sequence. For each registered binary mask of a defect candidate region, the centroid coordinates and contour boundary vertex coordinate sequence of its foreground region are calculated. The centroid coordinates are ordered pairs formed by the mean of the row coordinates and the mean of the column coordinates of all pixels in the foreground region. Connecting the centroid coordinates of the defect candidate regions between every two adjacent frames forms a spatial displacement trajectory line of the defect extending over time. This trajectory line reflects the macroscopic movement trend of the defect position over time. Simultaneously, the contour boundary vertex coordinate sequences of the defect candidate regions in each frame are arranged in chronological order to form a defect morphological evolution trajectory. This trajectory records the change process of the defect contour shape over time.

[0153] Step S363: Perform trajectory density interpolation processing on the defect spatial displacement trajectory line, insert a preset number of intermediate transition centroid coordinates between the centroid coordinates of the defect candidate region in adjacent time phases to obtain a dense centroid coordinate trajectory line, perform deformation interpolation processing on the defect morphology evolution trajectory, generate intermediate transition contour boundaries corresponding to the number of intermediate transition centroid coordinates between the contour boundary sequences of the defect candidate region in adjacent time phases to obtain a dense morphology interpolation sequence.

[0154] In this embodiment, the time intervals between actual acquisition phases may be long. Directly using images with only a few discrete frames from the time series for training makes it difficult for the model to learn continuous evolutionary dynamics. Trajectory density interpolation processing involves linear interpolation between the centroid coordinates of adjacent phases, inserting a predetermined number of intermediate transition centroid coordinates at equal intervals. These intermediate transition centroid coordinates are evenly distributed along the time axis. After interpolation, the original sparse centroid sequence becomes a dense trajectory line with centroid coordinates. Simultaneously, the morphological evolution trajectory is also densed through interpolation: for two sets of contour boundary vertex coordinate sequences in adjacent phases, a contour interpolation algorithm based on shape context matching is used to generate a coordinate sequence of intermediate transition contour boundary vertices corresponding to the number of intermediate transition centroid coordinates. During interpolation, the physical rationality of contour shape changes is ensured; that is, the interpolated contour should be located between the morphological envelopes of the two end contours. The resulting dense morphological interpolation sequence describes the continuous changing morphology of the defect contour along the time axis at a frequency much higher than the original acquisition.

[0155] Step S364: Construct a defect evolution transition state parameter set based on the dense trajectory line of the centroid coordinates and the dense sequence of morphological interpolation. Each set of parameters in the defect evolution transition state parameter set corresponds to an intermediate transition phase. Each set of parameters includes the defect centroid coordinates and defect contour boundary vertex coordinate sequence of the intermediate transition phase.

[0156] In this embodiment, the centroid coordinates of each intermediate transition in the dense centroid coordinate trajectory line are packaged with the intermediate transition contour boundary vertex coordinate sequence of the corresponding time index in the morphological interpolation dense sequence to form a set of defect evolution transition state parameters. Each set of parameters includes the timestamp offset of the intermediate transition phase (time difference relative to the first frame of the sequence), the defect centroid coordinates, and the defect contour boundary vertex coordinate sequence. All transition state parameter sets are arranged in chronological order to form a defect evolution transition state parameter set.

[0157] Step S365: Read each set of transition state parameters sequentially from the defect evolution transition state parameter set, generate a binary mask of the transition state defect region using the defect contour boundary vertex coordinate sequence in the transition state parameters, and spatially register the binary mask of the transition state defect region with the defect centroid coordinates in the transition state parameters to obtain a binary mask of the transition state defect candidate region.

[0158] In this embodiment, the set of transition state parameters for defect evolution is traversed. For each set of transition state parameters, a closed polygon region is constructed in the image coordinate system based on the sequence of vertex coordinates of the defect contour boundary. This region is then filled to generate a binary mask for the transition state defect region. The centroid coordinates of the foreground region of this binary mask are aligned with the centroid coordinates of the defect in the transition state parameters. If there is an offset, the binary mask is translated as a whole to make them coincide. After registration, a binary mask for the candidate transition state defect region is generated.

[0159] Step S366: The step of calling the binary mask set of the defect candidate region and the inspection image unit of the initial inspection image set to perform defect morphology migration synthesis processing. For each transition state defect candidate region binary mask and its corresponding intermediate transition phase, the inspection image unit with the acquisition timestamp closest to the intermediate transition phase is selected from the initial inspection image set as the background image unit.

[0160] In this embodiment, for each candidate defect region of the transition state, a binary mask is used to obtain the timestamp of its corresponding intermediate transition phase. The inspection image unit with the smallest absolute value of the difference between the acquisition timestamp and the timestamp of the intermediate transition phase is retrieved from the initial inspection image set, and this inspection image unit is used as the background image unit. The background image unit provides the real background texture and lighting conditions of the mine surface during the intermediate transition phase.

[0161] Step S367: Perform defect morphology migration synthesis processing on the binary mask of the transition state defect candidate region and the background image unit. Under the premise of keeping the background of the background image unit unchanged, replace the texture distribution of the transition state defect candidate region with the defect texture pattern in the preset defect morphology library corresponding to the defect type of the spatial location, and generate the defect evolution transition state enhanced inspection image unit.

[0162] In this embodiment, the defect morphology migration and synthesis process in step S140 is invoked. The binary mask of the transitional defect candidate region is used as the synthesis target, and the background image unit is used as the base image. A defect texture pattern consistent with the spatial location in adjacent real-time phases is selected from a preset defect morphology library. Adaptive deformation of texture primitives and Poisson image fusion are then performed to generate an enhanced inspection image unit for the transitional defect evolution state. This image unit fills the sample gap between the two original real-time acquisition phases, and its defect morphology follows an interpolation trajectory, forming a smooth evolutionary relationship with the defect morphology of the preceding and following real-time phases.

[0163] Step S368: Arrange all the enhanced inspection image units corresponding to the defect evolution transition states during the intermediate transitions and the original enhanced inspection image units in the enhanced inspection image set according to the acquisition time sequence, and fill them into the corresponding time positions in the time sequence inspection image sequence to generate a set of enhanced inspection image sequences with continuous defect evolution trajectories.

[0164] In this embodiment, using the temporal inspection image sequence corresponding to each spatial location as the organizational framework, the enhanced inspection image units of the defect evolution transition state of each intermediate transition phase generated in step S367 are inserted into the corresponding positions in the sequence according to their timestamps. Together with the enhanced inspection image units corresponding to the real time in the original sequence, they form an enhanced inspection image sequence with dense temporal sampling and continuous defect evolution trajectory. The set of the above-mentioned enhanced inspection image sequences corresponding to all spatial locations constitutes the set of enhanced inspection image sequences with continuous defect evolution trajectory.

[0165] Step S369: Add the set of enhanced inspection image sequences with continuous defect evolution trajectories as expanded samples to the training dataset to train the defect inspection and recognition model, thereby improving the model's ability to identify intermediate forms during defect evolution and predict defect development trends.

[0166] In this embodiment, each image unit, along with its temporal label and defect morphology label, from the set of enhanced inspection image sequences showing continuous defect evolution trajectories is added to the training dataset of the defect inspection and recognition model. During training, the input to the model is no longer limited to isolated frame images, but also includes temporal image groups composed of consecutive adjacent frames. By learning the patterns of defect morphological changes over time, the model improves its ability to identify intermediate forms of defect evolution and predict future defect development trends.

[0167] Step S370: Perform defect background context association enhancement processing on the enhanced inspection image set, and generate enhanced inspection image units that maintain background context consistency by extracting the spatial dependency relationship between the defect region and the surrounding background structure in the inspection image unit.

[0168] In this embodiment, the actual morphology of mine defects does not appear in isolation; its texture direction and scale are often related to the surrounding background rock structure. For example, the extension direction of rock fissures is usually consistent with the direction of mountain joint surfaces, and corrosion areas often extend along the boundaries of rock strata textures. The defect background context association enhancement processing quantifies and models the above spatial dependencies and constrains this association to the defect morphology migration and synthesis process, so that the synthesized defect texture is consistent with the background structure in terms of direction and distribution.

[0169] Step S371: Extract the defect region marked by the binary mask of the defect candidate region from the inspection image unit of the initial inspection image set, perform background structure context extraction processing on the background region within a preset buffer distance outside the defect region, and encode the texture direction and edge distribution of the background region using the directional gradient histogram description algorithm to generate a background structure context description vector. The background structure context description vector includes the texture main direction distribution histogram and texture density distribution histogram of the background region.

[0170] In this embodiment, for the defect region marked by the binary mask of the defect candidate region in each inspection image unit, a preset buffer distance is extended outward from its boundary to form a ring-shaped background region band surrounding the defect region. Background structure context extraction is performed on this ring-shaped background region band. First, the gradient direction and gradient magnitude of each pixel within this region are calculated, and the gradient direction is quantized into a preset number of direction intervals. The cumulative sum of the gradient magnitudes within each direction interval is calculated to construct a texture principal direction distribution histogram. The direction interval corresponding to the peak of the histogram is the principal direction of the texture in this background region. Simultaneously, the ring-shaped background region band is divided into sub-blocks, and the average magnitude of the pixel gradient within each sub-block is calculated. The average gradient magnitudes of each sub-block are arranged according to the spatial position of the sub-blocks to form a texture density distribution histogram. The texture principal direction distribution histogram and the texture density distribution histogram are encoded into a fixed-length background structure context description vector.

[0171] Step S372: Extract the defect texture direction features of the defect texture pattern inside the defect region, encode the texture direction in the defect region using the same orientation gradient histogram description algorithm as the background structure context description vector, generate a defect texture direction distribution histogram, calculate the orientation consistency measure between the defect texture direction distribution histogram and the texture main direction distribution histogram in the background structure context description vector, and obtain the defect background orientation matching degree.

[0172] In this embodiment, for the defect texture pattern sample to be used in the defect area, or for the original defect texture already existing in the defect area, the histogram of orientation gradients with the same parameters as in step S371 is used to extract its texture orientation distribution histogram. This defect texture orientation distribution histogram is compared with the texture principal orientation distribution histogram in the background structure context description vector at the same position in step S371, and the orientation consistency metric between the two histograms is calculated. The orientation consistency metric is the histogram intersection between two normalized histograms, i.e., the smaller value of the two histograms in each orientation interval is taken and summed. The orientation consistency metric value is normalized; the closer it is to the upper limit, the more consistent the defect texture orientation is with the background texture orientation; the closer it is to the lower limit, the more the orientation deviates. This orientation consistency metric is the defect-background orientation matching degree.

[0173] Step S373: Based on the defect background direction matching degree, perform directional screening on the defect texture pattern samples in the preset defect morphology library, and select defect texture pattern samples whose texture main direction distribution histogram matching degree with the background structure context description vector is higher than the directional matching threshold from the crack texture pattern sample set, corrosion texture pattern sample set, deformation texture pattern sample set and peeling texture pattern sample set as candidate directional defect texture sample set.

[0174] In this embodiment, before performing defect morphology transfer synthesis, a preset defect morphology library is selectively filtered. All defect texture pattern samples in the preset defect morphology library are traversed. For each defect texture pattern sample, its directional consistency metric with the histogram of the main texture direction distribution in the background structure context description vector generated in step S371 is calculated. A directional matching threshold is set, retaining defect texture pattern samples with a matching degree higher than the threshold and discarding samples with a matching degree lower than the threshold. The retained samples constitute a candidate directional defect texture sample set. Subsequent defect morphology transfer synthesis processing will select defect texture patterns only from this candidate directional defect texture sample set, thereby ensuring that the direction of the implanted defect texture is coordinated with the main texture direction of the surrounding background structure.

[0175] Step S374: Perform texture rearrangement processing on the defect texture pattern samples in the candidate directional defect texture sample set under background structure context constraints. Adjust the spatial arrangement density of texture primitives of the candidate directional defect texture samples according to the texture density distribution histogram of the background structure context description vector, so that the texture primitive distribution density of the rearranged defect texture pattern is consistent with the texture density of the background area around the defect area, and generate a defect texture pattern constrained by the background context.

[0176] In this embodiment, orientation matching alone is insufficient to guarantee texture harmony; the density of the defect texture must also match the background texture. For each sample in the candidate oriented defect texture sample set, its texture primitives and their original spatial arrangement are extracted using the texture primitive analysis method in step S142. Then, based on the texture density distribution histogram in the background structure context description vector, a spatial density field is constructed. This density field takes values ​​consistent with the background texture density at the defect region boundary, smoothly transitioning into the defect region to ensure that no abrupt change in texture density occurs at the boundary after the defect texture is implanted. The texture primitives of the defect texture pattern sample are rearranged using this spatial density field: the spatial spacing of the texture primitives is compressed at locations where the spatial density field requires higher density; the spatial spacing of the texture primitives is expanded at locations where lower density is required. The rearranged texture primitives are then resynthesized into a defect texture pattern constrained by the background context.

[0177] Step S375: The defect texture pattern constrained by the background context is used as the defect texture pattern in the defect morphology transfer synthesis process. Texture implantation processing is performed on the foreground region marked by the binary mask of the defect candidate region. During the texture implantation process, the texture direction of the defect candidate region matches the main texture direction of the background region, and the density of texture primitives in the defect candidate region transitions continuously with the density of textures in the surrounding background region, thereby generating an enhanced inspection image unit with background context association.

[0178] In this embodiment, the original defect texture pattern sample directly read from the preset defect morphology library is replaced by the defect texture pattern constrained by the background context generated in step S374, and the texture implantation process of steps S144 to S146 is executed. Since the defect texture pattern constrained by the background context has undergone direction filtering and density rearrangement, the direction of the defect texture after implantation is naturally aligned with the main direction of the background texture, and the density of the texture smoothly transitions from the boundary of the defect region to the interior, without any abrupt boundary in texture density. The enhanced inspection image unit associated with the background context is highly natural and coordinated at the junction of the background texture and the defect texture.

[0179] Step S376: Perform background context differentiation enhancement processing on enhanced inspection image units of different defect types at the same spatial location. Extract the background structure context description vector of the background region around the defect area for different types of defects. Generate a defect texture pattern with background context constraints corresponding to each defect type for each defect type, so that the background context constraints in the enhanced inspection image units of different types of defects remain consistent.

[0180] In this embodiment, for cases where multiple defect types need to be synthesized at the same spatial location, since different defect types may be located at different specific coordinates within that spatial location, their respective local background structural contexts may differ. Therefore, steps S371 to S375 are performed separately for each defect type, extracting the background structural context description vector corresponding to that defect type's location individually, and separately filtering and generating the defect texture pattern corresponding to the background context constraints of that defect type. This ensures that the enhanced inspection image units for different types of defects can maintain the consistency of background context constraints within their respective regions.

[0181] Step S377: Merge the enhanced inspection image unit with background context association and the enhanced inspection image unit after background context differentiation enhancement processing with the enhanced inspection image set to obtain an expanded enhanced inspection image set with background context association characteristics. The expanded enhanced inspection image set is used to train the defect inspection and recognition model to enhance the robustness of the model in identifying defects in complex background texture environments.

[0182] In this embodiment, the enhanced inspection image units generated in step S375 with background context association and the enhanced inspection image units generated in step S376 with background context differentiation enhancement are merged into an enhanced inspection image set, forming an expanded enhanced inspection image set with background context association characteristics. When training the defect inspection and recognition model with this expanded enhanced inspection image set, the model can access more samples where the background texture and defect texture are truly correlated, thereby reducing the number of false detections of defects due to texture inconsistencies and enhancing the robustness of recognition in complex background texture environments.

[0183] Step S380: Perform defect generation adversarial enhancement processing on the enhanced inspection image set, and generate realistic enhanced inspection image units by constructing an adversarial training mechanism between the defect generation network and the defect discrimination network.

[0184] In this embodiment, the aforementioned defect morphology transfer synthesis processing is mainly based on texture primitive analysis and Poisson image fusion. The synthesized defects are close to real in terms of micro-texture and macro-shape, but at extreme detail levels, there may still be distribution differences compared to naturally formed defects. The defect generation adversarial enhancement processing introduces an adversarial training mechanism of generative adversarial networks. It uses a defect discrimination network to capture the subtle distribution differences between real and synthesized defects and drives the defect generation network to continuously improve the realism of the synthesized image until the discrimination network can no longer distinguish between real and fake.

[0185] Step S381: Construct a defect generation network, which includes a defect texture encoding sub-network, a defect morphology generation sub-network, and an image synthesis sub-network. The defect texture encoding sub-network receives the texture primitive feature description vector of randomly sampled defect texture pattern samples from a preset defect morphology library as input, and converts the texture primitive feature description vector into a defect texture latent space feature map through multiple cascaded transposed convolutional layers. The defect texture latent space feature map contains a high-dimensional abstract representation of the defect texture.

[0186] In this embodiment, the defect generation network adopts an encoder-decoder structure and mainly includes three functional sub-networks. The defect texture encoding sub-network is a deconvolutional neural network. The input is a texture primitive feature description vector extracted from defect texture pattern samples randomly sampled from a preset defect morphology library after texture primitive analysis. This vector is first mapped to an initial feature map that is spatially tiled through a fully connected layer. Then, it is sequentially passed through multiple cascaded transposed convolutional layers for upsampling and feature transformation. Each transposed convolutional layer is followed by batch normalization and nonlinear activation operations to gradually generate a latent spatial feature map of defect texture with rich texture details. This latent spatial feature map of defect texture is close to the size of the final defect region to be implanted in terms of spatial size, and encodes a high-dimensional abstract representation of the defect texture in the channel dimension, including multi-dimensional information such as the shape, orientation, and density of texture primitives.

[0187] Step S382: The defect morphology generation subnetwork receives the shape constraint parameters of the binary mask of the defect candidate region and the latent spatial feature map of the defect texture as input. Through the morphological attention guidance mechanism, the latent spatial feature map of the defect texture is deformed and clipped in the spatial dimension according to the shape of the defect region specified by the shape constraint parameters to generate a shape-adaptive defect texture generation map. The morphological attention guidance mechanism calculates the spatial attention weight map according to the shape constraint parameters of the defect region. The weight value of the spatial attention weight map inside the defect region is higher than the weight value outside the defect region.

[0188] In this embodiment, the defect morphology generation subnetwork is responsible for constraining the latent spatial feature map of the defect texture to a specific defect region shape. The input shape constraint parameters consist of the sequence of polygon vertex coordinates of the defect region extracted from the binary mask of the defect candidate region and the corresponding region shape constraint parameters. This subnetwork internally constructs a morphological attention guidance mechanism: first, the shape constraint parameters are rendered as a binary reference map with the same spatial size as the latent spatial feature map of the defect texture, with upper limits assigned to the interior of the defect region and lower limits assigned to the exterior; this binary reference map is Gaussian blurred to generate a continuously transitioning spatial attention weight map, where the weight values ​​approach the upper limit inside the defect region, decay at the boundary, and approach the lower limit outside. The spatial attention weight map is then multiplied element-wise along the channel dimension of the latent spatial feature map of the defect texture to achieve position-weighted attention guidance, significantly suppressing the feature map outside the defect region. Subsequently, thin-plate spline interpolation is used, with the boundary vertices in the shape constraint parameters as control points, to spatially deform the attention-guided feature map, accurately mapping it to the actual boundary of the defect region, generating a shape-adaptive defect texture generation map.

[0189] Step S383: The image synthesis sub-network receives the shape-adaptive defect texture generation map, the defect candidate region binary mask, and the original inspection image units in the initial inspection image set as input. Using the defect candidate region binary mask as a fusion weight template, the shape-adaptive defect texture generation map and the original inspection image units are fused pixel by pixel. Pixels in the defect candidate region are replaced by pixels in the shape-adaptive defect texture generation map, while pixels outside the defect candidate region remain unchanged from the pixels in the original inspection image units, thus generating candidate enhanced inspection image units.

[0190] In this embodiment, the image synthesis sub-network performs the final image synthesis operation. This sub-network takes into input a shape-adaptive defect texture generation map, a binary mask of defect candidate regions, and the original inspection image unit. The binary mask of the defect candidate regions is used as a fusion weight template. For pixels marked as foreground in the binary mask, the output pixel value is equal to the corresponding pixel value in the shape-adaptive defect texture generation map; for pixels marked as background in the binary mask, the output pixel value is equal to the corresponding pixel value in the original inspection image unit. At the boundary of the defect region, weighted mixing is performed using the smooth transition band of the binary mask, with the mixing weight changing linearly with distance. The synthesized output is a candidate enhanced inspection image unit, in which the texture of the defect region is entirely generated by the defect generation network, while the background region completely retains the original image content.

[0191] Step S384: Construct a defect discrimination network. The defect discrimination network includes a multi-scale convolutional feature extraction module and a defect authenticity determination module. The multi-scale convolutional feature extraction module performs downsampling feature extraction on the input inspection image units, extracts texture and structural features of the defect region and background region at different scale levels, and generates a multi-scale defect discrimination feature map. The defect authenticity determination module performs global average pooling on the multi-scale defect discrimination feature map and inputs the pooling result into the fully connected layer to output an image authenticity score.

[0192] In this embodiment, the defect discrimination network is designed as a binary classification convolutional neural network. Its multi-scale convolutional feature extraction module consists of multiple stacked convolutional blocks. Downsampling is performed every two convolutional blocks, halving the spatial size of the feature map and doubling the number of channels. Through this hierarchical downsampling design, the module can extract texture and structural features of the defect and background regions at different spatial scales. After the output of the last convolutional block, a multi-scale defect discrimination feature map is obtained. The defect authenticity determination module receives the multi-scale defect discrimination feature map, performs global average pooling on it, and compresses the feature map of each channel into a scalar value to obtain a feature vector. This feature vector passes through two fully connected layers and finally outputs an image authenticity score through an output node. This score represents the probability that the input inspection image unit is a real defect image.

[0193] Step S385: Extract real defect inspection image units from the initial inspection image set to construct a real defect sample set; extract generated enhanced inspection image units from the enhanced inspection image set to construct an enhanced defect sample set; and alternately input the real defect inspection image units from the real defect sample set and the candidate enhanced inspection image units generated by the defect generation network into the defect discrimination network for adversarial training.

[0194] In this embodiment, during the training data preparation phase, inspection image units containing obvious real defects (confirmed by manual annotation) are selected from the initial inspection image set to construct a real defect sample set. Enhanced inspection image units are sampled from the generated enhanced inspection image set to construct an enhanced defect sample set. In one iteration of adversarial training, a batch of real defect inspection image units is randomly selected from the real defect sample set and input into the defect discrimination network. The realism loss is calculated, and the parameters of the defect discrimination network are updated to output a high realism score for the real images. Then, a defect generation network is used to process a batch of random defect texture pattern samples and binary masks of defect candidate regions to generate candidate enhanced inspection image units. These candidate enhanced inspection image units are input into the defect discrimination network, the realism loss is calculated, and the parameters of the defect discrimination network are updated again to output a low realism score for the generated images. Next, the parameters of the defect discrimination network are frozen. Using the feedback gradient of the realism scores output by the defect discrimination network for a batch of new candidate enhanced inspection image units, the parameters of the defect texture encoding subnetwork, defect morphology generation subnetwork, and image synthesis subnetwork in the defect generation network are backpropagated to update the parameters, so that the generated image can deceive the discrimination network to obtain a high realism score. The above steps are performed alternately and iteratively.

[0195] Step S386: During adversarial training, the defect generation network updates the network weight parameters of the defect texture encoding sub-network, the defect morphology generation sub-network, and the image synthesis sub-network based on the feedback gradient of the image authenticity score of the candidate enhanced inspection image unit by the defect discrimination network. The defect discrimination network updates the network weight parameters of the multi-scale convolutional feature extraction module and the defect authenticity determination module based on the authenticity discrimination error of the real defect inspection image unit and the candidate enhanced inspection image unit, until the defect discrimination network can no longer distinguish between the real defect inspection image unit and the candidate enhanced inspection image unit.

[0196] In this embodiment, the loss functions during adversarial training are designed for the defect generation network and the defect discrimination network, respectively. The loss function of the defect discrimination network is a binary cross-entropy loss, with the expected output of the real defect inspection image unit as the upper limit and the expected output of the candidate augmentation inspection image unit as the lower limit. The loss function of the defect generation network is also a binary cross-entropy loss, but the expected output of the candidate augmentation inspection image unit is set to the upper limit, thereby driving the generation network to produce more realistic images. An adaptive moment estimation optimizer is used to alternately optimize the two networks, with the initial learning rate set at a low level to ensure training stability. During training, the classification accuracy of the discrimination network on a batch of validation sets is monitored. When the classification accuracy fluctuates within a preset convergence range for a long time and cannot be steadily improved, it is determined that the adversarial training has reached an equilibrium state. At this point, the defect discrimination network can no longer effectively distinguish between real defect inspection image units and candidate augmentation inspection image units.

[0197] Step S387: The defect generation network that has reached the adversarial training equilibrium state is used as the enhanced inspection image generator. The inspection image units in the initial inspection image set and the binary masks of the defect candidate regions in the defect candidate region binary mask set are input into the enhanced inspection image generator in pairs to generate adversarial enhanced inspection image units with high-fidelity defect texture details and natural fusion boundaries. All adversarial enhanced inspection image units constitute the adversarial enhanced inspection image set.

[0198] In this embodiment, the parameters of the defect generation network model, saved after reaching adversarial training equilibrium, are loaded into the enhanced inspection image generator. Each inspection image unit in the initial inspection image set is paired with its corresponding binary mask of the defect candidate region, and each pair is input into the enhanced inspection image generator to generate a series of adversarial enhanced inspection image units. The defect texture in the aforementioned adversarial enhanced inspection image units is entirely generated by the generative network optimized through adversarial training, exhibiting extremely high detail realism and natural fusion boundaries.

[0199] Step S388: Merge the adversarial enhanced inspection image set with the enhanced inspection image set to obtain a comprehensive enhanced defect inspection image set containing adversarial generation enhanced samples. The comprehensive enhanced defect inspection image set is used to train the defect inspection recognition model to improve the model's ability to distinguish subtle defect texture differences.

[0200] In this embodiment, the adversarial enhanced inspection image set generated in step S387 serves as another source of enhanced samples. It is merged with the enhanced inspection image sets generated by the aforementioned multiple enhancement strategies, and after deduplication and the addition of generation strategy labels, a final comprehensive enhanced image set for defect inspection is formed. This comprehensive enhanced image set integrates massive amounts of samples generated by all enhancement strategies, including defect morphology transfer synthesis, scene adaptive fusion, morphological continuity constraints, local defect feature weakening loop enhancement, multi-defect overlap separation enhancement, cross-modal defect feature mapping enhancement, defect severity grading enhancement, defect evolution trajectory constraint enhancement, defect background context association enhancement, and defect generation adversarial enhancement. Using this comprehensive enhanced image set as training data for the defect inspection and recognition model can comprehensively enhance the model's recognition performance from multiple dimensions, including defect type coverage, robustness to illumination in the acquisition scene, temporal evolution continuity, background texture coordination, defect severity perception, and subtle texture discrimination capabilities.

[0201] Based on the same inventive concept, please refer to Figure 2 The diagram shows a schematic block diagram of an image data enhancement processing system 100 combined with defect inspection provided in an embodiment of this application. The image data enhancement processing system 100 combined with defect inspection may include a communication unit 110, a machine-readable storage medium 120, and a processor 130.

[0202] In this embodiment, alternatively, the machine-readable storage medium 120 can also be integrated into the processor 130 and can communicate and interact with external systems through the communication unit 110. The machine-readable storage medium 120 stores machine-executable instructions for executing the scheme of this application, and the processor 130 executes the machine-executable instructions stored in the machine-readable storage medium 120 to implement the image data enhancement processing method combined with defect inspection provided in the aforementioned method embodiments.

[0203] It should be noted that, in order to simplify the description of the present invention and thus help to understand one or more embodiments of the invention, multiple features may sometimes be grouped into one embodiment, drawing or description thereof in the foregoing description of the embodiments of the present invention.

Claims

1. An image data enhancement processing method combined with defect inspection, characterized in that, The method includes: Acquire the initial inspection image set corresponding to the defect inspection task, wherein the initial inspection image set contains inspection image units with acquisition time information and spatial location information; The initial inspection image set is subjected to multi-level differential feature construction processing. The inspection image units are decomposed layer by layer using the pixel response differences corresponding to different defect morphologies to generate a hierarchical differential feature map set containing defect morphology sensitive features and background suppression features. Based on the hierarchical differential feature map set, defect region segmentation processing is performed. The boundary of potential defect region is determined according to the response distribution of defect morphology sensitive features in the hierarchical differential feature map, and a binary mask set of defect candidate regions is generated. The binary mask set of the defect candidate region is combined with the inspection image unit of the initial inspection image set to perform defect morphology migration and synthesis processing. While keeping the background of the inspection image unit unchanged, the texture distribution of the defect candidate region is replaced with the defect texture pattern in the preset defect morphology library to generate an enhanced inspection image set containing multiple defect types.

2. The image data enhancement processing method combined with defect inspection according to claim 1, characterized in that, The initial inspection image set undergoes multi-level differential feature construction processing. This involves decomposing the inspection image units layer by layer using the pixel response differences corresponding to different defect morphologies, generating a hierarchical differential feature map set containing defect morphology-sensitive features and background suppression features. This includes: A multi-scale pixel neighborhood structure is constructed for each inspection image unit in the initial inspection image set. The multi-scale pixel neighborhood structure is centered on each pixel in the inspection image unit and establishes a multi-layer nested annular sampling region according to a preset neighborhood expansion coefficient. The pixel sampling point distribution density of each annular sampling region is different. Pixel value encoding is performed on the multi-layer nested annular sampling regions respectively. The pixel value deviation between the pixel sampling point and the center pixel point in each annular sampling region is calculated to generate pixel deviation response vectors in neighborhoods of different scales. The dimension of the pixel deviation response vector corresponds to the number of pixel sampling points in the annular sampling region of that layer. According to the preset defect morphology pixel response template library, crack-type defect response template, corrosion-type defect response template, and deformation-type defect response template are selected from the defect morphology pixel response template library. The pixel deviation response vectors in the neighborhood of different scales are subjected to layer-by-layer convolution matching processing with each defect response template to generate the defect morphology matching response intensity at each scale. The defect morphology matching response intensity at each scale is subjected to inter-scale feature fusion processing through cross-scale response aggregation operation. The defect morphology matching response intensity of adjacent scales is cascaded along the scale dimension to generate a multi-scale defect morphology sensitive feature vector. Each element in the multi-scale defect morphology sensitive feature vector corresponds to the matching degree of a defect morphology response template at a certain scale. Background suppression filtering is performed on the multi-scale defect morphology sensitive feature vector. The contribution uniformity of each element in the multi-scale defect morphology sensitive feature vector within the spatial neighborhood is calculated. Feature elements with contribution uniformity lower than a preset differentiation threshold are extracted as defect morphology sensitive features. Feature elements with contribution uniformity not lower than the preset differentiation threshold are zeroed out to obtain background suppression features. The defect morphology sensitive features and the background suppression features are stacked and recombined element by element according to their original spatial positions to generate differential feature layers of different levels. Each differential feature layer corresponds to a matching response spatial distribution of a defect morphology response template. Based on the complementary relationship between the features of the differential feature layers corresponding to different defect morphology response templates within the same inspection image unit, the mutual information metric between each differential feature layer is calculated, and the differential feature layers with mutual information metrics higher than the preset complementary threshold are incorporated into the same hierarchical differential feature map group. Layer fusion processing is performed on the differential feature layers within the same hierarchical differential feature map group to generate a fused differential feature map that characterizes the comprehensive response intensity of the corresponding defect morphology. The fused differential feature maps generated by different hierarchical differential feature map groups are combined along the hierarchical dimension to obtain a set of hierarchical differential feature maps.

3. The image data enhancement processing method combined with defect inspection according to claim 1, characterized in that, The defect region segmentation process based on the hierarchical differential feature map set, which determines the boundary of potential defect regions according to the response distribution of defect morphology-sensitive features in the hierarchical differential feature map, and generates a set of binary masks for candidate defect regions, includes: For each fused differential feature map in the hierarchical differential feature map set, feature response extreme point search processing is performed. All local maximum point coordinates are located in the spatial dimension of the fused differential feature map to obtain a set of candidate defect core point coordinates. Each coordinate point in the set of candidate defect core point coordinates is accompanied by its corresponding defect morphology comprehensive response intensity value. Using each candidate defect core point in the set of candidate defect core point coordinates as a region growth seed point, a continuous neighborhood pixel traversal search is performed along eight spatial directions in the fused differential feature map to determine the maximum extension distance of the comprehensive response intensity value of the defect morphology of each growth seed point along each spatial direction that is continuously higher than the average response value of the adjacent pixels. The maximum extension distance of each growth seed point in eight spatial directions is used as the growth boundary distance parameter in that spatial direction. The endpoints of the eight growth boundary distance parameters are closed by the convex hull connection algorithm to generate the initial defect region polygon boundary corresponding to the candidate defect core point. Extract the initial defect region polygon boundaries corresponding to the core points of all candidate defects, perform region merging judgment on the initial defect region polygon boundaries that overlap in spatial location, calculate the ratio of the overlapping area to the area of ​​each region, and merge regions with an overlap ratio higher than the fusion threshold into a unified defect region polygon boundary. Extract the vertex coordinate sequence of the boundary polygon on the polygon boundary of the unified defect region, perform curvature analysis on the vertex coordinate sequence of the boundary polygon, and identify the vertices with abrupt changes in boundary curvature as key points for boundary shape adjustment. The vertices with abrupt changes in boundary curvature are vertices whose boundary curvature values ​​are higher than the average curvature value of their adjacent vertices. Boundary smoothing redrawing is performed between adjacent boundary curvature abrupt change vertices. An adaptive distance weighting method is used to resample the original boundary line segments between adjacent boundary curvature abrupt change vertices to generate smoothly transitioned replacement boundary curve segments. The replacement boundary curve segments are then spliced ​​with the unadjusted boundary line segments to form the adjusted precise boundary of the defect area. The area inside the precise boundary of the defect region is filled. All pixels inside the precise boundary of the defect region are set as foreground marker values, and pixels outside the precise boundary of the defect region are set as background marker values ​​to generate a binary mask unit. The foreground area of ​​the binary mask unit corresponds to the potential defect area in the inspection image unit. Traverse all fused differential feature maps in the hierarchical differential feature map set, perform spatial position consistency verification on the binary mask unit generated by each fused differential feature map, compare the consistency of binary labels of different hierarchical fused differential feature maps at the same spatial position, merge the foreground regions with consistent labels, and generate a hierarchical unified defect candidate region binary mask. The hierarchical unified binary mask of the defect candidate region is bound to the spatial location information of the corresponding inspection image unit in the initial inspection image set to generate a binary mask entry of the defect candidate region carrying a spatial location marker. All binary mask entries of the defect candidate region carrying a spatial location marker are integrated into a set of binary masks of the defect candidate region.

4. The image data enhancement processing method combined with defect inspection according to claim 1, characterized in that, The process involves performing defect morphology migration and synthesis on the binary mask set of the defect candidate region and the inspection image units of the initial inspection image set. While maintaining the background of the inspection image units, the texture distribution of the defect candidate region is replaced with a defect texture pattern from a preset defect morphology library, generating an enhanced inspection image set containing multiple defect types. This includes: Read defect texture pattern samples of a specified defect type from a preset defect morphology library. The preset defect morphology library includes a set of crack texture pattern samples, a set of corrosion texture pattern samples, a set of deformation texture pattern samples, and a set of peeling texture pattern samples. Each defect texture pattern sample has a unique texture pattern identifier. The defective texture pattern sample is subjected to texture feature extraction processing. The defective texture pattern sample is decomposed into primitives by texture primitive analysis method. The texture primitive shape features and texture primitive spatial arrangement rules of the defective texture pattern sample are extracted, and texture primitive feature description vector and texture spatial distribution rule encoding are generated. Geometric morphological feature extraction is performed on the binary mask entries of the defect candidate region in the set of binary masks of defect candidate regions. The area, major axis direction angle and perimeter of the foreground region of the binary mask of defect candidate region are calculated and the region shape constraint parameters are generated. The region shape constraint parameters are used to guide the placement of texture primitives in the defect candidate region. Based on the region shape constraint parameters of the foreground region of the binary mask of the defect candidate region and the texture primitive feature description vector, adaptive deformation processing of texture primitive is performed. The direction of the texture primitive feature description vector is rotated and adjusted so that the direction of the texture primitive is aligned with the direction angle of the major axis of the region. The size is scaled and adjusted so that the size of the texture primitive matches the area of ​​the region, thus obtaining the adapted texture primitive. The adapted texture primitives are encoded and implanted into the foreground region marked by the binary mask of the defect candidate region of the corresponding inspection image unit in the initial inspection image set according to the texture space distribution rules using the Poisson image fusion method. During the implantation process, the background region pixel values ​​marked by the binary mask of the defect candidate region remain unchanged, and the texture implantation only acts on the foreground region pixels, resulting in a single-type defect enhanced inspection image unit. The single-type defect enhanced inspection image unit is subjected to texture transition smoothing processing. The boundary contour line between the foreground region and the background region is extracted. The pixel value gradient smoothing optimization is performed along the transition zone with a preset width range on both sides of the boundary contour line to eliminate the texture boundary jump caused by defect texture implantation and generate a single-type defect enhanced inspection image unit with coherent boundary. The texture pattern identifiers of all defect texture pattern samples in the preset defect morphology library are traversed. For each texture pattern identifier, the texture primitive adaptive deformation processing and Poisson image fusion processing are repeatedly performed to generate multiple single-type defect enhanced inspection image unit variants corresponding to different defect texture pattern identifiers for the same inspection image unit. Multiple single-type defect-enhanced inspection image unit variants generated by the same inspection image unit are combined with the original version of the inspection image unit to form a defect-enhanced image group for the inspection image unit. Each image unit in the defect-enhanced image group carries a defect texture pattern identifier and a spatial location marker of the corresponding defect candidate region binary mask. Defect enhancement image group generation processing is performed on all inspection image units. The defect enhancement image groups corresponding to all inspection image units are collected to generate an enhanced inspection image set containing multiple temporal phases and multiple defect types. The enhanced inspection image set is used to expand the defect type coverage of the training samples of the defect inspection and recognition model.

5. The image data enhancement processing method combined with defect inspection according to claim 1, characterized in that, The method further includes: The enhanced inspection image set is subjected to scene adaptive fusion processing to extract the acquisition scene features of the inspection image units in the initial inspection image set, and the synthetic defect area in the enhanced inspection image set is subjected to illumination consistency adjustment and texture naturalization processing according to the acquisition scene features. Scene lighting features are extracted from the inspection image units of the initial inspection image set. The scene lighting features include the lighting direction vector, the lighting intensity distribution, and the color temperature value. Illumination response modeling is performed on the synthetic defect region marked by the binary mask of the defect candidate region of each enhanced inspection image unit in the enhanced inspection image set. The pixel response of the synthetic defect region under different illumination conditions is calculated based on the illumination direction vector, illumination intensity distribution and color temperature value, and an illumination response prediction map of the synthetic defect region is generated. The illumination response prediction map of the synthesized defect region is subjected to illumination inversion correction processing. The difference in illumination direction consistency between the synthesized defect region and the background region of the inspection image unit is compared. The pixel values ​​in the synthesized defect region are adjusted to make the illumination direction of the synthesized defect region consistent with that of the background region, thus obtaining the illumination-consistent synthesized defect region. The illumination consistency synthesis defect area is subjected to texture naturalization processing. The texture roughness difference between the illumination consistency synthesis defect area and the surrounding background area in the texture transition zone is extracted, a texture gradient modulation signal is generated, and the texture density of the transition zone pixels is fine-tuned using the texture gradient modulation signal to generate the texture naturalization synthesis defect area. The texture naturalization synthesis defect area is replaced back into the original enhanced inspection image unit. Multi-scale feathering fusion processing is performed on the replacement boundary to generate scene-adaptive enhanced inspection image units. All scene-adaptive enhanced inspection image units constitute a scene-adaptive enhanced inspection image set. The scene-adaptive enhanced inspection image set is merged with the initial inspection image set to form a comprehensive training sample set for defect inspection. The comprehensive training sample set for defect inspection is used to train the defect inspection recognition model to improve the model's defect recognition generalization ability under different acquisition scenarios.

6. The image data enhancement processing method combined with defect inspection according to claim 1, characterized in that, The method further includes: The defect regions in the enhanced inspection image set are subjected to morphological continuity constraint processing. Based on the preset defect growth and evolution law, the morphological continuity of defect regions at the same spatial location but different acquisition time phases in the enhanced inspection image set is adjusted. From the initial inspection image set, inspection image units with the same spatial location marker but different acquisition time phases are selected to construct a temporal inspection image sequence corresponding to the spatial location. The temporal inspection image sequence is arranged in the order of acquisition time. The binary mask of the defect candidate region corresponding to each two adjacent time phases in the time phase inspection image sequence is subjected to morphological comparison processing, and the area change and boundary offset vector of the defect candidate region between adjacent time phases are extracted respectively. Using a pre-trained defect growth and evolution prediction network, the evolution trend analysis is performed based on the area change and boundary offset vector to predict the morphological transition state of the defect candidate region between adjacent time phases, and generate a defect morphological intermediate state prediction mask. The defect growth and evolution prediction network is trained under the guidance of physical constraint rules of defect morphological evolution. The intermediate state prediction mask of the defect morphology is compared with the enhanced inspection image unit of the corresponding time phase to locate the region in the enhanced inspection image unit where the defect morphology is inconsistent with the defect growth and evolution law as the morphological abnormal region. The binary mask of the defect candidate region in the morphologically abnormal region is subjected to morphological correction processing. The boundary of the morphologically abnormal region is adjusted according to the boundary shape of the defect morphological intermediate state prediction mask to conform to the defect growth and evolution law, and a morphologically continuous modified binary mask of the defect candidate region is generated. Using the binary mask of the defect candidate region after morphological continuity correction, the defect morphological migration synthesis process is re-executed to generate enhanced inspection image units that are consistent with the defect growth and evolution law. All the regenerated enhanced inspection image units are integrated into a set of morphological continuity enhanced inspection images.

7. The image data enhancement processing method combined with defect inspection according to claim 1, characterized in that, The method further includes: The initial inspection image set is subjected to local defect feature weakening and cyclic enhancement processing, and the interference of background structure in the inspection image unit on defect feature extraction is gradually reduced through iteration. From the initial inspection image set, inspection image units with background structure complexity higher than a preset threshold are selected as inspection image units to be enhanced. Background structure noise separation processing is performed on the inspection image units to be enhanced. The regular texture structure information in the inspection image units is extracted using a multi-directional strip structure filter and a background structure feature layer is generated. The periodic arrangement pattern of the background structure in space is calculated based on the background structure feature layer to generate a background structure periodic map, which records the repetition pattern of the background texture cells. Based on the background structure periodic map, background structure pattern suppression processing is performed on the image unit to be enhanced. In the frequency domain space, the frequency components corresponding to the background structure are targeted and attenuated, while the non-periodic frequency components corresponding to the defect area are retained, thus generating an enhanced intermediate image unit after background suppression. The enhanced inspection intermediate image unit after background suppression is subjected to defect edge sharpening and reconstruction processing. The gradient magnitude distribution information of the defect region in the enhanced inspection intermediate image unit is extracted. Gradient enhancement is performed on pixels with gradient magnitude lower than the edge preservation threshold to generate an inspection image unit with enhanced defect region boundary. The inspection image unit with enhanced defect region boundary is used as the input for the next round of background structure noise separation processing. The background structure noise separation processing, background structure pattern suppression processing and defect edge sharpening and reconstruction processing are performed iteratively until the iteration end condition is met, resulting in a set of inspection image units enhanced by multiple rounds of iteration. The inspection image unit set enhanced by multiple rounds of iteration is merged with the enhanced inspection image set to generate a hybrid enhanced inspection image set containing multi-morphological enhancement and multiple rounds of iteration enhancement.

8. The image data enhancement processing method combined with defect inspection according to claim 1, characterized in that, The method further includes: The overlapping regions of multiple defects in the enhanced inspection image set are separated and enhanced to solve the problem of mutual interference between the enhancement synthesis effect when multiple defect types are spatially overlapping in the inspection image unit. From the initial inspection image set, identify complex defect inspection image units with multiple overlapping defect types. Perform spatial location analysis on the overlapping areas of different defect types in the complex defect inspection image units. Determine the layering order of defect types based on the response intensity ranking of each defect response template in the overlapping area in the hierarchical differential feature map set. According to the layered order of defect type, the complex defect inspection image unit is subjected to layer-by-layer defect morphology separation processing. In each layer processing, only the binary mask of the defect candidate region corresponding to the current defect type is retained, and the pixels of the overlapping regions of other defect types are marked as regions to be filled. The background texture completion process is performed on the area to be filled. The background texture information of the surrounding non-overlapping areas is used to interpolate and fill the pixel values ​​of the area to be filled, generating a pure defect enhancement base layer that removes interference from other defect types outside the current layer. Defect morphology migration compositing is performed on the defect candidate region of the current layer corresponding to the defect type in the pure defect enhancement base layer to obtain a single-type enhancement layer of the current defect type. The single-type enhancement layer only contains the enhancement compositing effect of the current layer defect type. Obtain the single-type enhancement layer corresponding to all defect types, perform layer overlay and compositing on multiple single-type enhancement layers according to the layer order, use the binary mask of the defect candidate region of each defect type as the layer transparency control template, and perform transparency weighted overlay on the defect region and background region in each single-type enhancement layer to generate a multi-defect overlapping and separated enhanced inspection image unit. All inspection image units with enhanced separation of multiple defects are merged with the enhanced inspection image set to obtain an expanded enhanced inspection image set that includes the enhancement effect of separation of multiple defects.

9. The image data enhancement processing method combined with defect inspection according to claim 1, characterized in that, The method further includes: The initial inspection image set is subjected to cross-modal defect feature mapping enhancement processing, and the defect-sensitive characteristics in the non-visible light modal image corresponding to the spatial position of the inspection image unit are used for image enhancement. Obtain a set of non-visible light inspection images that have the same spatial location markers as the inspection image units in the initial inspection image set. The set of non-visible light inspection images includes thermal infrared inspection image units and ultrasonic scanning inspection image units. The thermal infrared inspection image unit is subjected to temperature anomaly region extraction processing. Based on the temperature difference between the pixel value in the thermal infrared inspection image unit and the surrounding area, the temperature anomaly connected domain is identified, and a thermal infrared anomaly region mask is generated. The thermal infrared anomaly region mask marks the corresponding position of possible internal defects in the visible light inspection image unit. The ultrasonic scanning inspection image unit is processed to extract the acoustic impedance change region, identify the boundary of the region where the acoustic impedance value changes abruptly in the ultrasonic scanning inspection image unit, and generate an acoustic impedance abnormal region mask. The acoustic impedance abnormal region mask marks the potential distribution range of subsurface defects in the visible light inspection image unit. The thermal infrared anomaly region mask and the acoustic impedance anomaly region mask are spatially superimposed to complete the cross-modal anomaly mask fusion process. The parts of the two anomaly region masks that overlap in spatial position are assigned high confidence mark values, and the anomaly regions that appear only in a single mode are assigned low confidence mark values ​​to generate a cross-modal defect confidence distribution map. Based on the spatial distribution of different confidence marker values ​​in the cross-modal defect confidence distribution map, defect feature enhancement processing is performed on the corresponding pixels of the high confidence marker area in the visible light inspection image unit to enhance the visual contrast between the high confidence marker area and the surrounding normal area, thereby generating a cross-modal enhanced inspection image unit. The region with enhanced defect features in the cross-modal enhanced inspection image unit is used as a defect candidate region. The binary mask set of the defect candidate region and the enhanced inspection image set are used to perform defect morphology transfer synthesis processing to generate an enhanced inspection image unit set that includes cross-modal defect feature guidance.

10. An image data enhancement and processing system combined with defect inspection, characterized in that, include: processor; A machine-readable storage medium for storing machine-executable instructions of the processor; The processor is configured to execute the image data enhancement processing method combined with defect inspection as described in any one of claims 1 to 9 by executing the machine-executable instructions.

Images