Multi-mode industrial product dynamic defect detection system based on edge calculation

Abstract

The invention relates to the technical field of visual inspection, in particular to a multi-mode industrial product dynamic defect detection system based on edge calculation. According to the method, by introducing a multi-modal image construction and enhancement mode, three-dimensional acquisition and enhanced expression of detail information of the surface of the wind power blade are realized, and by means of inter-modal image synchronization and a space registration mechanism, the consistency extraction capability of defect information under different sensor view angles is improved; through combination of density change trend analysis and direction mutation identification means, surface micro cracks, edge damages and other structural changes can be accurately identified in continuous frames, and further through multi-source discrimination and mutual elimination comparison of abnormal indexes, environmental interference and misjudgment risks are effectively eliminated, and the accuracy of the detection result is improved. Therefore, a stable defect track area with direction consistency and distribution continuity is screened out, accurate detection and partition identification of the surface defects of the wind power blade under the dynamic condition are achieved, and the reliability and precision of defect positioning are remarkably improved.

Description

Technical Field

[0001] The present invention relates to the field of visual inspection technology, and in particular to a multimodal industrial product dynamic defect detection system based on edge computing. Background Art

[0002] The field of visual inspection technology involves the use of image acquisition and image processing technology to analyze and identify target objects, including image acquisition, image preprocessing, feature extraction, target recognition and defect discrimination. It is widely used in multiple application scenarios such as industrial manufacturing, intelligent monitoring, and quality control.

[0003] The traditional industrial product dynamic defect detection system is a technology that performs real-time defect detection on products that are continuously moving on the production line during the industrial manufacturing process. This system primarily targets surface defects such as scratches, pits, cracks, stains, and chipped edges that may appear during high-speed transmission or continuous processing.

[0004] Since traditional dynamic defect detection systems for industrial products mainly rely on single-modal images for defect judgment in a state of continuous motion, they lack the fusion processing of multi-angle and multi-source information. As a result, their ability to recognize minor defects such as cracks, pits, and scratches is significantly reduced in the presence of lighting interference, image blur or complex background conditions. During the detection process, they rely on instantaneous feature judgment between static image frames, ignoring the continuity trend in the defect evolution process, which is prone to missed detection and misjudgment. For example, in the high-speed rotation detection scenario of wind turbine blades, the infrared heat diffusion effect may be misjudged as a structural defect, and areas with slight contour changes cannot be effectively extracted due to noise interference, resulting in a decrease in overall detection stability and an inability to meet the needs of large-scale, high-dynamic production scenarios for accurate defect recognition. Summary of the Invention

[0005] The purpose of the present invention is to solve the shortcomings of the prior art and propose a multimodal industrial product dynamic defect detection system based on edge computing.

[0006] To achieve the above objectives, the present invention adopts the following technical solution: A multimodal industrial product dynamic defect detection system based on edge computing includes: Image construction module: obtains a multimodal image set of wind turbine blades, performs distributed image enhancement and time series calibration operations on the multimodal image set, and uploads it to the cloud node to construct a synchronized image set; Edge feature registration module: analyzing the edge structure information of each modal image of the wind turbine blade in the synchronous image set, and extracting a set of spatial alignment feature points of the wind turbine blade; Regional density analysis module: analyzes the regional density change trend of each feature point in the spatial alignment feature point set in the wind turbine blade detection area and marks abnormal density areas; Multi-source anomaly discrimination module: constructs a multi-source anomaly index set of the wind turbine blade according to the anomaly density area, and performs interference cancellation comparison processing on the multi-source anomaly index set to form an interference cancellation point set; Defect screening module: screening the dynamic defect area of ​​the wind turbine blade according to the interference cancellation point set to obtain the wind turbine blade defect detection result.

[0007] As a further solution of the present invention, the synchronized image set includes a modal synchronization label, a time alignment index, and an image frame sequence number; the spatial alignment feature point set includes an inter-modal correspondence, a spatial repositioning parameter, and a structural key point identifier; the abnormal density area includes a density change mark, an abnormal point spatial distribution, and a regional mutation position; the multi-source abnormal indicator set includes an edge strength parameter, a thermal diffusion parameter, and a contour continuity parameter; and the dynamic defect area includes a defect position coordinate, a defect area number, and a detection result mark.

[0008] As a further solution of the present invention, the image construction module includes: Image acquisition submodule: This module acquires visible light image data, infrared thermal image data, and laser profile image data collected by edge devices at the wind turbine site. It determines the image frame sequence based on the sensor trigger information recorded at the time of image acquisition, establishes image frame channel division based on the image source device number and data inflow sequence, and generates a multimodal image index sequence. Image enhancement submodule: Based on the multimodal image index sequence, performs brightness contrast equalization on visible light image data, performs regional grayscale contrast expansion on infrared thermal image data, and performs edge detail sharpening on laser profile image data to establish a multi-channel image enhancement set; Synchronous calibration submodule: Call the multi-channel image enhancement set, combine the image trigger cycle data recorded by the edge device to calculate the time deviation between image frames, filter the time error between image frames according to the set image synchronization threshold, reconstruct the cross-channel image frame group structure based on the screening results, and obtain a synchronized image set.

[0009] As a further solution of the present invention, the edge feature registration module includes: Edge extraction submodule: Analyzes the edge structure of the wind turbine blade visible light image, infrared thermal image and laser contour image in the synchronized image set, extracts the intersection position of the grayscale gradient and texture in the image, and establishes a modal boundary feature distribution map; Key point matching submodule: Based on the edge point positions of each image frame of the wind turbine blade in the modal boundary feature distribution map, the directional main feature points in the adjacent areas between each mode are identified, and pairing is performed according to the response intensity and direction difference to establish a modal key point correspondence set; Spatial transformation submodule: calls the key point group in the corresponding set of the modal key points, constructs the image coordinate mapping relationship, unifies the image spatial position and corrects the offset error, and obtains the spatial alignment feature point set.

[0010] As a further solution of the present invention, the regional density analysis module includes: Density calculation submodule: obtains the number of adjacent points of each feature point in each frame of the image within a specified neighborhood according to the coordinate distribution of the spatially aligned feature point set in the wind turbine blade detection area, and extracts the local density trend value based on the distribution position of all points in the neighborhood in the image space; Gradient identification submodule: Based on the local density trend, calculate the cosine value of the angle between the density direction vectors of adjacent frames, identify the area with prominent direction changes, extract the coordinate range of the target area, and obtain the direction mutation interval; Abnormal marking submodule: According to the angle difference and spatial position distribution of each area in the direction mutation interval, continuous areas with density direction reversal angles exceeding 45 degrees are screened and marked with numbers to obtain abnormal density areas.

[0011] As a further solution of the present invention, the multi-source anomaly discrimination module includes: Index extraction submodule: Based on the spatial position of the abnormal density area in the wind turbine blade multimodal image, the edge intensity change rate in the visible light image, the heat diffusion direction vector in the infrared thermal image, and the contour discontinuity coefficient in the laser contour image are obtained to establish a multi-source abnormality index set; Direction comparison submodule: calls the heat diffusion direction vector and edge intensity change rate assigned in the multi-source anomaly indicator set, performs directional polarity comparison on the modal indicators under each area number, determines whether a direction reversal relationship is formed, and obtains a direction reversal number set; Interference screening submodule: Based on the direction reversal number set and the contour discontinuity coefficient in the multi-source anomaly indicator set, the values ​​of the matching area are jointly screened to identify the point numbers that meet the direction reversal and abnormal boundary changes, and establish an interference cancellation point set.

[0012] As a further solution of the present invention, the defect screening module includes: Trajectory judgment submodule: extracts the density change trend and extension direction fluctuation range of each feature point in the interference cancellation point set in the image frame sequence, identifies points with both density fluctuation and direction stability, and obtains a trajectory stable feature point group; Region construction submodule: calls the position coordinates of the trajectory stable feature point group, identifies the mutually adjacent point sets in the single frame image, divides the region boundaries according to the distance relationship between the points, and obtains the connected region clustering results; Result output submodule: Based on the boundary morphology and area distribution in the connected region clustering results, perform integrity judgment on the regional structure, screen the areas with continuous edges and concentrated distribution, and establish the wind turbine blade defect detection results.

[0013] Compared with the prior art, the advantages and positive effects of the present invention are: In the present invention, by introducing a multimodal image construction and enhancement method, the three-dimensional acquisition and enhanced expression of the surface detail information of the wind turbine blade is achieved. With the help of inter-modal image synchronization and spatial registration mechanism, the consistency extraction capability of defect information under different sensor perspectives is improved. Combined with density change trend analysis and directional mutation identification means, structural changes such as surface microcracks and edge damage can be accurately identified in continuous frames. Further, through multi-source discrimination and mutual cancellation comparison of abnormal indicators, environmental interference and misjudgment risks are effectively eliminated, thereby screening out stable defect trajectory areas with directional consistency and distribution continuity, realizing accurate detection and zoning identification of wind turbine blade surface defects under dynamic conditions, significantly improving the reliability and accuracy of defect positioning, and enhancing the adaptability and practicality of the system under complex working conditions. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1 is a system flow chart of the present invention; Figure 2 A flowchart of the image construction module of the present invention; Figure 3 This is a flow chart of the edge feature registration module of the present invention; Figure 4 This is a flow chart of the regional density analysis module of the present invention; Figure 5 This is a flow chart of the multi-source anomaly discrimination module of the present invention; Figure 6 This is a flow chart of the defect screening module of the present invention. DETAILED DESCRIPTION

[0015] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0016] In the description of the present invention, it should be understood that the terms "length," "width," "up," "down," "front," "back," "left," "right," "vertical," "horizontal," "top," "bottom," "inside," "outside," and the like, indicating positions or relationships, are based on the positions or relationships shown in the accompanying drawings and are intended only to facilitate the description of the present invention and simplify the description. They do not indicate or imply that the devices or elements referred to must have a specific orientation, be constructed, or operate in a specific orientation. Therefore, they should not be construed as limiting the present invention. Furthermore, in the description of the present invention, "plurality" means two or more, unless otherwise expressly and specifically defined.

[0017] See also Figure 1 , a multimodal industrial product dynamic defect detection system based on edge computing, the system includes: Image construction module: obtains a multimodal image set of wind turbine blades, performs distributed image enhancement and time series calibration operations on the multimodal image set, and uploads it to the cloud node to construct a synchronized image set; Edge feature registration module: Analyzes the edge structure information of each modal image of the wind turbine blade in the synchronized image set and extracts the spatial alignment feature point set of the wind turbine blade; Regional density analysis module: Analyzes the regional density change trend of each feature point in the set of spatially aligned feature points in the wind turbine blade detection area and marks abnormal density areas; Multi-source anomaly discrimination module: Constructs a multi-source anomaly indicator set for wind turbine blades based on the anomaly density area, and performs interference cancellation comparison processing on the multi-source anomaly indicator set to form an interference cancellation point set; Defect screening module: Based on the interference cancellation point set, the dynamic defect area of ​​the wind turbine blade is screened to obtain the wind turbine blade defect detection results; The synchronized image set includes modal synchronization labels, time alignment indexes, and image frame sequence numbers; the spatial alignment feature point set includes inter-modal correspondences, spatial repositioning parameters, and structural key point identifiers; the abnormal density area includes density change marks, abnormal point spatial distribution, and regional mutation positions; the multi-source abnormal indicator set includes edge strength parameters, thermal diffusion parameters, and contour continuity parameters; the dynamic defect area includes defect position coordinates, defect area numbers, and detection result marks.

[0018] See also Figure 2 , the image building blocks include: Image acquisition submodule: This module acquires visible light image data, infrared thermal image data, and laser profile image data collected by edge devices at the wind turbine site. It determines the image frame sequence based on the sensor trigger information recorded at the time of image acquisition, establishes image frame channel division based on the image source device number and data inflow sequence, and generates a multimodal image index sequence. To obtain visible light image data, infrared thermal image data, and laser profile image data collected by the edge device on site of the wind turbine, it is necessary to access the data channel of the edge device one by one. First, identification is performed based on the device number. Each number corresponds to a unique image sensor. For example, the device numbered WT001 of a wind turbine corresponds to a visible light camera. The images collected through this channel are accurately marked with a timestamp and the image frames are sorted according to the trigger signal recorded by the edge control system. For example, if 10 frames are triggered per second, the interval between each frame is 0.1 seconds, and the corresponding trigger timestamps are T, T+0.1, T+0.2, etc.; Next, the system will group the images according to the device number, such as putting the WT001 channel image into channel A, the WT002 infrared image into channel B, and the WT003 The laser contour image is placed in channel C, and the data of each channel is cached according to the FIFO (first-in-first-out) rule to establish a record index of the image inflow order. For example, the record is: the first frame of channel A is T0, the first frame of channel B is T1, and the first frame of channel C is T2. And so on. On this basis, a multimodal index sequence is constructed between the three types of image data, such as the index linked list [A: T0, B: T1, C: T2], which is used for subsequent image enhancement and synchronous processing. Taking the operation and maintenance inspection of wind turbines as an example, the surface image of the blade is obtained by the deployed visible light camera, the infrared camera records the surface temperature field change of the equipment, and the laser contour sensor records the contour deformation or crack path. These data are used to complete the construction of the preliminary multimodal alignment index structure through the above acquisition process.

[0019] Image enhancement submodule: Based on the multimodal image index sequence, it performs brightness contrast equalization on visible light image data, regional grayscale contrast expansion on infrared thermal image data, and edge detail sharpening on laser profile image data to establish a multi-channel image enhancement set; According to the multimodal image index sequence, the image data is processed with brightness contrast equalization, grayscale contrast expansion and edge detail sharpening. First, for the visible light image data, the image pixel matrix is ​​read and its grayscale value distribution is counted. Assuming the image size is 1920×1080, the total number of pixels is If the grayscale values ​​of the image are concentrated in the range of 70-120, histogram equalization needs to be performed to improve the grayscale distribution range. The specific operations include: first, counting each grayscale value (in ) , calculate the probability distribution function : ; in, represents the gray level, represents the number of pixels at that gray level, and Represents the image width and height, such as a certain grayscale , the corresponding number of pixels is , then its probability distribution is: ; Then calculate the cumulative distribution function (CDF) : ; by For example, suppose from arrive The cumulative probability of , then calculate the balanced gray value after mapping according to the cumulative probability : ; in is the total number of gray levels, Indicates the rounding down operation, substituting the data into the result: ; That is, the original grayscale value of 80 will be mapped to the enhanced grayscale value of 89. This process completes the mapping for all pixels to form an enhanced image data set. Next, the regional grayscale contrast expansion processing is performed on the infrared thermal image. Assume that the pixel grayscale range in the infrared image is 50–100, corresponding to the actual temperature range of 30°C–60°C. In order to enhance the contrast, the grayscale range needs to be linearly stretched to [0, 255] using the linear transformation function: ; in is the original grayscale value, , ,by For example, substitute the calculation into: ; After rounding to 128, the grayscale value of 75 is mapped to the enhanced grayscale value of 128, making it easier to distinguish different temperature areas in the image. If you need to set a grayscale threshold to determine the area with abnormal temperature rise, you can set the temperature reference value to 40°C, and the corresponding grayscale is: ; The value after mapping is: ; However, if the value exceeds 255, it needs to be clipped to 255, indicating that the high temperature area has reached the thermal threshold. Next, the edge detail sharpening operation is performed on the laser contour image, and the gradient amplitude is calculated using the Sobel operator. The horizontal gradient kernel is used respectively. With vertical gradient kernel : ; For each pixel The convolution calculation is performed on its 3×3 neighborhood, and the horizontal gradient value obtained by the design is , the vertical gradient value is , then the total gradient amplitude is: ; This value is used as the pixel brightness to generate the sharpened image, and non-maximum suppression and double threshold connection are further performed to retain the main contour area and exclude weak boundary noise, ultimately forming a three-channel image enhancement set for subsequent synchronous calibration processing.

[0020] Synchronous calibration submodule: This module calls the multi-channel image enhancement set and calculates the time deviation between image frames based on the image trigger cycle data recorded by the edge device. It then filters the time error between image frames according to the set image synchronization threshold. Based on the screening results, it reconstructs the cross-channel image frame group structure and obtains a synchronized image set. In the multi-channel image set after image enhancement, the time difference calculation is performed in combination with the image trigger cycle data recorded by the edge device. Assume that the image acquisition cycle is 0.1 seconds, that is, 10 frames of images are collected per second. The devices are channel A (visible light image), channel B (infrared thermal image) and channel C (laser contour image). First, the timestamp sequence corresponding to the image frame in each channel is extracted. For example, the image timestamp of channel A is 、 、 , channel B is 、 、 , channel C is 、 、 , where the symbol Indicates the Frame image in the aisle( ) in seconds. Next, we calculate the image frame synchronization error, using the difference between the maximum and minimum timestamps as the inter-frame time difference indicator. The calculation formula is as follows: ; in Indicates the The maximum time difference between the image acquisition timestamps of the frame group in the three channels. If the timestamp corresponding to frame group 1 is 、 、 ,but: ; Similarly, in frame group 2 、 、 ,but: ; To determine whether the frame group meets the synchronization requirements, it is necessary to synchronize with the set image threshold The comparison criteria are: ,in is the image synchronization threshold, which is set based on the following: considering the maximum clock synchronization error of the edge device is , the image acquisition buffer error is , then the synchronization threshold can be set as: ; Therefore, when If , it is considered as a synchronous frame group, otherwise it is removed. For example, if both frame group 1 and frame group 2 meet this condition, they are retained and enter the synchronous image set. If frame group 3 is 、 、 ,but: ; The threshold condition is also met and the image frames enter the set. Next, the system reconstructs all image frames that meet the synchronization condition into a synchronized image frame group structure. Each group structure is represented by a triple: , forming a frame group index list: frame group 1 is , frame group 2 is , frame group 3 is , all frame groups satisfy , forming the final synchronized image set. Taking the wind power inspection site as an example, if the channel A image shows that the blade is partially covered with foreign matter, the channel B image shows an abnormal temperature increase in the corresponding area, and the channel C image shows an irregular contour in the area, synchronous calibration can be used to confirm that the three modes are collected at the same time point, effectively establishing a basic data set for cross-channel consistency analysis.

[0021] See also Figure 3 , the edge feature registration module includes: Edge extraction submodule: Analyzes the edge structure of visible light images, infrared thermal images, and laser contour images of wind turbine blades in the synchronized image collection, extracts the intersection position of grayscale gradient and texture in the image, and establishes a modal boundary feature distribution map; To analyze the edge structure of the visible light image, infrared thermal image and laser contour image of the wind turbine blade in the synchronous image set, it is necessary to first read the image matrix data frame by frame, obtain the RGB channel or grayscale channel image respectively, and select different edge analysis methods according to the image type. For example, for visible light images, the grayscale gradient detection method is used to extract the edge of the image brightness change area. For infrared thermal images, the boundary is extracted by the thermal gradient of the temperature field. For laser contour images, the structural curvature change is used as the basis for edge recognition. The edge structures of the three modal images are stored in the structure data list respectively. In this process, the texture intersection area in the image needs to be extracted. This step The first step is to obtain the concentrated points of texture change frequency by scanning the image area. For example, at the transition position of the blade root structure, the grayscale change is concentrated and the texture edges intersect, so it is easy to extract the significant intersection line. By performing a sliding window scan on the brightness or thermal value change amplitude of each pixel neighborhood area in these images, the area with grayscale mutation in the image is identified, and the dense intersection position of texture features is extracted in the range around the mutation point to complete the spatial positioning of the basic edge features in the multimodal image. Combined with the image pixel coordinate mapping structure, the edge points and texture intersection points are stored in the data index table with the image frame number. By comparing the boundary information of the same area in the modal image, the difference position is recorded and the modal boundary feature distribution map is established.

[0022] Key point matching submodule: Based on the edge point positions of each image frame of the wind turbine blade in the modal boundary feature distribution map, the directional main feature points in the adjacent areas between each mode are identified, and paired according to the response intensity and direction difference to establish a corresponding set of modal key points; According to the edge point position of each image frame of the wind turbine blade in the modal boundary feature distribution map, the directional main feature points in the adjacent areas between each mode are identified, and the pairing operation is performed according to the response intensity and direction difference. First, the edge point position of each frame image is extracted from the modal boundary feature distribution map, and the edge linear or corner point features are recorded according to the coordinate and neighborhood comparison structure. When performing directional main feature extraction, multi-scale angle direction detection is used for each edge point, and the main direction indicator label is established at the position where the angle change is most prominent. The edge direction projection analysis is performed in combination with the modal information to screen out the points that show directional consistency or continuity in the image area. A feature point set with a rotation trend is used to identify feature points in adjacent areas of each modal image that form structural symmetry or similar positions. A preliminary matching pool is established, and then the response strength of each candidate key point is calculated. The main feature point set within a certain threshold is selected by sorting the response strength. For example, the lower limit of the response strength is set to 200, and only high-response key points are retained. The direction difference between the key points is calculated, and their structural consistency is evaluated by the angle between the direction vectors. If the direction difference is less than the set angle of 10 degrees, it is determined to be a paired key point. Finally, a corresponding set of mutually related modal key points is established between the three modal images.

[0023] Spatial transformation submodule: calls the key point group in the modal key point correspondence set, constructs the image coordinate mapping relationship, unifies the image spatial position and corrects the offset error, and obtains the spatial alignment feature point set; The keypoint group in the modal keypoint correspondence set is called, and the image coordinate mapping relationship is constructed. The image spatial position is unified and the offset error is corrected to obtain the spatially aligned feature point set. First, the coordinate values ​​of the keypoints in the three modal images contained in the modal keypoint correspondence set are read. The keypoint coordinates under channels A, B, and C are uniformly loaded according to each frame number. The reference modal image is selected according to the image geometry. For example, with channel A image as the reference, the relative displacement of its corresponding keypoints and the keypoints in channels B and C images is calculated. By comparing the image coordinate difference, it is determined whether the offset type is translation, rotation, or scaling. If the offset is mainly linear translation, a two-dimensional translation matrix is ​​constructed for position correction. If rotation and scaling are both present, an affine transformation matrix is ​​constructed between the coordinate points. The keypoints in the different modal images are unified to the reference coordinate system through the mapping matrix. The calculated error result of the unified coordinate values ​​is then compared to see if it is below the set tolerance threshold. For example, the spatial offset tolerance is set to 2 pixels. If the distance between the mapped keypoints is less than this value, the match is accepted. Otherwise, local refinement is performed. After processing all frame groups, a spatially aligned feature point set under unified spatial coordinates is formed.

[0024] See also Figure 4 , the regional density analysis module includes: Density calculation submodule: Based on the coordinate distribution of the spatially aligned feature point set in the wind turbine blade detection area, the number of adjacent points of each feature point in each frame of the image within the specified neighborhood is obtained, and the local density trend is extracted based on the distribution position of all points in the neighborhood in the image space; According to the coordinate distribution of the spatially aligned feature point set in the wind turbine blade detection area, all the spatially aligned feature points in each frame image are first read as a two-dimensional coordinate sequence. The image resolution is set to 1920×1080, and a grid coordinate system is established on the image plane. The neighborhood search radius is defined as 25 pixels. A neighborhood search window is constructed at the center of each feature point, and the number of other feature points in the window is counted as its number of adjacent points. If a feature point detects a total of 12 other feature points within its neighborhood radius, then its local number of adjacent points is 12. This process is repeated to obtain the number of adjacent points of all feature points in each frame image. In this way, the density value of each point is defined as the number of adjacent points, and based on all The coordinate position and density value of the feature points are used to draw a two-dimensional density distribution map in the image space. By dividing the image space into several sub-areas, the average density value of the points contained in each area is obtained, and then a color gradient encoding is used for visualization to form a spatial trend map of density changes with position. The density trend is described as the local density trend quantity, which comes from the degree of clustering of feature points in space. If the number of adjacencies of multiple points in a certain area exceeds 15, it can be considered that the local density of the area is high. If the number of adjacencies in a certain area is less than 5, it is classified as a low-density area. After completing the feature point adjacency analysis of all frame images, the system saves the two-dimensional spatial density trend corresponding to each frame in a list structure.

[0025] Gradient recognition submodule: Based on the local density trend, it calculates the cosine value of the angle between the density direction vectors of adjacent frames, identifies the area with prominent direction changes, extracts the coordinate range of the target area, and obtains the direction mutation interval; Based on the local density trend, each frame image is first divided into multiple density segment regions. The density center of the points in the region is calculated by the weighted average of the coordinates of all feature points. A certain area A in the frame image contains feature points The coordinates of each feature point are , the density value is , then the density center coordinates of the area Expressed as: ; in, Indicates the The density center coordinates of area A in the frame, For the The horizontal and vertical coordinates of the feature points in the image coordinate system, The number of adjacent points of the feature point in the local neighborhood is used as its density value. The density center of each region is obtained based on the point set, and then the change direction of the regional density center between consecutive frames is calculated. The definition of Frame and The displacement vector of the density center between frames is: ; in, For the The horizontal and vertical coordinates of the density center of frame area A, For the The frame corresponds to the coordinate value, and the resulting vector Represents the directional trend of density change, and then the magnitude of the directional change is determined by calculating the cosine value of the angle between two adjacent vectors. The calculation formula is: ; in: Represents the dot product of two direction vectors, calculated as ; is a vector The modulus length, that is ; is the angle between the density direction vectors of the two frames.

[0026] Let the density center point of the third frame in region A be , the 4th frame is , the 5th frame is ,but: , .

[0027] Compute the dot product: ; Calculate the modulus length: ; ; The cosine value is: ; The angle of direction change is: .

[0028] Set the density center of the 6th frame again ,but: ; Dot product: ; ; The cosine value is calculated as: ; Angle: .

[0029] This indicates that the direction is almost completely reversed, so the area is identified as a mutation area. The system records the direction changes of the area between frames 3, 4, and 5, and converts the coordinate points Add a list of directional mutation intervals for subsequent anomaly detection.

[0030] Abnormal marking submodule: Based on the angle difference and spatial position distribution of each area in the direction mutation interval, continuous areas with density direction reversal angles exceeding 45 degrees are screened and numbered to obtain abnormal density areas; Based on the angular difference and spatial position distribution of each region in the directional mutation interval, the angular difference sequence of each region between consecutive frames is first read from the marked directional mutation interval list and archived by region number. All records with an angle difference greater than 45 degrees are regarded as candidate inversion regions. The analysis span is set to three frames. If a region has continuous large angular mutations in three frames, that is, the direction change between each adjacent frame is greater than 45 degrees, the system marks the region as a continuous density direction inversion region. For each candidate region, its coordinate coverage in the image plane is further calculated, and regions with a spatial span smaller than a certain pixel area are eliminated. For example, the minimum region boundary is set to 50×50 pixels, and only image regions with actual scale are retained. Then, the filtered regions are numbered one by one according to the processing order. The numbering method is recorded by combining the image frame number and the region sequence number to form a unique identification mark. At the same time, a correspondence between the region number and the image frame position is established, and the marking information is written into the abnormal region index structure. The system finally outputs the numbers, corresponding frame ranges and spatial coverage coordinates of all abnormal density regions as output results.

[0031] See also Figure 5 ,The multi-source anomaly discrimination module includes: Index extraction submodule: Based on the spatial location of the abnormal density area in the wind turbine blade multimodal image, the edge intensity change rate in the visible light image, the heat diffusion direction vector in the infrared thermal image, and the contour discontinuity coefficient in the laser contour image are obtained to establish a multi-source abnormality index set; According to the spatial position of the abnormal density area in the multimodal image of the wind turbine blade, the physical structure or thermal feature changes of the three types of images in the corresponding area are first extracted respectively, and a multi-source abnormal index set is established. For the edge intensity change rate in the visible light image, the system selects the grayscale gradient value of each pixel point on the specified boundary segment after extracting the edge of the area and records it as , indicating the The grayscale intensity of the sampling points, the sequence contains sampling points, i.e. , average the grayscale differences between these sampling points and define the edge intensity change rate as: ; in, Indicates the rate of change of edge intensity, in grayscale value. is the total number of sampled pixels, For the The gray value of a pixel, Indicates the gray value of the next adjacent pixel, and the absolute value item indicates the gray mutation amplitude. Assuming that the total gray value is extracted on the edge of an abnormal area The grayscale value sequence of pixels is: [30, 35, 40, 80, 85, 90, 60, 55, 50, 45]. The adjacent differences are calculated as follows: , , , , , , , , .

[0032] The sum of these differences is 105, which we plug into the formula: .

[0033] This value represents the degree of grayscale fluctuation between pixels. A sharp change indicates a significant edge mutation. Next, we process the heat diffusion direction vector in the infrared thermal image. First, we obtain the temperature values ​​of the center point of the region and its upper, lower, left, and right neighboring pixels. Let the center pixel temperature be , the temperatures of the upper, lower, left and right neighboring points are , construct the direction vector as follows: ; in, Represents the heat diffusion direction vector, the first component represents the horizontal temperature difference, and the second component represents the vertical temperature difference. Suppose in a certain area: , , , , , then the vector is , indicating that heat diffuses to the lower right direction, and then the vector modulus is calculated: ; the unit direction vector is approximately , as the directional description of the thermal characteristics of the area, and finally processing the contour discontinuity coefficient in the laser contour image, the system obtains a set of equally spaced sampling points along the scan line after edge extraction, and assumes that the total sampling contour points, the coordinates of each point are ,in is the horizontal pixel position of the point, is the vertical height value of the contour. Contour jumps are identified by calculating the height difference between two adjacent points, which is defined as follows: ; in, Indicates the With the The height variation between the contour points is greater than the set threshold. , which is recorded as a jump event, and the number of all eligible jump point pairs is counted as , contour discontinuity coefficient for: ; Suppose a certain area is sampled points, the analysis shows that the height difference of 7 adjacent point pairs exceeds the set threshold Pixels, then: This value indicates that the regional contour continuity is poor and there is a risk of structural mutation. This coefficient, together with the edge intensity change rate and the heat diffusion direction vector, constitutes the three-modal anomaly indicator set of the region. The system organizes data by region number and stores it in the multi-source anomaly indicator set for subsequent direction comparison and interference screening.

[0034] Direction comparison submodule: Calls the allocated heat diffusion direction vector and edge intensity change rate in the multi-source anomaly indicator set, performs directional polarity comparison on the modal indicators under each region number, determines whether a direction reversal relationship exists, and obtains a set of direction reversal numbers; The allocated heat diffusion direction vector and edge intensity change rate in the multi-source anomaly indicator set are called to perform directional polarity comparison on the modal indicators under each area number. During the execution process, the system first extracts the directional attribute of the heat diffusion direction vector of each numbered area, converts it into an angle representation, and converts the edge direction corresponding to the edge intensity change rate in the visible light image in the same way. The polarity relationship is determined by calculating the directional angle difference between the two. If the angle difference is close to 180 degrees, it means that the two modal directions are opposite, and it is recorded as a direction reversal area. The system repeats the above operation for each numbered area, and adds the area number determined to be direction reversal to the reversal number set. At the same time, the areas with significant directional angles in the reversal relationship are weighted and recorded in the set according to the number.

[0035] Interference elimination submodule: Based on the direction reversal number set and the contour discontinuity coefficient in the multi-source anomaly indicator set, the values ​​of the matching area are jointly screened to identify the point numbers that meet the direction reversal and abnormal boundary changes, and establish the interference mutual cancellation point set; According to the direction reversal number set and the contour discontinuity coefficient in the multi-source anomaly indicator set, the system extracts the corresponding contour discontinuity value of all area numbers marked as direction reversal one by one, compares it with the set deformation anomaly threshold, and determines whether the area meets the contour structure jump condition at the same time. If the contour discontinuity coefficient of a certain area exceeds the set benchmark and the direction polarity is reversed, the area number is judged to be an interference anomaly. The system marks the point number as an interference cancellation point, and excludes other numbered areas that do not meet the dual conditions. All point numbers that meet the joint screening conditions are uniformly recorded as an interference cancellation point set.

[0036] See also Figure 6 , the defect screening module includes: Trajectory judgment submodule: extracts the density change trend and extension direction fluctuation range of each feature point in the interference cancellation point set in the image frame sequence, identifies points with both density fluctuation and direction stability, and obtains a trajectory stable feature point group; The density change trend and extension direction fluctuation range of each feature point in the interference cancellation point set in the image frame sequence are extracted. First, a trajectory restoration operation is performed on the position of each point in different time frames. The local density value of the area where the point is located in consecutive frames is counted. The fluctuation range of the density value is used to determine the change amplitude of the point in the time series. Points with a density fluctuation range of medium or below are marked as stable density points. At the same time, the direction vector of the point and the points in the previous and next frames are extracted in each frame. The angle between the direction change and the direction fluctuation range of consecutive frames is calculated. If the direction change of the point in three or more consecutive frames is less than the set fluctuation baseline, it is marked as a directionally stable point. The system performs a joint screening judgment on the density change trend and direction fluctuation range features, sets the density change upper limit and the direction fluctuation threshold, and determines whether the two conditions match. For example, if the density change does not exceed 20% of the global mean and the direction change angle does not exceed 15 degrees, the point is considered to have both stable density and stable direction. After the system screens all points that meet these criteria, it forms a trajectory stable feature point group.

[0037] Region construction submodule: calls the position coordinates of the trajectory stable feature point group, identifies the adjacent point sets in a single frame image, divides the region boundaries according to the distance relationship between the points, and obtains the connected region clustering results; The position coordinates of the trajectory stable feature point group are called to identify the set of adjacent points in a single frame image. First, the coordinates of the trajectory stable feature points in each frame image are spatially clustered, and the adjacency relationship within the region is used to determine whether two points belong to the same structural block. Specifically, the judgment is made through the distance between the points and the number of neighborhood intersections. The distance threshold and the lower limit of the number of neighboring point connections are set. The distance between any two points is judged. If it is less than the set spatial threshold and they are reachable to each other in the spatial adjacency graph, they are grouped together. The system traverses all point pairs to construct an adjacency graph structure, and then performs a region expansion operation on the basis of this structure. The boundary contours of the connected components that form a closed graph are extracted, and a preliminary description is made based on the minimum bounding box area and shape ratio surrounded by the contour boundary. Finally, the set of clustering regions that meet the adjacency and closure conditions is output to form a connected region clustering result.

[0038] Result output submodule: Based on the boundary shape and area distribution in the connected region clustering results, the integrity of the regional structure is judged, and regions with continuous edges and concentrated distribution are screened to establish wind turbine blade defect detection results; Based on the boundary shape and area distribution in the connected region clustering results, the integrity judgment of the regional structure is performed. The system extracts contour segments from the cluster region boundary and calculates the continuity index of the contour line. By statistically analyzing the angle change between each two consecutive contour points on the boundary, it is judged whether there are breaks, sharp turns or reverse structures. The boundary continuity score is calculated in combination with the degree of contour closure, and regions with scores lower than the set standard are eliminated. The area and shape distribution characteristics of the region are further analyzed, and the area, perimeter, maximum side length ratio, length-to-width ratio and other indicators of each region are statistically analyzed to determine whether it belongs to a concentrated regional structure. If the area accounts for more than a certain proportion of the entire map and the shape tends to be a regular closed structure, the region passes the structural integrity judgment. The system numbers and records the regions that meet the edge continuity and concentrated shape, and marks them as wind turbine blade defect detection results. There may be various problems in the actual application of wind turbine blade defect detection results, mainly including false positive problems caused by edge misrecognition, that is, edge mutations caused by image noise, stains or surface coating changes are mistakenly judged as defects. In addition, in the multimodal image fusion process, spatial alignment errors will cause misalignment of different modal features, which in turn leads to defect positioning offset or loss. At the same time, for slight cracks, early bubbles or delamination micro-defects on the surface of wind turbine blades, due to low signal intensity or weak texture, the system is prone to missed detection. In addition, in the defect area clustering process, some fuzzy boundary areas may be divided into multiple discontinuous blocks by the system, resulting in incomplete defect boundaries. Finally, under complex backgrounds or multiple occlusion conditions, the density change trend of feature points is not obvious, which will also reduce the robustness and accuracy of system recognition.

[0039] The above are merely preferred embodiments of the present invention and do not limit the present invention in any other form. Any technician familiar with the profession may use the technical content disclosed above to change or modify it into an equivalent embodiment with equivalent changes and apply it to other fields. However, any simple modification, equivalent change and modification made to the above embodiment based on the technical essence of the present invention without departing from the content of the technical solution of the present invention shall still fall within the scope of protection of the technical solution of the present invention.

Claims

1. A multimodal industrial product dynamic defect detection system based on edge computing, characterized by: The system comprises: Image construction module: obtains a multimodal image set of wind turbine blades, performs distributed image enhancement and time series calibration operations on the multimodal image set, and uploads it to the cloud node to construct a synchronized image set; Edge feature registration module: analyzing the edge structure information of each modal image of the wind turbine blade in the synchronous image set, and extracting a set of spatial alignment feature points of the wind turbine blade; Regional density analysis module: analyzes the regional density change trend of each feature point in the spatial alignment feature point set in the wind turbine blade detection area and marks abnormal density areas; Multi-source anomaly discrimination module: constructs a multi-source anomaly index set of the wind turbine blade according to the anomaly density area, and performs interference cancellation comparison processing on the multi-source anomaly index set to form an interference cancellation point set; Defect screening module: screening the dynamic defect area of ​​the wind turbine blade according to the interference cancellation point set to obtain the wind turbine blade defect detection result.

2. The multimodal industrial product dynamic defect detection system based on edge computing according to claim 1 is characterized in that: The synchronized image set includes a modal synchronization tag, a time alignment index, and an image frame sequence number; the spatial alignment feature point set includes inter-modal correspondence, spatial repositioning parameters, and structural key point identifiers; the abnormal density area includes a density change mark, abnormal point spatial distribution, and regional mutation position; the multi-source abnormal indicator set includes an edge strength parameter, a thermal diffusion parameter, and a contour continuity parameter; and the dynamic defect area includes defect position coordinates, defect area number, and detection result mark.

3. The multimodal industrial product dynamic defect detection system based on edge computing according to claim 1 is characterized in that: The image construction module includes: Image acquisition submodule: This module acquires visible light image data, infrared thermal image data, and laser profile image data collected by edge devices at the wind turbine site. It determines the image frame sequence based on the sensor trigger information recorded at the time of image acquisition, establishes image frame channel division based on the image source device number and data inflow sequence, and generates a multimodal image index sequence. Image enhancement submodule: Based on the multimodal image index sequence, performs brightness contrast equalization on visible light image data, performs regional grayscale contrast expansion on infrared thermal image data, and performs edge detail sharpening on laser profile image data to establish a multi-channel image enhancement set; Synchronous calibration submodule: Call the multi-channel image enhancement set, combine the image trigger cycle data recorded by the edge device to calculate the time deviation between image frames, filter the time error between image frames according to the set image synchronization threshold, reconstruct the cross-channel image frame group structure based on the screening results, and obtain a synchronized image set.

4. The multimodal industrial product dynamic defect detection system based on edge computing according to claim 3 is characterized in that: The edge feature registration module includes: Edge extraction submodule: Analyzes the edge structure of the wind turbine blade visible light image, infrared thermal image and laser contour image in the synchronized image set, extracts the intersection position of the grayscale gradient and texture in the image, and establishes a modal boundary feature distribution map; Key point matching submodule: Based on the edge point positions of each image frame of the wind turbine blade in the modal boundary feature distribution map, the directional main feature points in the adjacent areas between each mode are identified, and pairing is performed according to the response intensity and direction difference to establish a modal key point correspondence set; Spatial transformation submodule: calls the key point group in the corresponding set of the modal key points, constructs the image coordinate mapping relationship, unifies the image spatial position and corrects the offset error, and obtains the spatial alignment feature point set.

5. The multimodal industrial product dynamic defect detection system based on edge computing according to claim 4 is characterized in that: The regional density analysis module includes: Density calculation submodule: obtains the number of adjacent points of each feature point in each frame of the image within a specified neighborhood according to the coordinate distribution of the spatially aligned feature point set in the wind turbine blade detection area, and extracts the local density trend value based on the distribution position of all points in the neighborhood in the image space; Gradient identification submodule: Based on the local density trend, calculate the cosine value of the angle between the density direction vectors of adjacent frames, identify the area with prominent direction changes, extract the coordinate range of the target area, and obtain the direction mutation interval; Abnormal marking submodule: According to the angle difference and spatial position distribution of each area in the direction mutation interval, continuous areas with density direction reversal angles exceeding 45 degrees are screened and marked with numbers to obtain abnormal density areas.

6. The multimodal industrial product dynamic defect detection system based on edge computing according to claim 5 is characterized in that: The multi-source anomaly discrimination module includes: Index extraction submodule: Based on the spatial position of the abnormal density area in the wind turbine blade multimodal image, the edge intensity change rate in the visible light image, the heat diffusion direction vector in the infrared thermal image, and the contour discontinuity coefficient in the laser contour image are obtained to establish a multi-source abnormality index set; Direction comparison submodule: calls the heat diffusion direction vector and edge intensity change rate assigned in the multi-source anomaly indicator set, performs directional polarity comparison on the modal indicators under each area number, determines whether a direction reversal relationship is formed, and obtains a direction reversal number set; Interference screening submodule: Based on the direction reversal number set and the contour discontinuity coefficient in the multi-source anomaly indicator set, the values ​​of the matching area are jointly screened to identify the point numbers that meet the direction reversal and abnormal boundary changes, and establish an interference cancellation point set.

7. The multimodal industrial product dynamic defect detection system based on edge computing according to claim 6 is characterized in that: The defect screening module includes: Trajectory judgment submodule: extracts the density change trend and extension direction fluctuation range of each feature point in the interference cancellation point set in the image frame sequence, identifies points with both density fluctuation and direction stability, and obtains a trajectory stable feature point group; Region construction submodule: calls the position coordinates of the trajectory stable feature point group, identifies the mutually adjacent point sets in the single frame image, divides the region boundaries according to the distance relationship between the points, and obtains the connected region clustering results; Result output submodule: Based on the boundary morphology and area distribution in the connected region clustering results, perform integrity judgment on the regional structure, screen the areas with continuous edges and concentrated distribution, and establish the wind turbine blade defect detection results.

Images