Multi-modal industrial product dynamic defect detection system based on edge computing

Abstract

The present application relates to the technical field of visual detection, in particular to a multi-modal industrial product dynamic defect detection system based on edge computing. In the present application, through the introduction of multi-modal image construction and enhancement mode, the three-dimensional acquisition and enhanced expression of the surface detail information of the wind turbine blade are realized, with the aid of the image synchronization and spatial registration mechanism among modes, the consistency extraction capability of the defect information under different sensor perspectives is improved, combined with the density change trend analysis and direction mutation identification means, the structural changes such as surface micro-cracks and edge damage can be accurately identified in continuous frames, further through the multi-source discrimination and mutual comparison of abnormal indexes, the environmental interference and misjudgment risk are effectively eliminated, so as to screen out the defect track area which is stable, has consistent direction and continuous distribution, realize the accurate detection and partition identification of the surface defects of the wind turbine blade under dynamic conditions, and significantly improve the reliability and precision of defect positioning.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of visual inspection, in particular to a multi-modal industrial product dynamic defect detection system based on edge computing. BACKGROUND

[0002] The technical field of visual inspection involves analyzing and recognizing target objects using image acquisition and image processing technology, including image acquisition, image preprocessing, feature extraction, target recognition and defect discrimination, and is widely used in industrial manufacturing, intelligent monitoring, quality control and other application scenarios.

[0003] Among them, the traditional industrial product dynamic defect detection system refers to a technical means for real-time defect detection of continuously moving products on the production line during industrial manufacturing. This system mainly targets surface defects such as scratches, pits, cracks, stains, and edge collapse that may occur during high-speed transmission or continuous processing of products.

[0004] Since the traditional industrial product dynamic defect detection system mainly relies on single modal images for defect judgment in a continuous motion state, it lacks fusion processing of multi-angle and multi-source information, resulting in a significant decline in its ability to recognize minor defects such as cracks, pits, and scratches under conditions of light interference, image blur, or complex background. In the detection process, it relies on instantaneous features between static image frames to judge, ignoring the continuity trend in the defect evolution process, which is prone to missed detection and misjudgment. For example, in the high-speed rotating detection scene of wind turbine blades, the infrared heat diffusion effect may be misjudged as a structural defect, and the contour changes in the slight area cannot be effectively extracted due to noise interference, resulting in a decline in overall detection stability and failing to meet the demand for accurate defect recognition in large-scale, high-dynamic production scenarios. SUMMARY

[0005] The purpose of the present application is to solve the shortcomings in the prior art, and a multi-modal industrial product dynamic defect detection system based on edge computing is proposed.

[0006] In order to achieve the above-mentioned purpose, the present application adopts the following technical scheme: a multi-modal industrial product dynamic defect detection system based on edge computing comprises:

[0007] An image construction module: acquires a multi-modal image set of a wind turbine blade, performs distributed image enhancement and time calibration operations on the multi-modal image set, and uploads it to a cloud node to construct a synchronous image set;

[0008] An edge feature registration module: analyzes the edge structure information of each modal image of the wind turbine blade in the synchronous image set, and extracts a set of spatial alignment feature points of the wind turbine blade;

[0009] Region density analysis module: analyze the region density change trend of each feature point in the spatial alignment feature point set of the wind turbine blade detection region, and mark the abnormal density region;

[0010] Multi-source anomaly discrimination module: construct a multi-source anomaly index set of the wind turbine blade according to the abnormal density region, and perform interference mutual cancellation comparison processing on the multi-source anomaly index set to form an interference mutual cancellation point set;

[0011] Defect screening module: according to the interference mutual cancellation point set, screen the dynamic defect region of the wind turbine blade, and obtain the wind turbine blade defect detection result.

[0012] As a further scheme of the present application, the synchronous image set includes modal synchronization label, time alignment index, image frame serial number, the spatial alignment feature point set includes inter-modal correspondence, spatial relocation parameter, structure key point identification, the abnormal density region includes density change label, abnormal point spatial distribution, region mutation position, the multi-source anomaly index set includes edge intensity parameter, thermal diffusion parameter, contour continuity parameter, and the dynamic defect region includes defect position coordinate, defect region number, detection result label.

[0013] As a further scheme of the present application, the image construction module includes:

[0014] Image acquisition sub-module: acquire visible light image data, infrared thermal image data and laser contour image data collected by edge equipment of wind turbine unit field, determine image frame sequence according to sensor trigger information recorded at image acquisition time, establish image frame channel division according to image source device number and data inflow sequence, and generate multi-modal image index sequence;

[0015] Image enhancement sub-module: according to the multi-modal image index sequence, perform brightness contrast equalization on visible light image data, region gray scale contrast expansion on infrared thermal image data, and edge detail sharpening processing on laser contour image data, and establish multi-channel image enhancement set;

[0016] Synchronous calibration sub-module: call the multi-channel image enhancement set, calculate the time deviation between image frames combined with the image trigger cycle data recorded by the edge equipment, screen the time error between image frames according to the set image synchronization threshold, reconstruct the cross-channel image frame group structure based on the screening result, and acquire the synchronous image set.

[0017] As a further scheme of the present application, the edge feature registration module includes:

[0018] Edge extraction submodule: analyze the edge structure of the visible light image, infrared thermal image and laser profile image of the wind turbine blade in the synchronous image set, extract the intersection position of the gray scale gradient and texture in the image, and establish a modal boundary feature distribution map;

[0019] Key point matching submodule: according to the edge point position of each image frame of the wind turbine blade in the modal boundary feature distribution map, identify the direction main feature points in the adjacent region of each mode, and perform pairing according to the response intensity and direction difference, and establish a modal key point corresponding set;

[0020] Spatial transformation submodule: calling the key point group in the modal key point corresponding set, constructing an image coordinate mapping relationship, unifying the image space position and correcting the offset error, and obtaining a spatial alignment feature point set.

[0021] As a further scheme of the application, the area density analysis module comprises:

[0022] Density calculation submodule: according to the coordinate distribution of the spatial alignment feature point set in the wind turbine blade detection area, the number of adjacent points of each feature point in each image in the specified neighborhood range is obtained, and the local density trend quantity is extracted according to the distribution position of the total neighborhood point number in the image space;

[0023] Gradient identification submodule: based on the local density trend quantity, the included angle cosine value of the density direction vector between adjacent frames is calculated, the region with obvious direction change is identified and the coordinate range of the target region is extracted, and the direction mutation interval is obtained.

[0024] Abnormal marking submodule: according to the angle difference value and spatial position distribution of each region in the direction mutation interval, the continuous region with density direction reversal angle exceeding 45 degrees is screened and numbered, and the abnormal density region is obtained.

[0025] As a further scheme of the application, the multi-source anomaly discrimination module comprises:

[0026] Index extraction submodule: according to the spatial position of the abnormal density region in the wind turbine multi-modal image, the edge intensity change rate in the visible light image, the thermal diffusion direction vector in the infrared thermal image and the profile discontinuous coefficient in the laser profile image are obtained, and a multi-source anomaly index set is established;

[0027] Direction comparison submodule: calling the thermal diffusion direction vector and edge intensity change rate in the multi-source anomaly index set, comparing the direction polarity of the modal index under each region number, judging whether it constitutes a direction reversal relationship, and obtaining a direction reversal number set;

[0028] Interference screening submodule: according to the direction reversal number set and the profile discontinuity coefficient in the multi-source anomaly index set, the numerical value of the matching area is jointly screened, the point number meeting the direction reversal and boundary change anomaly is identified, and the interference mutual elimination point set is established.

[0029] As a further scheme of the present application, the defect screening module comprises:

[0030] Trajectory judgment submodule: extract the density change trend and extension direction fluctuation range of each feature point in the interference mutual elimination point set in the image frame sequence, identify the point position with both density fluctuation and direction stability, and obtain the trajectory stable feature point group;

[0031] Region construction submodule: call the position coordinates of the trajectory stable feature point group, identify the mutually adjacent point set in the single frame image, divide the region boundary according to the distance relationship between points, and obtain the connected region clustering result;

[0032] Result output submodule: according to the boundary shape and area distribution in the connected region clustering result, perform integrity judgment on the region structure, screen the region with continuous edge and concentrated distribution, and establish the wind turbine blade defect detection result.

[0033] Compared with the prior art, the present application has the following advantages and positive effects:

[0034] In the present application, by introducing the multi-modal image construction and enhancement method, the three-dimensional acquisition and enhanced expression of the wind turbine blade surface detail information are realized, the consistency extraction ability of the defect information under different sensor angles is improved by means of the image synchronization and spatial registration mechanism between modes, the density change trend analysis and direction mutation identification means are combined, so that the surface micro-cracks, edge damage and other structural changes can be accurately identified in continuous frames, and through the multi-source discrimination and mutual elimination comparison of the abnormal index, the environmental interference and misjudgment risk are effectively eliminated, so that the defect trajectory region with stability, direction consistency and distribution continuity is screened out, the accurate detection and partition identification of the wind turbine blade surface defects under dynamic conditions are realized, the reliability and precision of defect positioning are significantly improved, and the adaptability and practicality of the system under complex working conditions are enhanced. BRIEF DESCRIPTION OF DRAWINGS

[0035] Figure 1 The system flowchart of the present application;

[0036] Figure 2 The flowchart of the image construction module of the present application;

[0037] Figure 3 The flowchart of the edge feature registration module of the present application;

[0038] Figure 4Flow chart of the region density analysis module of the present application;

[0039] Figure 5 Flow chart of the multi-source anomaly discrimination module of the present application;

[0040] Figure 6 Flow chart of the defect screening module of the present application. DETAILED DESCRIPTION

[0041] In order to make the objects, technical solutions and advantages of the present application clearer, the present application will be further described in detail below in combination with the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and do not limit the present application.

[0042] In the description of the present application, it should be understood that the terms "length", "width", "upper", "lower", "front", "rear", "left", "right", "vertical", "horizontal", "top", "bottom", "inner", "outer" and the like indicate the orientation or positional relationship shown in the drawings, and are only used to facilitate the description of the present application and simplify the description, and therefore cannot be understood as indicating or implying that the device or element referred to must have a particular orientation, be constructed and operated in a particular orientation, and therefore cannot be understood as limiting the present application. In addition, in the description of the present application, the meaning of "a plurality of" is two or more, unless otherwise explicitly and specifically limited.

[0043] Please refer to Figure 1 , the multi-modal industrial product dynamic defect detection system based on edge computing, the system comprises:

[0044] Image construction module: acquire a multi-modal image set of a wind turbine blade, perform distributed image enhancement and time calibration operations on the multi-modal image set, and upload to a cloud node to construct a synchronous image set;

[0045] Edge feature registration module: analyze the edge structure information of each modal image of the wind turbine blade in the synchronous image set, and extract a spatial alignment feature point set of the wind turbine blade;

[0046] Region density analysis module: analyze the region density variation trend of each feature point in the spatial alignment feature point set of the wind turbine blade detection region, and mark the abnormal density region;

[0047] Multi-source anomaly discrimination module: construct a multi-source anomaly index set of the wind turbine blade according to the abnormal density region, and perform interference mutual cancellation comparison processing on the multi-source anomaly index set to form an interference mutual cancellation point set;

[0048] Defect screening module: screen a dynamic defect region of the wind turbine blade according to the interference mutual cancellation point set to obtain a wind turbine blade defect detection result;

[0049] The synchronous image set comprises a modal synchronization label, a time alignment index, and an image frame serial number, the spatial alignment feature point set comprises an inter-modal correspondence, a spatial repositioning parameter, and a structural key point identifier, the abnormal density area comprises a density change label, an abnormal point spatial distribution, and a region mutation position, and the multi-source abnormal index set comprises an edge intensity parameter, a thermal diffusion parameter, and a contour continuity parameter.

[0050] Referring to Figure 2 , the image construction module comprises:

[0051] The image acquisition submodule acquires visible light image data, infrared thermal image data, and laser profile image data collected by edge devices on site, determines image frame order according to sensor trigger information recorded at the time of image acquisition, establishes image frame channel division according to image source device number and data inflow order, and generates a multi-modal image index sequence.

[0052] To acquire visible light image data, infrared thermal image data, and laser profile image data collected by edge devices on site, the data channels of the edge devices need to be accessed one by one. First, the devices are identified based on the device numbers, with each number corresponding to a unique image sensor. For example, the visible light camera corresponding to the device with the number WT001 of a certain wind turbine acquires images through the channel, which are accurately labeled with timestamps. The image frames are sorted according to the trigger signals recorded by the edge control system, for example, 10 frames are triggered every second, with an interval of 0.1 second between each frame, and the corresponding trigger timestamps are T, T+0.1, T+0.2, etc. Next, the system groups the images according to the device numbers, for example, the images in channel A are from WT001, the infrared images in channel B are from WT002, and the laser profile images in channel C are from WT003. The data in each channel is cached according to the FIFO (First-In-First-Out) rule, and a record index of the image inflow order is established, for example, the first frame in channel A is T0, the first frame in channel B is T1, and the first frame in channel C is T2. This index sequence is used for subsequent image enhancement and synchronization processing. For example, during the operation and maintenance of a wind turbine, the visible light camera records the surface images of the blades, the infrared camera records the temperature field changes of the device surface, and the laser profile sensor records the profile deformation or crack path. These data are processed through the above-mentioned acquisition process to construct a preliminary multi-modal alignment index structure.

[0053] The image enhancement submodule performs brightness contrast equalization on the visible light image data, regional gray scale contrast expansion on the infrared thermal image data, and edge detail sharpening processing on the laser profile image data according to the multi-modal image index sequence, and establishes a multi-channel image enhancement set.

[0054] According to the multi-modal image index sequence, the image data is processed by brightness contrast equalization, gray contrast expansion and edge detail sharpening. First, for the visible light image data, the image pixel matrix is read and the gray value distribution is counted. Assuming that the image size is 1920x1080, the total number of pixels is If the image gray value is concentrated in the interval of 70-120, histogram equalization processing is needed to improve the gray distribution range. The specific operation includes: first, count the frequency of each gray value (where ) and calculate the probability distribution function :

[0055] ;

[0056] wherein, represents the gray level, represents the number of pixels of the gray level, and represent the image width and height respectively. For example, a certain gray level corresponds to the number of pixels , and its probability distribution is:

[0057] ;

[0058] Then calculate the cumulative distribution function (CDF) :

[0059] ;

[0060] Take for example. Assuming that the cumulative probability from to is , then the equalized gray value after mapping is calculated according to the cumulative probability :

[0061] ;

[0062] wherein is the total number of gray levels, represents the floor operation, and substituting the data gives:

[0063] ;

[0064] ​That is, the original gray value 80 will be mapped to the enhanced gray value 89, and this process is completed for all pixels to form an enhanced image data set. Next, the regional gray contrast expansion processing is performed on the infrared thermal image. The pixel gray value range in the infrared image is 50-100, and the corresponding actual temperature range is 30℃-60℃. In order to enhance the contrast, the gray value range needs to be linearly stretched to [0, 255]. The linear transformation function is as follows:

[0065] ;

[0066] wherein is the original gray value, , , and is taken as an example, the calculation is as follows:

[0067] ;

[0068] After rounding, it is 128, which means that the gray value 75 is mapped to the enhanced gray value 128, so that different temperature regions in the image are more easily distinguished. If it is necessary to set a gray threshold to judge the temperature rise abnormal region, the temperature reference value can be set to 40℃, and the corresponding gray value is:

[0069] ;

[0070] The value after mapping is:

[0071] ;

[0072] But it needs to be clipped to 255 if it exceeds 255, which means that the high-temperature region reaches the thermal threshold. Next, the edge detail sharpening operation is performed on the laser contour image. The Sobel operator is used to calculate the gradient amplitude. The horizontal gradient kernel and the vertical gradient kernel are used respectively:

[0073] ;

[0074] For each pixel , the convolution calculation is performed on its 3x3 neighborhood. The designed horizontal gradient value is , the vertical gradient value is , and the total gradient amplitude is:

[0075] ;

[0076] The value is used as the pixel brightness to generate the sharpened image. Further non-maximum suppression and double-threshold connection are performed to retain the main contour region and exclude weak boundary noise. Finally, the three-channel image enhancement set is formed, which is used for subsequent synchronous calibration processing.

[0077] Synchronization calibration sub-module: call multi-channel image enhancement set, combine edge device recorded image trigger cycle data to calculate time deviation between image frames, screen image frame time error according to set image synchronization threshold, reconstruct cross-channel image frame group structure based on screening result, obtain synchronization image set;

[0078] In the multi-channel image set after image enhancement, combine the image trigger cycle data recorded by the edge device to calculate the time difference. Assuming that the image acquisition period is 0.1 seconds, i.e. 10 frames of images are collected per second, the devices are channel A (visible light image), channel B (infrared thermal image), and channel C (laser profile image). First, extract the timestamp sequence corresponding to each image frame in each channel. For example, the image timestamp of channel A is 、 、 , that of channel B is 、 、 , and that of channel C is 、 、 , where the symbol represents the acquisition time of the frame image in the channel ( ) in seconds. Next, perform image frame synchronization error calculation. The maximum and minimum timestamp difference is used as the frame time difference indicator, and the calculation formula is as follows:

[0079] ;

[0080] where represents the maximum time difference of the image acquisition timestamp of the frame group in the three channels. If the timestamp corresponding to frame group 1 is 、 、 , then:

[0081] ;

[0082] Similarly, in frame group 2, 、 、 , then:

[0083] ;

[0084] To determine whether the frame group meets the synchronization requirements, it needs to be compared with the set image synchronization threshold . The judgment criteria are: , where is the image synchronization threshold. The setting is based on the following considerations: the maximum clock synchronization error of the edge device is , image acquisition buffer error is , the synchronization threshold can be set as:

[0085] ;

[0086] Therefore, when , it is considered as a synchronous frame group, otherwise it is rejected, for example, frame group 1 and frame group 2 both meet the condition, and are retained into the synchronous image set, if frame group 3 is , , , then:

[0087] ;

[0088] also meet the threshold condition, enter the set, and then the system reconstructs all image frames that meet the synchronization condition into a synchronous image frame group structure, and each group structure is represented as a three-tuple: , forming a frame group index list: frame group 1 is , frame group 2 is , and frame group 3 is , all frame groups meet , and constitute a final synchronous image set. Taking a wind power inspection site as an example, if the channel A image shows that the local foreign matter covers the blade, the channel B image shows that the temperature in the corresponding area abnormally rises, and the channel C image shows that the outline of the area is irregular, by confirming that the three modalities are collected at the same time point through synchronization calibration, the cross-channel consistency analysis basic data set is effectively established.

[0089] Please refer to Figure 3 , the edge feature registration module includes:

[0090] An edge extraction submodule: analyze the edge structure of the visible light image, the infrared thermal image and the laser contour image of the wind turbine blade in the synchronous image set, extract the intersection position of the gray gradient and the texture in the image, and establish a modal boundary feature distribution map;

[0091] The edge structures of the visible light images, infrared thermal images and laser profile images of the wind turbine blade in the synchronous image set are analyzed. First, the image matrix data is read frame by frame, and the RGB channel or grayscale channel images are obtained. Different edge analysis methods are selected according to the image type. For example, the gray gradient detection method is used for visible light images to extract the edge of the image brightness change area. The thermal gradient of the temperature field is used for infrared thermal images to extract the boundary. The structural curvature change is used as the basis for edge recognition for laser profile images. The edge structures of the three modal images are stored in the structure data list. In this process, the texture intersection area in the image needs to be extracted. This step is obtained by scanning the central point of the texture change frequency in the image area. For example, at the transition position of the blade root structure, the gray scale changes and the texture edges intersect, and the significant boundary line is easy to extract. The brightness or thermal value change amplitude of each pixel neighborhood area in the image is scanned by a sliding window to identify the area with a gray scale mutation in the image. The texture feature dense intersection position is extracted in the range around the mutation point. The spatial positioning of the basic edge features in the multi-modal image is completed. The edge points and texture intersection points are stored in the data index table with the image frame number. The differences in the boundary information of the same area in the modal images are compared to establish the modal boundary feature distribution map.

[0092] The key point matching sub-module identifies the direction main feature points in the adjacent area between each modal according to the edge point position of each image frame of the wind turbine blade in the modal boundary feature distribution map, and performs pairing according to the response intensity and direction difference to establish the modal key point corresponding set.

[0093] The key point matching sub-module identifies the direction main feature points in the adjacent area between each modal according to the edge point position of each image frame of the wind turbine blade in the modal boundary feature distribution map, and performs pairing according to the response intensity and direction difference to establish the modal key point corresponding set.

[0094] The spatial transformation submodule: calls the key point group in the modal key point corresponding set, constructs the image coordinate mapping relationship, unifies the image space position and corrects the offset error, and obtains the spatial alignment feature point set;

[0095] The spatial transformation submodule: calls the key point group in the modal key point corresponding set, constructs the image coordinate mapping relationship, unifies the image space position and corrects the offset error, and obtains the spatial alignment feature point set, first reads the key point coordinate values in the three-modal images contained in the modal key point corresponding set, uniformly loads the key point coordinates under channels A, B and C according to each frame number, selects a reference modal image according to the image geometric structure, for example, takes the channel A image as the reference, calculates the relative displacement of the corresponding key points in the channel B and C images, compares the image coordinate difference values, judges the offset type to be translation, rotation or scaling, if the offset mainly shows linear translation, a two-dimensional translation matrix is constructed for position correction, if rotation and scaling exist at the same time, an affine transformation matrix is constructed between the coordinate points, the key points in different modal images are unified to the reference coordinate system through the mapping matrix, and then whether the coordinate value comparison error after unification is lower than the set tolerance threshold is calculated, for example, the spatial offset tolerance is set to 2 pixels, if the distance between the mapped key points is less than the value, the matching is accepted, otherwise, local refinement adjustment is performed, after processing all frame groups, the spatial alignment feature point set in the unified spatial coordinate is formed.

[0096] Please refer to Figure 4 , the regional density analysis module includes:

[0097] The density calculation submodule: according to the coordinate distribution of the spatial alignment feature point set in the wind turbine blade detection area, the number of adjacent points of each feature point in each image in the specified neighborhood range is obtained, and the local density trend quantity is extracted according to the distribution position of the total neighborhood point number in the image space;

[0098] Based on the coordinate distribution of spatially aligned feature points in the wind turbine blade detection area, all spatially aligned feature points in each frame are first read as a two-dimensional coordinate sequence. Assuming an image resolution of 1920×1080, a grid coordinate system is established on the image plane, defining a neighborhood search radius of 25 pixels. A neighborhood search window is constructed at the center of each feature point, and the number of other feature points within the window is counted as its neighboring points. If a feature point detects 12 other feature points within its neighborhood radius, its local neighboring points are 12. This process is repeated to obtain the neighboring number of all feature points in each frame. The density value of each point is defined in this way based on this neighboring number, and then... The coordinates and density values ​​of feature points are plotted in the image space to create a two-dimensional density distribution map. By dividing the image space into several sub-regions, the average density value of the points contained in each region is calculated and then visualized using color gradient encoding, forming a spatial trend mapping of density changes with location. This density trend is described as a local density trend quantity, which comes from the degree of clustering of feature points in space. If the number of adjacencies of multiple points in a certain region exceeds 15, the local density of that region can be considered high. If the number of adjacencies in a certain region is less than 5, it is classified as a low-density region. After completing the feature point adjacency analysis of all frames of images, the system saves the two-dimensional spatial density trend corresponding to each frame in a list structure.

[0099] Gradient recognition submodule: Based on local density trend, calculate the cosine of the angle between density direction vectors of adjacent frames, identify regions with prominent direction changes and extract the coordinate range of the target region to obtain the direction change interval;

[0100] Based on local density trend, each frame of the image is first divided into multiple density segment regions. The density center of each point within a region is calculated by the weighted average of the coordinates of all feature points. Let the first... A region A in a frame image contains feature points There are , where the coordinates of each feature point are . Density value Then the coordinates of the density center point of this region Represented as:

[0101] ;

[0102] in, Indicates the first The density center coordinates of region A in the frame. For the first The horizontal and vertical coordinates of each feature point in the image coordinate system The density value is the number of neighboring points of the feature point in its local neighborhood. Based on this point set, the density center is calculated for each region. Then, the direction of change of the density center between consecutive frames is calculated, defining the... Frame and the The displacement vector of the density center between frames is:

[0103] ;

[0104] in, For the first The x and y coordinates of the density center of frame region A, For the first The corresponding coordinate values ​​of the frame, resulting in a vector The direction of density change is represented by the cosine of the angle between two adjacent vectors, which is then used to determine the magnitude of the directional change. The formula for this calculation is as follows:

[0105] ;

[0106] in: The dot product of two direction vectors is calculated as follows: ; For vectors The modulus length, i.e. ; The angle between the density direction vectors of the two frames.

[0107] Let the density center point of the 3rd frame in region A be... The 4th frame is Frame 5 is ,but: , .

[0108] Calculate the dot product: ;

[0109] Calculate the modulus: ; ;

[0110] The cosine value is: ;

[0111] The angle of change of direction is: .

[0112] Set the density center of the 6th frame. ,but: ;

[0113] Dot product: ;

[0114] ;

[0115] The cosine value is calculated as follows: ;

[0116] included angle: .

[0117] The description direction is almost completely reversed, so the region is identified as a mutation region, and the system records the direction change of the region between frame 3, frame 4 and frame 5, and the coordinate point Add the direction mutation interval list for subsequent anomaly detection.

[0118] Anomaly marking sub-module: according to the angle difference and spatial position distribution of each region in the direction mutation interval, filter the continuous regions whose density direction reversal angle exceeds 45 degrees and mark the number, and obtain the abnormal density region;

[0119] According to the angle difference and spatial position distribution of each region in the direction mutation interval, first read the angle difference sequence of each region between consecutive frames from the marked direction mutation interval list, archive according to region number, take all angle difference values greater than 45 degrees as candidate reverse regions, set the analysis span to three frames, if a region continuously appears large angle mutation in three frames, that is, the direction change between each adjacent frame is greater than 45 degrees, the system marks the region as a density direction reversal continuous region, further calculates the coordinate coverage range of each candidate region in the image plane, and removes the regions with spatial span less than a certain pixel area, for example, set the minimum boundary of the region to 50x50 pixels, only keep the image regions with actual size, then number the regions filtered according to the processing order one by one, the numbering method records the unique identification according to the combination of image frame number and region serial number, forms a unique identification, at the same time, establishes the corresponding relationship between region number and image frame position, writes the marking information into the abnormal region index structure, and finally outputs the number, corresponding frame range and spatial coverage coordinates of all abnormal density regions as the output result.

[0120] Please refer to Figure 5 , the multi-source anomaly discrimination module comprises:

[0121] Index extraction sub-module: according to the spatial position of the abnormal density region in the multi-modal image of the wind turbine blade, the edge intensity change rate in the visible light image, the thermal diffusion direction vector in the infrared thermal image and the contour discontinuity coefficient in the laser profile image are obtained, and a multi-source anomaly index set is established;

[0122] According to the spatial position of the abnormal density region in the multi-modal image of the wind turbine blade, first extract the physical structure or thermal feature change in the corresponding region of the three types of images, establish a multi-source anomaly index set, and for the edge intensity change rate in the visible light image, the system extracts the edge of the region, selects the gray gradient value of each pixel point on the specified boundary line segment as , which represents the gray intensity of the th sampling point, the sequence contains a total of sampling points, that is , the gray difference between these sampling points is averaged, and the edge intensity change rate is defined as:

[0123] ;

[0124] wherein, represents the edge intensity change rate, unit is gray value, is the total number of sampling pixels, is the gray value of the pixel point, represents the gray value of the adjacent next pixel point, and the absolute value term represents the gray mutation amplitude. Assuming that a total of pixel points are extracted on the edge of a certain abnormal area, and the gray value sequence is: [30, 35, 40, 80, 85, 90, 60, 55, 50, 45], the adjacent difference value is calculated as: , , , , , , , , .

[0125] The sum of these differences is 105, which is substituted into the formula: .

[0126] The value represents the degree of gray fluctuation between pixels, and a sharp change indicates a significant edge mutation. Next, the heat diffusion direction vector in the infrared thermal image is processed. First, the temperature values of the center point and its upper and lower neighborhood pixel points are obtained, and the center pixel temperature is , the upper, lower, left and right neighborhood point temperatures are , and the direction vector is constructed as follows:

[0127] ;

[0128] wherein, represents the heat diffusion direction vector, the first component represents the horizontal temperature difference, and the second component represents the vertical temperature difference. Assuming that in a certain region: , , , , , the vector is , indicating that heat diffuses to the right and down direction. Then the vector length is calculated as: ; the unit direction vector is about , which is used as the directional description of the heat characteristics of the region. Finally, the profile discontinuity coefficient in the laser profile image is processed. The system obtains a set of equally spaced sampling points along the scanning line after edge extraction. Assuming that a total of profile points are sampled, and the coordinates of each point are , wherein is the horizontal pixel position of the point, is the profile height value in the vertical direction, the profile jump is identified by calculating the height difference of two adjacent points, defined as follows:

[0129] ;

[0130] wherein, is the height change amplitude between the first and the second profile point, if the difference is greater than the set threshold , it is recorded as a jump event, the number of all jump point pairs meeting the conditions is , and the profile discontinuity coefficient is:

[0131] ;

[0132] Suppose a certain area has a total of points, and the analysis shows that the height difference of 7 adjacent point pairs exceeds the set threshold pixels, then: , which indicates that the region profile continuity is poor, and there is a risk of structural mutation. This coefficient, together with the edge intensity change rate and the thermal diffusion direction vector, forms a three-modal anomaly index set for the region. The system organizes the data into a multi-source anomaly index set according to the region number and stores it for subsequent direction comparison and interference screening processing.

[0133] Direction comparison sub-module: call the thermal diffusion direction vector and edge intensity change rate allocated in the multi-source anomaly index set, and compare the direction polarity of the modal index under each region number to determine whether it constitutes a direction reversal relationship and obtain the direction reversal number set.

[0134] Call the thermal diffusion direction vector and edge intensity change rate allocated in the multi-source anomaly index set, and compare the direction polarity of the modal index under each region number. The system first extracts the direction attribute of the thermal diffusion direction vector of each numbered region during execution, converts it to an angle representation, and converts the edge direction corresponding to the edge intensity change rate in the visible light image in the same way. By calculating the direction angle difference between the two, the polarity relationship is determined. If the angle difference is close to 180 degrees, it means that the two modal directions are opposite, then it is recorded as a direction reversal region. The system repeats the above operation for each numbered region, adds the region number of the region determined to be direction reversed to the reversal number set, and at the same time, marks the regions with significant direction angles in the reversal relationship with a weight, and records them in the set according to the number.

[0135] Interference screening sub-module: according to the profile discontinuity coefficient in the direction reversal number set and the multi-source anomaly index set, the numerical value of the matching region is jointly screened to identify the point number that meets the direction reversal and boundary change anomaly, and the interference mutual elimination point set is established;

[0136] According to the profile discontinuity coefficient in the direction reversal number set and the multi-source anomaly index set, the system extracts the corresponding profile discontinuity value of each region number marked as direction reversal one by one, compares it with the set deformation anomaly threshold, judges whether the region meets the profile structure jump condition at the same time, if the profile discontinuity coefficient of a region exceeds the set reference and the direction polarity is in the reverse state, the region number is determined as interference anomaly, the system marks the point number as interference mutual elimination point, and excludes other numbers that do not meet the double conditions, and records all point numbers that meet the joint screening condition as interference mutual elimination point set.

[0137] Please refer to Figure 6 , the defect screening module comprises:

[0138] Trajectory judgment sub-module: extract the density change trend and extension direction fluctuation range of each feature point in the interference mutual elimination point set in the image frame sequence, identify the point that has both density fluctuation and direction stability, and obtain the trajectory stable feature point group;

[0139] Extract the density change trend and extension direction fluctuation range of each feature point in the interference mutual elimination point set in the image frame sequence, first, the trajectory restoration operation is performed for the position of each point in different time frames, the local density value of the region where the point is located in the continuous frames is counted, the change amplitude of the point in the time sequence is judged by the fluctuation range of the density value, the points with density fluctuation range in the medium or below interval are marked as stable density points, and the direction vector of the point in each frame and the previous and next frames is extracted, the included angle of the direction change is calculated and the direction fluctuation interval of the continuous frames is obtained, if the direction change of the point in three or more continuous frames is less than the set fluctuation reference, it is marked as a direction stable point. The system performs joint screening judgment on the density change trend and the direction fluctuation range of the two types of features, sets the upper limit of the density change and the direction fluctuation threshold, judges whether it meets the double condition matching, for example, the density change is not more than 20% of the global average, and the direction change angle is not more than 15 degrees, then the point is considered to have stable density and stable direction at the same time, and the system forms a trajectory stable feature point group after screening all the points that meet the standard.

[0140] Region construction sub-module: call the position coordinates of the trajectory stable feature point group, identify the point set that is adjacent to each other in a single frame image, divide the region boundary according to the distance relationship between points, and obtain the connected region clustering result;

[0141] The position coordinates of the trajectory stable feature point group are called, and the mutually adjacent point set in a single frame image is identified. First, the coordinates of the trajectory stable feature points in each frame image are spatially clustered and divided. Whether two points belong to the same structural block is determined by using the adjacency relationship in the region. Specifically, the distance between the points and the number of intersection of the neighborhood are used for judgment. A distance threshold and a lower limit of the number of adjacent points are set. The distance between any two points is determined. If it is less than the set spatial threshold and is reachable in the spatial adjacency graph, it is classified into a group. The system iterates all point pairs to construct an adjacency graph structure. Then, based on the structure, a region expansion operation is performed. The boundary contour of the connected component forming a closed graph is extracted. The minimum circumscribed frame area and shape ratio surrounded by the contour boundary are used for preliminary description. Finally, the clustering region set that meets the adjacency and closure conditions is output, and the connected region clustering result is formed.

[0142] The result output sub-module: according to the boundary shape and area distribution in the connected region clustering result, the integrity of the region structure is judged, the regions with continuous edges and concentrated distribution are screened, and the wind turbine blade defect detection result is established;

[0143] According to the boundary shape and area distribution in the connected region clustering result, the integrity of the region structure is judged, the contour line segment is extracted from the boundary of the clustering region, the continuity index of the contour line is calculated, the angle change amplitude between every two continuous contour points on the boundary is counted, whether there is a fracture, sharp turn or reverse structure is judged, and the boundary continuity score is calculated combined with the closed degree of the contour. The regions with scores lower than the set standard are excluded. The area and shape distribution characteristics of the region are further analyzed. The area, perimeter, maximum side length ratio, length-width ratio and other indexes of each region are counted. Whether it belongs to the concentrated region structure is judged. If the area accounts for more than a certain proportion in the whole image and the shape tends to be a regular closed structure, the region passes the structural integrity judgment. The system numbers and records the regions that meet the conditions of continuous edges and concentrated shape, and marks them as the wind turbine blade defect detection result. In practical application, the wind turbine blade defect detection result may have many kinds. The main problem is false positive caused by edge misidentification. That is, the edge mutation caused by image noise, stains or surface coating changes is misjudged as a defect. In addition, in the multi-modal image fusion process, spatial registration errors may cause different modal features to be misplaced, thereby causing defect positioning to shift or be missing. At the same time, for the slight cracks, early bubbles or delamination type micro-defects on the surface of the wind turbine blade, the system is prone to miss detection due to low signal strength or weak texture. In addition, in the defect region clustering process, some fuzzy boundary regions may be divided into multiple discontinuous blocks by the system, causing the defect boundary to be incomplete. Finally, under the condition of complex background or multiple occlusions, the feature point density change trend is not obvious, which will also reduce the robustness and accuracy of the system recognition.

[0144] The above merely describes the preferred embodiments of the present application, and is not intended to limit the present application in other forms. Any skilled person in the art can modify or change the disclosed technical content into equivalent embodiments with equivalent changes, and apply them to other fields. However, any simple modification, equivalent change and modification made to the above embodiments according to the technical essence of the present application, without departing from the technical solution content of the present application, still falls within the protection scope of the present application.

Claims

1. A multimodal industrial product dynamic defect detection system based on edge computing, characterized in that, The system includes: Image construction module: acquires a multimodal image set of wind turbine blades, performs distributed image enhancement and temporal calibration operations on the multimodal image set, and uploads it to the cloud node to build a synchronous image set; Edge feature registration module: Analyzes the edge structure information of each modal image of the wind turbine blade in the synchronized image set, and extracts the spatial alignment feature point set of the wind turbine blade; Regional density analysis module: Analyzes the regional density variation trend of each feature point in the spatially aligned feature point set of the wind turbine blade detection area, and marks abnormal density areas; Multi-source anomaly discrimination module: Constructs a multi-source anomaly index set for wind turbine blades based on the anomaly density region, and performs interference cancellation comparison processing on the multi-source anomaly index set to form an interference cancellation point set; Defect screening module: Based on the interference cancellation point set, the module screens the dynamic defect areas of the wind turbine blades to obtain the wind turbine blade defect detection results; The regional density analysis module includes: Density calculation submodule: Based on the coordinate distribution of the spatially aligned feature point set in the wind turbine blade detection area, obtain the number of adjacent points of each feature point in each frame of the image within a specified neighborhood range, and extract the local density trend based on the distribution of the number of points in the entire neighborhood in the image space. Gradient recognition submodule: Based on the local density trend, calculate the cosine of the angle between the density direction vectors of adjacent frames, identify regions with prominent direction changes and extract the coordinate range of the target region to obtain the direction change interval; Anomaly Marking Submodule: Based on the angle difference and spatial distribution of each region in the direction change interval, filter out continuous regions with density direction reversal angles exceeding 45 degrees and mark them with numbers to obtain abnormal density regions; The multi-source anomaly detection module includes: The index extraction submodule: Based on the spatial location of the abnormal density region in the multimodal image of the wind turbine blade, it obtains the edge intensity change rate in the visible light image, the heat diffusion direction vector in the infrared thermal image, and the contour discontinuity coefficient in the laser contour image, and establishes a multi-source abnormal index set. Direction comparison submodule: Calls the heat diffusion direction vector and edge intensity change rate already assigned in the multi-source anomaly index set, performs direction polarity comparison on the modal index under each region number, determines whether a direction reversal relationship is formed, and obtains the direction reversal number set; Interference Removal Submodule: Based on the set of direction reversal numbers and the contour discontinuity coefficients in the set of multi-source anomaly indicators, the values ​​of the matching region are jointly screened to identify the point numbers that satisfy the direction reversal and boundary change anomalies, and an interference cancellation point set is established. The defect screening module includes: The trajectory determination submodule extracts the density change trend and extension direction fluctuation range of each feature point in the interference cancellation point set in the image frame sequence, identifies points that simultaneously have density fluctuations and direction stability, and obtains a group of trajectory-stable feature points. Region construction submodule: Calls the position coordinates of the trajectory stable feature point group, identifies the set of mutually adjacent points in a single frame image, divides the region boundary according to the distance relationship between points, and obtains the clustering results of connected regions; The result output submodule performs integrity judgment on the regional structure based on the boundary shape and area distribution in the clustering results of the connected regions, filters regions with continuous edges and concentrated distribution, and establishes the wind turbine blade defect detection results.

2. The multimodal industrial product dynamic defect detection system based on edge computing according to claim 1, characterized in that, The synchronized image set includes modal synchronization labels, time alignment indexes, and image frame sequence numbers. The spatial alignment feature point set includes inter-modal correspondences, spatial relocation parameters, and structural key point identifiers. The abnormal density region includes density change markers, spatial distribution of abnormal points, and abrupt change locations in the region. The multi-source anomaly index set includes edge intensity parameters, thermal diffusion parameters, and contour continuity parameters. The dynamic defect region includes defect location coordinates, defect region number, and detection result markers.

3. The multimodal industrial product dynamic defect detection system based on edge computing according to claim 1, characterized in that, The image construction module includes: Image acquisition submodule: acquires visible light image data, infrared thermal image data and laser contour image data collected by the edge devices of the wind turbine, determines the image frame order based on the sensor trigger information recorded at the time of image acquisition, establishes image frame channel division according to the image source device number and data inflow order, and generates a multimodal image index sequence; Image enhancement submodule: Based on the multimodal image index sequence, it performs brightness contrast equalization on visible light image data, performs regional grayscale contrast expansion on infrared thermal image data, performs edge detail sharpening on laser contour image data, and establishes a multi-channel image enhancement set; Synchronization calibration submodule: calls the multi-channel image enhancement set, calculates the time deviation between image frames by combining the image trigger cycle data recorded by the edge device, filters the time error between image frames according to the set image synchronization threshold, reconstructs the cross-channel image frame group structure based on the filtering results, and obtains the synchronized image set.

4. The multimodal industrial product dynamic defect detection system based on edge computing according to claim 3, characterized in that, The edge feature registration module includes: Edge extraction submodule: Analyzes the edge structure of visible light images, infrared thermal images and laser contour images of wind turbine blades in the synchronized image set, extracts the gray-level gradient and texture intersection positions in the images, and establishes a modal boundary feature distribution map; Key point matching submodule: Based on the edge point positions of each image frame of the wind turbine blade in the modal boundary feature distribution map, identify the main directional feature points in adjacent areas between each mode, and perform pairing according to the difference in response intensity and direction to establish a set of corresponding modal key points; Spatial transformation submodule: calls the key point group in the set corresponding to the modal key points, constructs the image coordinate mapping relationship, unifies the image spatial position and corrects the offset error, and obtains the spatial alignment feature point set.

Images