Fusion analysis system for visual appearance and electromagnetic detection data of steel wire rope based on AI

Abstract

The application discloses an AI-based fusion analysis system for visual appearance and electromagnetic detection data of a steel wire rope, and relates to the technical field of intelligent monitoring of industrial equipment.The system collects image frames and electromagnetic signals of the steel wire rope, calculates the running space length corresponding to the image frames and the electromagnetic signals according to the instantaneous speed of the steel wire rope, calculates the time offset ATve between each image frame and each electromagnetic signal, performs spatial mapping on the positions of the steel wire rope corresponding to the image frames and the electromagnetic signals, extracts features from the mapped image frames and electromagnetic signals, calculates the modal feature matching degree MAT after feature extraction, and performs matching degree evaluation; when the matching degree evaluation is qualified, the drift error value SYN is calculated and timing drift evaluation is performed; based on the multi-modal convolutional neural network, defect classification is performed, a steel wire rope defect report is generated, and information such as defect type, position and severity is provided. The application improves the accuracy and real-time performance of steel wire rope defect detection.

Description

Technical Field

[0001] This invention relates to the field of intelligent monitoring technology for industrial equipment, specifically to an AI-based fusion analysis system for the visual appearance and electromagnetic detection data of steel wire ropes. Background Technology

[0002] With the rapid development of artificial intelligence (AI) technology, especially in image processing and signal analysis, AI has become a key driving force in many industrial inspection applications. AI-based systems, through deep learning algorithms, automate the processing of complex data, significantly improving inspection accuracy and efficiency. For traditional physical inspection methods, AI can enhance the intelligence level of inspection systems by combining multimodal data analysis. In recent years, the fusion analysis technology of visual appearance and electromagnetic inspection data has become a research hotspot in the field of wire rope inspection. As a critical piece of equipment widely used in high-altitude operations, transportation systems, and heavy industry, the safety of wire ropes is directly related to personnel safety and equipment operational stability. By combining image data and electromagnetic signals, AI-based inspection systems can identify surface and internal defects of wire ropes in real time and accurately during their use, providing important information for safety monitoring and maintenance decisions.

[0003] Defect detection in wire ropes has traditionally relied on manual inspection and single testing methods, such as visual inspection, magnetic particle testing, or ultrasonic testing. However, traditional methods face several significant problems: First, manual inspection is not only time-consuming and labor-intensive but also susceptible to human error, especially when the wire rope is in a complex operating environment, where inspectors may easily overlook minute or deep-seated defects. Second, single testing methods often fail to comprehensively capture all potential defects in the wire rope. For example, visual inspection can only detect surface defects, while electromagnetic testing, although capable of capturing internal defects, is not easy to provide precise spatial location. These limitations of inspection technologies make early detection and precise location of defects difficult, increasing the risk of accidents. Summary of the Invention

[0004] To address the shortcomings of existing technologies, this invention provides an AI-based fusion analysis system for the visual appearance and electromagnetic detection data of steel wire ropes, thus solving the problems mentioned in the background technology.

[0005] To achieve the above objectives, the present invention is implemented through the following technical solution: an AI-based fusion analysis system for the visual appearance and electromagnetic detection data of steel wire ropes, including a data acquisition module, a spatial mapping module, a feature extraction module, a matching degree evaluation module, a time series analysis module, and a feature fusion module; The data acquisition module is used to acquire wire rope image frames and electromagnetic signals and perform simultaneous time indexing. The spatial mapping module is used to collect the instantaneous velocity Vr of the wire rope at different time points, and to calculate the running space length of the wire rope corresponding to the image frame and the electromagnetic signal, respectively. The feature extraction module is used to calculate the time offset ΔTve between each image frame and each electromagnetic signal, and to spatially map the positions of the steel wire ropes corresponding to the image frames and electromagnetic signals. Then, it extracts the pixel edge intensity Ged and the electromagnetic signal response amplitude Rma from the mapped image frames and electromagnetic signals. The matching degree evaluation module is used to calculate the modal feature matching degree MAT and perform matching degree evaluation. The time series analysis module is used to calculate the drift error value SYN when the matching degree evaluation is qualified, and to perform time series drift evaluation; The feature fusion module is used to construct a dual-modal fusion tensor Ff using aligned image frames and electromagnetic signals when there is no drift in temporal synchronization, and then generate a defect report after classification based on a multimodal convolutional neural network (CNN).

[0006] Preferably, the data acquisition module includes a visual image acquisition unit, an electromagnetic signal acquisition unit, and a time index management unit; The visual image acquisition unit is used to continuously capture images of the surface of the steel wire rope passing through the detection channel using an industrial camera, generating an image frame sequence Iv, where Iv = {Iv1, Iv2, ..., Iv...}. n}, where n represents the number of image frames, and the acquisition timestamp Tv of each image frame is recorded, Tv={Tv1, Tv2, ..., Tv n}; The electromagnetic signal acquisition unit is used to generate an electromagnetic signal sequence Ee, where Ee = {Ee1, Ee2, ..., Ee...}, based on the electromagnetic signals collected by the electromagnetic sensor during the operation of the wire rope. m}, where m represents the number of electromagnetic signals collected, and the timestamp Te of each electromagnetic signal is recorded, Te = {Te1, Te2, ..., Te}. m}; The time index management unit is used to sort the image frame sequence Iv and the electromagnetic signal sequence Ee in chronological order, generate a unified time index set containing timestamps Tv and Te, and embed global time series labels for alignment, mapping and matching with a unified time reference.

[0007] Preferably, the spatial mapping module includes a timing synchronization unit and a segment calculation unit; The timing synchronization unit is used to obtain the instantaneous speed Vr of the wire rope at different time points based on the encoder installed at the front end of the detection area, and to perform timing alignment and interpolation synchronization of all speed sampling points by reading the image frame acquisition timestamp Tv and the electromagnetic signal acquisition timestamp Te. The segment calculation unit is used to perform velocity integration calculations within the sampling time interval of the image frame and the electromagnetic segment based on the instantaneous velocity value Vr combined with the image frame acquisition timestamp Tv and the electromagnetic signal acquisition timestamp Te, respectively, to obtain the steel wire rope running space length segment corresponding to each image frame and each electromagnetic segment, as follows; The length of the wire rope running space corresponding to the i-th image frame is Lv. i The calculation is as follows: Where Vr(t) represents the instantaneous velocity at time t, and Tv i Tv represents the acquisition timestamp of the i-th image frame. i-1 dt represents the acquisition timestamp of the (i-1)th image frame, and dt represents the time integration variable; The length of the wire rope running space corresponding to the j-th segment of electromagnetic signal is Le. j The calculation is as follows: Where Vr(t) represents the instantaneous velocity at time t, and Te j Te represents the timestamp of the acquisition of the j-th segment of electromagnetic signal. j-1 dt represents the timestamp of the acquisition of the (j-1)th segment of electromagnetic signal, and dt represents the time integration variable.

[0008] Preferably, the feature extraction module includes a time offset calculation unit and a feature extraction unit; The time offset calculation unit is used to calculate the time offset ΔTve between each image frame and each electromagnetic signal based on the image frame acquisition timestamp Tv and the electromagnetic signal acquisition timestamp Te. Specifically: ,in, Represents the smallest index symbol, indicating the frame timestamp Tv of the i-th image frame. i The electromagnetic signal with the smallest time difference.

[0009] Preferably, the feature extraction unit includes a spatial mapping unit, an image feature extraction unit, and an electromagnetic feature extraction unit; The spatial mapping unit is used to calibrate the industrial camera to obtain the camera's intrinsic and extrinsic parameter matrices. The extrinsic parameter matrix is ​​used to transform the physical coordinates of the electromagnetic signal to the camera coordinate system. The camera's intrinsic parameter matrix is ​​used to map the three-dimensional points of the electromagnetic signal's physical coordinates in the camera coordinate system to the corresponding pixel coordinates in the image coordinate system through perspective projection. The image feature extraction unit is used to perform edge detection on the image frame according to the Sobel edge detection algorithm and calculate the edge intensity Ged of each pixel in the image frame, specifically: Among them, Ged i (x, y) represents the edge intensity at pixel (x, y) in the i-th image frame. x (x, y) and I y(x, y) represent the gradients of the frame image in the x-axis and y-axis directions, respectively; The electromagnetic feature extraction unit is used to perform bandpass filtering on the electromagnetic signal to remove high-frequency noise and low-frequency interference, and to extract abnormal peaks in the electromagnetic signal using a peak detection algorithm to reflect defect areas in the wire rope. Then, the electromagnetic signal response amplitude Rma is extracted from the processed electromagnetic signal. Specifically: Where Sf is the filtered electromagnetic signal, and max represents the maximum index sign.

[0010] Preferably, the matching degree evaluation module includes a modal matching unit and a matching evaluation unit; The modal matching unit is used to perform matching degree analysis on edge intensity Ged and electromagnetic signal response amplitude Rma, and calculate modal feature matching degree MAT, which is used to analyze the spatial correlation between visual features and electromagnetic features and quantify the similarity between the two modes, as follows; ; Where MAT(i,j) represents the modal feature matching degree between the i-th image frame and the j-th electromagnetic signal segment, Ged i (x, y) represents the edge intensity at pixel (x, y) in the i-th image frame, Rma j (x, y) represents the electromagnetic signal response amplitude of the j-th segment of the electromagnetic signal at pixel (x, y) in the image frame.

[0011] Preferably, the matching evaluation unit is used to calculate the mean modal feature matching degree MAT when the historical visual image and electromagnetic signal are matched successfully using statistical methods, and to preset a matching degree qualification threshold PY based on the mean, and then perform matching degree evaluation with the obtained modal feature matching degree MAT. When the modal feature matching degree MAT < the matching degree qualification threshold PY, it indicates that there is a matching deviation between the visual image and the electromagnetic signal. At this time, a data calibration command is issued, and the time offset of the visual frame and the electromagnetic signal is calibrated by interpolation and retiming methods, and iterative analysis is performed by the spatial mapping module. If the modal feature matching degree MAT is greater than or equal to the matching degree qualification threshold PY, it means that the visual image and the electromagnetic signal are matched successfully, and the timing synchronization error analysis is triggered.

[0012] Preferably, the timing analysis module includes a timing error analysis unit and a timing error evaluation unit; The timing error analysis unit is used to perform error analysis on all image frames within the sliding window Wt based on the time offset ΔTve between each image frame and each segment of electromagnetic signal when the matching degree evaluation is qualified. The drift error value SYN is obtained by calculation, as follows. ; Wherein, ΔTve i This represents the time offset between the i-th image frame and the corresponding electromagnetic signal segment.

[0013] Preferably, the timing error assessment unit is used to calculate the average drift error value SYN when there is no drift in the historical timing synchronization according to the statistical method, and to preset the drift error standard threshold PW based on the average value, and then perform timing drift assessment with the obtained drift error value SYN. When the drift error value SYN ≤ drift error standard threshold PW, it means that there is no drift in timing synchronization, and the current program continues to run. When the drift error value SYN > the drift error standard threshold PW, it indicates that there is a drift in the timing synchronization. At this time, the error between the current image frame and the electromagnetic signal band is filled with missing data using a linear interpolation method, and iterative analysis is performed through the spatial mapping module.

[0014] Preferably, the feature fusion module includes a feature fusion unit and an output feedback unit; The feature fusion unit is used to merge the aligned image frames and electromagnetic signal bands through tensor splicing when the temporal drift assessment indicates that there is no drift in the temporal synchronization, forming a dual-modal fusion tensor Ff and inputting it into a multimodal convolutional neural network CNN. Based on the feature recognition capability of the multimodal convolutional neural network CNN, the dual-modal fusion tensor Ff is classified, and the defect category label, location coordinates and fused defect confidence are output respectively. The output feedback unit is used to organize the output defect category labels, location coordinates and fused defect confidence scores, automatically generate a defect report based on the classification results and generate a structured data format to push to the user terminal of relevant maintenance personnel, and mark the defect area on the user interface. The defect report includes the defect type, location, severity, and scope of impact.

[0015] This invention provides an AI-based fusion analysis system for the visual appearance and electromagnetic detection data of steel wire ropes. It offers the following advantages: (1) This system acquires image frames and electromagnetic signals of the wire rope simultaneously through visual imaging and electromagnetic signal capture. The image frames and electromagnetic signals are timestamped during acquisition, and all acquired data are arranged chronologically and synchronized to ensure time alignment between the image frames and electromagnetic signal data sources. This guarantees that the image frames and electromagnetic signals reflect the state of the wire rope at the same time, avoiding analysis errors caused by time differences. By calculating the instantaneous velocity Vr of the wire rope, the corresponding spatial length of each image frame and each segment of electromagnetic signal is determined, solving the problem of time asynchrony in spatial mapping.

[0016] (2) During data analysis, the system avoids inaccurate matching due to time differences by calculating the time offset ΔTve between the image frame and the electromagnetic signal. The time difference between each image frame and the electromagnetic signal is precisely calculated. This time offset processing ensures minimal time error between the image frame and the signal data, providing more accurate input data for subsequent defect detection. After spatial mapping of the image frame and signal data, feature extraction is performed to obtain the edge intensity Ged of the image frame and the electromagnetic signal response amplitude Rma. The similarity between the edge intensity Ged and the electromagnetic signal response amplitude Rma is quantified, and the modal feature matching degree MAT is calculated to evaluate the matching degree. Spatial correlation is established between different modes, and mismatch between vision and electromagnetic signals is avoided when identifying defects in complex environments. This feature extraction method improves the ability to identify complex defects, including details such as fine cracks, wear, or corrosion points.

[0017] (3) When the matching degree assessment indicates that the visual image and electromagnetic signal are matched satisfactorily, the system performs error analysis on all image frames within the sliding window Wt based on the time offset ΔTve between each image frame and each electromagnetic signal segment. It calculates the drift error value SYN and performs temporal drift assessment. If temporal synchronization drift is detected, it is corrected using interpolation techniques. This mechanism prevents the time offset from affecting the final analysis results and improves the accuracy of multimodal data fusion. Finally, after temporal synchronization and spatial alignment, the image frames and electromagnetic signals are fused into bimodal data and input into a deep learning network for defect classification analysis. The generated defect report includes information such as defect type, location, and severity, which is fed back to maintenance personnel, providing detailed guidance for the maintenance and repair of the wire rope. Attached Figure Description

[0018] Figure 1 This is a schematic diagram of the AI-based fusion analysis system for steel wire rope visual appearance and electromagnetic detection data according to the present invention. Figure 2 This is a block diagram illustrating the operating principle of the AI-based fusion analysis system for steel wire rope visual appearance and electromagnetic detection data. Detailed Implementation

[0019] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention. Example 1

[0020] Please see Figure 1This invention provides an AI-based fusion analysis system for the visual appearance and electromagnetic detection data of steel wire ropes. To achieve the above objectives, this invention is implemented through the following technical solutions: including a data acquisition module, a spatial mapping module, a feature extraction module, a matching degree evaluation module, a time series analysis module, and a feature fusion module. The data acquisition module is used to acquire wire rope image frames and electromagnetic signals and perform simultaneous time indexing. The spatial mapping module is used to collect the instantaneous velocity Vr of the wire rope at different time points, and to calculate the running space length of the wire rope corresponding to the image frame and the electromagnetic signal, respectively. The feature extraction module is used to calculate the time offset ΔTve between each image frame and each electromagnetic signal, and to spatially map the positions of the steel wire ropes corresponding to the image frames and electromagnetic signals. Then, it extracts the pixel edge intensity Ged and the electromagnetic signal response amplitude Rma from the mapped image frames and electromagnetic signals. The matching degree evaluation module is used to calculate the modal feature matching degree MAT and perform matching degree evaluation. The time series analysis module is used to calculate the drift error value SYN when the matching degree evaluation is qualified, and to perform time series drift evaluation; The feature fusion module is used to construct a dual-modal fusion tensor Ff using aligned image frames and electromagnetic signals when there is no drift in temporal synchronization, and then generate a defect report after classification based on a multimodal convolutional neural network (CNN).

[0021] In this embodiment, the data acquisition module synchronously captures image frames and electromagnetic signals of the wire rope, ensuring precise temporal alignment of visual and electromagnetic data. This synchronized data acquisition method resolves the time difference problem between the two data sources. Compared to the data asynchrony problem in traditional methods, it avoids time mismatch between image frames and electromagnetic signals, ensuring detection accuracy and data consistency. The spatial mapping module measures the instantaneous velocity Vr of the wire rope in real time and combines it with the acquisition timestamps of image frames and electromagnetic signals to calculate the spatial length of the wire rope at different time points, mapping the spatial relationship between image frames and electromagnetic signals. This spatial synchronization overcomes the errors in spatial mapping in traditional methods, improving the spatial accuracy of wire rope defect detection. The feature extraction module calculates the time offset ΔTve between each image frame and each segment of electromagnetic signal, finding the optimal matching point between each image frame and each segment of electromagnetic signal through time difference calculation, reducing matching errors caused by temporal asynchrony. It then performs spatial mapping on the wire rope positions corresponding to the image frames and electromagnetic signals, and extracts the pixel edge intensity Ged and electromagnetic signal response amplitude Rma from the mapped image frames and electromagnetic signals. Traditional detection methods often rely on single-modal data, while this system quantifies the matching degree between image frames and electromagnetic signals through a matching degree evaluation module. By calculating the modal feature matching degree (MAT) and evaluating the matching degree, the accuracy of defect identification is improved. This matching degree evaluation method not only eliminates the temporal differences between visual and electromagnetic signals but also avoids spatial mismatches between image frames and electromagnetic signals. When the matching degree evaluation indicates that the visual image and electromagnetic signal are a qualified match, the temporal analysis module performs error analysis on all image frames within the sliding window Wt based on the time offset ΔTve between each image frame and each electromagnetic signal segment. By calculating the drift error value SYN, the consistency of data on the time axis is ensured, and temporal drift evaluation is performed. When temporal synchronization drift exists, the error between the current image frame and the electromagnetic signal band is filled with missing data using a linear interpolation method, automatically performing error compensation to avoid the accuracy of detection results being affected by temporal issues. The feature fusion module performs dual-modal fusion of aligned and synchronized image frames and electromagnetic signals and performs defect classification based on a multimodal convolutional neural network. This process combines the complementary advantages of visual and electromagnetic data. The system improves detection accuracy and classification accuracy through multimodal data fusion, and can identify cracks, wear, corrosion and other defects in wire ropes, and generate detailed defect reports, providing strong data support and decision-making basis for wire rope maintenance. Example 2

[0022] Please refer to Figure 2 Specifically: the data acquisition module includes a visual image acquisition unit, an electromagnetic signal acquisition unit, and a time index management unit; The visual image acquisition unit is used to continuously capture images of the surface of the steel wire rope passing through the detection channel using an industrial camera, generating an image frame sequence Iv, where Iv = {Iv1, Iv2, ..., Iv...}. n}, where n represents the number of image frames, and the acquisition timestamp Tv of each image frame is recorded, Tv={Tv1, Tv2, ..., Tv n}; The electromagnetic signal acquisition unit is used to generate an electromagnetic signal sequence Ee, where Ee = {Ee1, Ee2, ..., Ee...}, based on the electromagnetic signals collected by the electromagnetic sensor during the operation of the wire rope. m}, where m represents the number of electromagnetic signals collected, and the timestamp Te of each electromagnetic signal is recorded, Te = {Te1, Te2, ..., Te}. m}; The time index management unit is used to sort the image frame sequence Iv and the electromagnetic signal sequence Ee in chronological order, generate a unified time index set containing timestamps Tv and Te, and embed global time series labels for alignment, mapping and matching with a unified time reference.

[0023] In this embodiment, the data acquisition module achieves precise synchronous acquisition and time alignment of wire rope surface images and electromagnetic signals through the collaborative work of three sub-modules: visual image acquisition, electromagnetic signal acquisition, and time index management. The visual image acquisition unit continuously captures images of the wire rope surface using an industrial camera, generating an image frame sequence and recording a timestamp for each frame. The electromagnetic signal acquisition unit synchronously acquires electromagnetic signals during operation using electromagnetic sensors and records a timestamp for each signal. The time index management unit sorts the image frames and electromagnetic signal sequences chronologically, generating a unified time index set, and precisely aligns and matches the two by embedding global time-series tags. This approach ensures a high degree of temporal and spatial synchronization between image data and electromagnetic signals, eliminating data inconsistencies caused by time deviations in traditional methods. It provides reliable and accurate input data for subsequent feature extraction, matching degree evaluation, and defect detection, significantly improving detection accuracy and efficiency. Example 3

[0024] Please refer to Figure 2 Specifically: the spatial mapping module includes a timing synchronization unit and a segment calculation unit; The timing synchronization unit is used to obtain the instantaneous speed Vr of the wire rope at different time points based on the encoder installed at the front end of the detection area, and to perform timing alignment and interpolation synchronization of all speed sampling points by reading the image frame acquisition timestamp Tv and the electromagnetic signal acquisition timestamp Te. The segment calculation unit is used to perform velocity integration calculations within the sampling time interval of the image frame and the electromagnetic segment based on the instantaneous velocity value Vr combined with the image frame acquisition timestamp Tv and the electromagnetic signal acquisition timestamp Te, respectively, to obtain the steel wire rope running space length segment corresponding to each image frame and each electromagnetic segment, as follows; The length of the wire rope running space corresponding to the i-th image frame is Lv. i The calculation is as follows: Where Vr(t) represents the instantaneous velocity at time t, and Tv i Tv represents the acquisition timestamp of the i-th image frame. i-1 dt represents the acquisition timestamp of the (i-1)th image frame, which is the acquisition timestamp of the image frame preceding the i-th image frame, and dt represents the time integral variable. The length of the wire rope running space corresponding to the j-th segment of electromagnetic signal is Le. j The calculation is as follows: Where Vr(t) represents the instantaneous velocity at time t, and Te j Te represents the timestamp of the acquisition of the j-th segment of electromagnetic signal. j-1 dt represents the acquisition timestamp of the (j-1)th segment of electromagnetic signal, which is the acquisition timestamp of the previous segment of electromagnetic signal before the j-th segment of electromagnetic signal, and dt represents the time integration variable.

[0025] In this embodiment, the timing synchronization unit obtains the instantaneous velocity Vr of the wire rope based on the encoder and synchronizes the acquisition timestamps of the image frames and electromagnetic signals through interpolation to ensure timing consistency across different data sources. The segment calculation unit uses the instantaneous velocity and timestamp information to calculate the wire rope running space length corresponding to each image frame and electromagnetic signal segment, resolving the spatial and temporal differences between image frames and electromagnetic signals. Through precise synchronization and calculation, the problem of decreased detection accuracy caused by timing errors and spatial mismatches in traditional methods is eliminated, improving the ability to identify wire rope defects and the accuracy of data processing, providing technical support for real-time monitoring and wire rope maintenance. Example 4

[0026] Please refer to Figure 2 Specifically: the feature extraction module includes a time offset calculation unit and a feature extraction unit; The time offset calculation unit is used to calculate the time offset ΔTve between each image frame and each electromagnetic signal segment based on the image frame acquisition timestamp Tv and the electromagnetic signal acquisition timestamp Te. It determines the minimum time difference between each visual frame and all electromagnetic signal segments by comparing their timestamps, thereby obtaining the time offset between the visual image and the electromagnetic signal. Specifically: ,in, Represents the smallest index symbol, indicating the frame timestamp Tv of the i-th image frame. iThe electromagnetic signal with the smallest time difference.

[0027] The feature extraction unit includes a spatial mapping unit, an image feature extraction unit, and an electromagnetic feature extraction unit; The spatial mapping unit is used to calibrate the industrial camera to obtain the camera's intrinsic and extrinsic parameter matrices. The extrinsic parameter matrix is ​​used to transform the physical coordinates of the electromagnetic signal to the camera coordinate system. The camera's intrinsic parameter matrix is ​​used to map the three-dimensional points of the electromagnetic signal's physical coordinates in the camera coordinate system to the corresponding pixel coordinates in the image coordinate system through perspective projection. The image feature extraction unit is used to perform edge detection on the image frame according to the Sobel edge detection algorithm and calculate the edge intensity Ged of each pixel in the image frame, specifically: Among them, Ged i (x, y) represents the edge intensity at pixel (x, y) in the i-th image frame. x (x, y) and I y (x, y) represent the gradients of the frame image in the x-axis and y-axis directions, respectively; The electromagnetic feature extraction unit is used to perform bandpass filtering on the electromagnetic signal to remove high-frequency noise and low-frequency interference, and to extract abnormal peaks in the electromagnetic signal using a peak detection algorithm to reflect defect areas in the wire rope. Then, the electromagnetic signal response amplitude Rma is extracted from the processed electromagnetic signal. Specifically: Where Sf is the filtered electromagnetic signal, and max represents the maximum index sign.

[0028] In this embodiment, the time offset calculation unit calculates the time offset ΔTve between each image frame and each electromagnetic signal by comparing the timestamps of the image frames and the electromagnetic signals. This ensures precise temporal synchronization between the image frames and the electromagnetic signals, providing a consistent time reference for subsequent analysis. The feature extraction unit aligns the electromagnetic signals with the image coordinate system through the spatial mapping unit, eliminating spatial errors and ensuring spatial consistency of the data. The image feature extraction unit extracts edge intensity from the image using the Sobel edge detection algorithm to identify detailed features on the surface of the wire rope. The electromagnetic feature extraction unit removes noise through bandpass filtering and extracts abnormal peaks in the electromagnetic signals, reflecting defect areas in the wire rope. This series of data processing and feature extraction steps enhances the system's defect identification capability, ensuring efficient and accurate detection of wire rope defects and avoiding false detections and missed detections caused by data inconsistencies in traditional methods. Example 5

[0029] Please refer to Figure 2 Specifically: the matching degree evaluation module includes a modal matching unit and a matching evaluation unit; The modal matching unit is used to perform matching degree analysis on edge intensity Ged and electromagnetic signal response amplitude Rma, and calculate modal feature matching degree MAT, which is used to analyze the spatial correlation between visual features and electromagnetic features and quantify the similarity between the two modes, as follows; ; Where MAT(i,j) represents the modal feature matching degree between the i-th image frame and the j-th electromagnetic signal segment, Ged i (x, y) represents the edge intensity at pixel (x, y) in the i-th image frame, Rma j (x, y) represents the electromagnetic signal response amplitude of the j-th segment of the electromagnetic signal at pixel (x, y) in the image frame.

[0030] The matching evaluation unit is used to calculate the average modal feature matching degree MAT when the historical visual image and electromagnetic signal are matched successfully using statistical methods, and to preset the matching degree qualification threshold PY based on the average value, and then perform matching degree evaluation with the obtained modal feature matching degree MAT. When the modal feature matching degree MAT < the matching degree qualification threshold PY, it indicates that there is a matching deviation between the visual image and the electromagnetic signal. At this time, a data calibration command is issued, and the time offset of the visual frame and the electromagnetic signal is calibrated by interpolation and retiming methods, and iterative analysis is performed by the spatial mapping module. If the modal feature matching degree MAT is greater than or equal to the matching degree qualification threshold PY, it means that the visual image and the electromagnetic signal are matched successfully, and the timing synchronization error analysis is triggered.

[0031] In this embodiment, the modal matching unit analyzes the edge intensity of the image frame and the amplitude of the electromagnetic signal response to calculate the modal feature matching degree (MAT) between them, quantifies the spatial correlation between the image and the electromagnetic signal, and identifies the similarity between the image frame and the electromagnetic signal. The derivation of this formula is based on the original formula of the cosine similarity method; cosine similarity is a standard method for measuring the similarity of the angle between two vectors in space, and the cosine similarity formula is: , where A i and B i There are two vectors A = (A1, A2, ..., A...). n ) and B = (B1, B2, ..., B nIn the i-th dimension, the numerator is the inner product of two vectors, and the denominator is the norm of each vector. In this formula, A corresponds to the edge intensity Ged(x,y) in the image frame, and B corresponds to the electromagnetic signal response amplitude Rma(x,y), ultimately obtaining the modal feature matching degree MAT. The matching evaluation unit calculates the average modal feature matching degree MAT when historical visual images and electromagnetic signals are successfully matched, and presets a matching degree qualification threshold PY based on the average value. It then evaluates the matching degree against the obtained modal feature matching degree MAT. When the modal feature matching degree MAT does not reach the matching degree qualification threshold PY, the system performs data calibration through interpolation and retiming methods to further correct time offsets and ensure data synchronization. When the modal feature matching degree MAT is greater than or equal to the matching degree qualification threshold PY, it triggers timing synchronization error analysis to ensure data temporal consistency. This improves the matching accuracy of visual and electromagnetic data and avoids the impact of time and space errors on defect identification. Example 6

[0032] Please refer to Figure 2 Specifically: the timing analysis module includes a timing error analysis unit and a timing error evaluation unit; The timing error analysis unit is used to perform error analysis on all image frames within the sliding window Wt based on the time offset ΔTve between each image frame and each segment of electromagnetic signal when the matching degree evaluation is qualified. The drift error value SYN is obtained by calculation, as follows. ; Wherein, ΔTve i This represents the time offset between the i-th image frame and the corresponding electromagnetic signal segment.

[0033] The timing error assessment unit is used to calculate the average drift error value SYN when there is no drift in the historical timing synchronization according to the statistical method, and to preset the drift error standard threshold PW based on the average value, and then perform timing drift assessment with the obtained drift error value SYN. When the drift error value SYN ≤ drift error standard threshold PW, it means that there is no drift in timing synchronization, and the current program continues to run. When the drift error value SYN > the drift error standard threshold PW, it indicates that there is a drift in the timing synchronization. At this time, the error between the current image frame and the electromagnetic signal band is filled with missing data using a linear interpolation method, and iterative analysis is performed through the spatial mapping module.

[0034] In this embodiment, the timing error analysis unit is used to perform error analysis on all image frames within the sliding window Wt when the matching degree evaluation indicates that the visual image and electromagnetic signal are matched satisfactorily. This analysis is based on the time offset ΔTve between each image frame and each segment of the electromagnetic signal. The drift error value SYN is calculated to ensure the accuracy of the timing alignment. The drift error value SYN is calculated by accumulating the time offset ΔTve between each image frame and the electromagnetic signal within the sliding window. This calculation originates from the field of multimodal signal processing and is based on the impact of the time offset between the image frame and the electromagnetic signal segment on subsequent data fusion during alignment on the time axis. The formula quantitatively analyzes the synchronization error between the two modal signals (image frame and electromagnetic signal) by calculating the time offset ΔTve. i The time difference between the image and the electromagnetic signal is represented by the sum of these offsets to obtain the total drift error value SYN, which is used to measure the accuracy of timing synchronization. The timing error evaluation unit calculates the average drift error value SYN in historical data using statistical methods, preset as the drift error standard threshold PW, and evaluates the acquired drift error value SYN for timing drift. When the drift error value SYN exceeds the drift error standard threshold PW, missing data is filled using a linear interpolation method to ensure timing synchronization. This process avoids data inconsistency problems caused by timing drift and ensures precise timing alignment between image frames and electromagnetic signal data. Through this mechanism, the system can more accurately classify and identify defects during multimodal data fusion, improving the efficiency and reliability of wire rope defect detection. Example 7

[0035] Please refer to Figure 2 Specifically: the feature fusion module includes a feature fusion unit and an output feedback unit; The feature fusion unit is used to merge the aligned image frames and electromagnetic signal bands through tensor splicing when the temporal drift assessment indicates that there is no drift in the temporal synchronization, forming a dual-modal fusion tensor Ff and inputting it into a multimodal convolutional neural network CNN. Based on the feature recognition capability of the multimodal convolutional neural network CNN, the dual-modal fusion tensor Ff is classified, and the defect category label, location coordinates and fused defect confidence are output respectively. The output feedback unit is used to organize the output defect category labels, location coordinates and fused defect confidence scores, automatically generate a defect report based on the classification results and generate a structured data format to push to the user terminal of relevant maintenance personnel, and mark the defect area on the user interface. The defect report includes the defect type, location, severity, and scope of impact.

[0036] In this embodiment, the feature fusion module performs tensor concatenation of time-synchronized image frames and electromagnetic signal bands to form a dual-modal fusion tensor Ff, which is then input into a multimodal convolutional neural network (CNN) for classification. Based on its feature recognition capabilities of the fused data, the network accurately outputs defect category labels, location coordinates, and the fused defect confidence score. This process ensures effective fusion between the image and electromagnetic signals. The output feedback unit automatically generates a detailed defect report based on the classification results. The report includes the defect type, location, severity, and impact range, and pushes the structured data to relevant maintenance personnel for rapid response and repair. This module not only improves the accuracy and efficiency of wire rope defect detection but also optimizes the data processing flow, reduces manual intervention, and significantly enhances the usability and practical application value of the detection results, contributing to real-time monitoring and timely maintenance of wire ropes.

[0037] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. An AI-based fusion analysis system for visual appearance and electromagnetic detection data of steel wire ropes, characterized in that: It includes a data acquisition module, a spatial mapping module, a feature extraction module, a matching degree evaluation module, a time series analysis module, and a feature fusion module; The data acquisition module is used to acquire wire rope image frames and electromagnetic signals and perform simultaneous time indexing to generate an image frame sequence Iv and an electromagnetic signal sequence Ee. The spatial mapping module is used to collect the instantaneous velocity Vr of the wire rope at different time points, and to calculate the running space length of the wire rope corresponding to the image frame and the electromagnetic signal, respectively. The feature extraction module is used to calculate the time offset ΔTve between each image frame and each electromagnetic signal, and to spatially map the positions of the steel wire ropes corresponding to the image frames and electromagnetic signals. Then, it extracts the pixel edge intensity Ged and the electromagnetic signal response amplitude Rma from the mapped image frames and electromagnetic signals. The matching degree evaluation module is used to calculate the modal feature matching degree MAT and perform matching degree evaluation. The time series analysis module is used to calculate the drift error value SYN when the matching degree evaluation is qualified, and to perform time series drift evaluation; The feature fusion module is used to construct a dual-modal fusion tensor Ff using aligned image frames and electromagnetic signals when there is no drift in temporal synchronization, and then generate a defect report after classification based on a multimodal convolutional neural network (CNN).

2. The AI-based fusion analysis system for steel wire rope visual appearance and electromagnetic detection data according to claim 1, characterized in that: The data acquisition module includes a visual image acquisition unit, an electromagnetic signal acquisition unit, and a time index management unit; The visual image acquisition unit is used to continuously capture images of the surface of the steel wire rope passing through the detection channel using an industrial camera, generating an image frame sequence Iv, where Iv = {Iv1, Iv2, ..., Iv...}. n }, where n represents the number of image frames, and the acquisition timestamp Tv of each image frame is recorded, Tv={Tv1, Tv2, ..., Tv n }; The electromagnetic signal acquisition unit is used to generate an electromagnetic signal sequence Ee, where Ee = {Ee1, Ee2, ..., Ee...}, based on the electromagnetic signals collected by the electromagnetic sensor during the operation of the wire rope. m }, where m represents the number of electromagnetic signals collected, and the timestamp Te of each electromagnetic signal is recorded, Te = {Te1, Te2, ..., Te}. m }; The time index management unit is used to sort the image frame sequence Iv and the electromagnetic signal sequence Ee in chronological order, generate a unified time index set containing timestamps Tv and Te, and embed global time series labels for alignment, mapping and matching with a unified time reference.

3. The AI-based fusion analysis system for steel wire rope visual appearance and electromagnetic detection data according to claim 2, characterized in that: The spatial mapping module includes a timing synchronization unit and a segment calculation unit; The timing synchronization unit is used to obtain the instantaneous speed Vr of the wire rope at different time points based on the encoder installed at the front end of the detection area, and to perform timing alignment and interpolation synchronization of all speed sampling points by reading the image frame acquisition timestamp Tv and the electromagnetic signal acquisition timestamp Te. The segment calculation unit is used to perform velocity integration calculations within the sampling time interval of the image frame and the electromagnetic segment based on the instantaneous velocity value Vr combined with the image frame acquisition timestamp Tv and the electromagnetic signal acquisition timestamp Te, respectively, to obtain the steel wire rope running space length segment corresponding to each image frame and each electromagnetic segment, as follows; The length of the wire rope running space corresponding to the i-th image frame is Lv. i The calculation is as follows: Where Vr(t) represents the instantaneous velocity at time t, and Tv i Tv represents the acquisition timestamp of the i-th image frame. i-1 dt represents the acquisition timestamp of the (i-1)th image frame, and dt represents the time integration variable; The length of the wire rope running space corresponding to the j-th segment of electromagnetic signal is Le. j The calculation is as follows: Where Vr(t) represents the instantaneous velocity at time t, and Te j Te represents the timestamp of the acquisition of the j-th segment of electromagnetic signal. j-1 dt represents the acquisition timestamp of the (j-1)th segment of electromagnetic signal, and dt represents the time integration variable.

4. The AI-based fusion analysis system for steel wire rope visual appearance and electromagnetic detection data according to claim 3, characterized in that: The feature extraction module includes a time offset calculation unit and a feature extraction unit; The time offset calculation unit is used to calculate the time offset ΔTve between each image frame and each electromagnetic signal based on the image frame acquisition timestamp Tv and the electromagnetic signal acquisition timestamp Te. Specifically: ,in, Represents the smallest index symbol, indicating the frame timestamp Tv of the i-th image frame. i The electromagnetic signal with the smallest time difference.

5. The AI-based fusion analysis system for steel wire rope visual appearance and electromagnetic detection data according to claim 4, characterized in that: The feature extraction unit includes a spatial mapping unit, an image feature extraction unit, and an electromagnetic feature extraction unit; The spatial mapping unit is used to calibrate the industrial camera to obtain the camera's intrinsic and extrinsic parameter matrices. The extrinsic parameter matrix is ​​used to transform the physical coordinates of the electromagnetic signal to the camera coordinate system. The camera's intrinsic parameter matrix is ​​used to map the three-dimensional points of the electromagnetic signal's physical coordinates in the camera coordinate system to the corresponding pixel coordinates in the image coordinate system through perspective projection. The image feature extraction unit is used to perform edge detection on the image frame according to the Sobel edge detection algorithm and calculate the edge intensity Ged of each pixel in the image frame, specifically: Among them, Ged i (x, y) represents the edge intensity at pixel (x, y) in the i-th image frame. x (x, y) and I y (x, y) represent the gradients of the frame image in the x-axis and y-axis directions, respectively; The electromagnetic feature extraction unit is used to perform bandpass filtering on the electromagnetic signal to remove high-frequency noise and low-frequency interference, and to extract abnormal peaks in the electromagnetic signal using a peak detection algorithm to reflect defect areas in the wire rope. Then, the electromagnetic signal response amplitude Rma is extracted from the processed electromagnetic signal. Specifically: Where Sf is the filtered electromagnetic signal, and max represents the maximum index sign.

6. The AI-based fusion analysis system for steel wire rope visual appearance and electromagnetic detection data according to claim 5, characterized in that: The matching degree evaluation module includes a modality matching unit and a matching evaluation unit; The modal matching unit is used to perform matching degree analysis on edge intensity Ged and electromagnetic signal response amplitude Rma, and calculate modal feature matching degree MAT, which is used to analyze the spatial correlation between visual features and electromagnetic features and quantify the similarity between the two modes, as follows; ； Where MAT(i,j) represents the modal feature matching degree between the i-th image frame and the j-th electromagnetic signal segment, Ged i (x, y) represents the edge intensity at pixel (x, y) in the i-th image frame, Rma j (x, y) represents the electromagnetic signal response amplitude of the j-th segment of the electromagnetic signal at pixel (x, y) in the image frame.

7. The AI-based fusion analysis system for steel wire rope visual appearance and electromagnetic detection data according to claim 6, characterized in that: The matching evaluation unit is used to calculate the average modal feature matching degree MAT when the historical visual image and electromagnetic signal are matched successfully using statistical methods, and to preset the matching degree qualification threshold PY based on the average value, and then perform matching degree evaluation with the obtained modal feature matching degree MAT. When the modal feature matching degree MAT < the matching degree qualification threshold PY, it indicates that there is a matching deviation between the visual image and the electromagnetic signal. At this time, a data calibration command is issued, and the time offset of the visual frame and the electromagnetic signal is calibrated by interpolation and retiming methods, and iterative analysis is performed by the spatial mapping module. If the modal feature matching degree MAT is greater than or equal to the matching degree qualification threshold PY, it means that the visual image and the electromagnetic signal are matched successfully, and the timing synchronization error analysis is triggered.

8. The AI-based fusion analysis system for steel wire rope visual appearance and electromagnetic detection data according to claim 7, characterized in that: The timing analysis module includes a timing error analysis unit and a timing error evaluation unit; The timing error analysis unit is used to perform error analysis on all image frames within the sliding window Wt based on the time offset ΔTve between each image frame and each segment of electromagnetic signal when the matching degree evaluation is qualified. The drift error value SYN is obtained by calculation, as follows. ； Wherein, ΔTve i This represents the time offset between the i-th image frame and the corresponding electromagnetic signal segment.

9. The AI-based fusion analysis system for steel wire rope visual appearance and electromagnetic detection data according to claim 8, characterized in that: The timing error assessment unit is used to calculate the average drift error value SYN when there is no drift in the historical timing synchronization according to the statistical method, and to preset the drift error standard threshold PW based on the average value, and then perform timing drift assessment with the obtained drift error value SYN. When the drift error value SYN ≤ drift error standard threshold PW, it means that there is no drift in timing synchronization, and the current program continues to run. When the drift error value SYN > the drift error standard threshold PW, it indicates that there is a drift in the timing synchronization. At this time, the error between the current image frame and the electromagnetic signal band is filled with missing data using a linear interpolation method, and iterative analysis is performed through the spatial mapping module.

10. The AI-based fusion analysis system for steel wire rope visual appearance and electromagnetic detection data according to claim 9, characterized in that: The feature fusion module includes a feature fusion unit and an output feedback unit; The feature fusion unit is used to merge the aligned image frames and electromagnetic signal bands through tensor splicing when the temporal drift assessment indicates that there is no drift in the temporal synchronization, forming a dual-modal fusion tensor Ff and inputting it into a multimodal convolutional neural network CNN. Based on the feature recognition capability of the multimodal convolutional neural network CNN, the dual-modal fusion tensor Ff is classified, and the defect category label, location coordinates and fused defect confidence are output respectively. The output feedback unit is used to organize the output defect category labels, location coordinates and fused defect confidence scores, automatically generate a defect report based on the classification results and generate a structured data format to push to the user terminal of relevant maintenance personnel, and mark the defect area on the user interface. The defect report includes the defect type, location, severity, and scope of impact.

Images