Mechanical arm tail end dual-light fusion detection data management system and method

Abstract

The invention discloses a mechanical arm tail end dual-light fusion detection data management system and method, and belongs to the technical field of industrial automatic detection. A visible light camera and an infrared camera are integrated at the tail end of a mechanical arm, discrete monitoring points are set according to the motion path of the mechanical arm, and the mechanical arm is based on pose data output by a motion controller; when moving to a preset monitoring point, synchronously triggering dual-light image acquisition, and carrying out space-time alignment on the acquired data; performing semantic segmentation processing on the visible light image, screening out a cable image from the visible light image to construct a cable identification model, obtaining boundary coordinates of a cable, performing feature matching with an infrared image, mapping a cable region to the infrared image, extracting a corresponding temperature data matrix, and calculating a maximum temperature difference value of the surface of the cable; judging an abnormal region in combination with a preset temperature gradient threshold value; and according to the anomaly detection result, the mechanical arm is triggered to dynamically adjust the detection path, the abnormal point is focused for reinspection, and anomaly early warning prompt is carried out according to the reinspection result.

Description

Technical Field

[0001] The present invention relates to the field of industrial automation detection technology, and in particular to a data management system and method for dual-light fusion detection at the end of a robotic arm. Background Art

[0002] In scenarios such as power equipment and industrial facilities, accurate monitoring of cable surface temperature is a key step in preventing fires and ensuring stable equipment operation. Currently, mainstream detection methods rely on infrared thermal imaging technology and fixed temperature sensor networks. Specifically, infrared cameras obtain equipment thermal distribution images in a non-contact manner, enabling rapid location of areas with abnormal temperature rises; distributed sensors, on the other hand, provide continuous temperature data collection. In recent years, some scenarios have introduced inspection robotic arms equipped with inspection equipment to improve detection coverage in complex environments. This approach, to a certain extent, addresses the low efficiency and high risk of manual inspections, and forms the technical foundation for current cable temperature monitoring.

[0003] However, in actual applications, although infrared thermal imaging technology can capture temperature distribution, its image resolution is low, making it difficult to accurately identify the fine structure of the cable surface, especially in environments where cables are densely arranged or the lighting is complex. The temperature sampling area is easily offset due to blurred boundaries. At the same time, although the robotic arm has expanded the detection range, its movement path usually relies on preset programs and cannot be dynamically adjusted according to real-time detection data. When a local temperature anomaly is detected, the robotic arm still runs along the original path, making it difficult to focus on the potential danger area for re-inspection in time, resulting in an increased missed detection rate. Summary of the Invention

[0004] The purpose of the present invention is to provide a data management system and method for dual-light fusion detection at the end of a robotic arm to solve the problems raised in the above background technology.

[0005] In order to solve the above technical problems, the present invention provides the following technical solution: a method for managing dual-light fusion detection data at the end of a robotic arm, comprising: Step S100: A visible light camera and an infrared camera are integrated at the end of the robotic arm. Discrete monitoring points are set according to the motion path of the robotic arm. Based on the posture data output by the motion controller, the robotic arm synchronously triggers dual-light image acquisition when the movement reaches the preset monitoring points, and the collected data is temporally and spatially aligned. Step S200: performing semantic segmentation on the visible light image, filtering cable images from the visible light image data to construct a basic data set, and building a cable recognition model through the U-Net architecture to obtain a cable boundary coordinate set; Step S300: performing feature matching on the cable boundary extracted from the visible light image and the infrared image, mapping the cable area to the infrared image, and extracting the corresponding temperature data matrix; Step S400: Calculate the maximum temperature difference on the cable surface based on the mapped infrared temperature data matrix, and determine the abnormal area in combination with the preset temperature gradient threshold; Step S500: According to the abnormality detection result, the robot arm is triggered to dynamically adjust the detection path, focus on the abnormal point for re-inspection, and issue an abnormality warning prompt based on the re-inspection result.

[0006] Furthermore, step S100 includes: Step S101: A visible light camera, an infrared camera, and a dual-light acquisition trigger are installed at the end of the robotic arm. The posture data of the robotic arm motion controller is used as a synchronization signal. Discrete detection points are set according to the motion path of the robotic arm. Each detection point corresponds to a posture data range centered on the theoretical posture and including position tolerance and posture tolerance. The dual-light acquisition trigger continuously receives the posture data output by the robotic arm motion controller and matches the current posture data with the posture data range corresponding to the preset detection point. When the posture matching condition of a certain detection point is met, the trigger simultaneously sends a pulse signal to drive the visible light camera and the infrared camera to collect images at the same time. Step S102: A high checkerboard calibration plate is placed vertically at the center of the overlapping fields of view of the visible light camera and the infrared camera. The dual-light camera is driven by a robotic arm to perform multi-view motion, and n sets of dual-modal images at different orientations, pitches, and rotation angles are collected. The visible light image and the infrared image are subjected to single-target calibration using the relevant functions in the OpenCV library. For the visible light image sequence, the sub-pixel coordinates of the checkerboard corner points are extracted, and their internal parameter matrix and distortion coefficients are calculated. For the infrared image sequence, the internal parameter matrix is ​​also calculated. The stereoCalibrate function is used to combine the dual-modal image data to solve the external parameter matrix containing the rotation matrix and translation vector. Step S103: In the real-time detection stage, the homography matrix is ​​calculated based on the internal and external parameters obtained by calibration, the infrared image pixel coordinates are mapped to the visible light image coordinate system through the perspective transformation formula, and the infrared image is perspective transformed. The transformed pixel values ​​are filled in through the bilinear interpolation algorithm to achieve spatial registration of the dual-modal images; the registered visible light image is data-bound with the infrared temperature matrix at the corresponding moment, and a timestamp is added to each set of data and stored in the database.

[0007] Furthermore, step S200 includes: Step S201: Filter images containing cables from the visible light image data as a basic dataset, annotate the cable areas in the dataset, create a corresponding pixel-level mask for each cable instance, and set the pixel values ​​in the mask to distinguish between cable areas and non-cable areas: the pixel value in the cable area is set to 1, and the pixel value in the non-cable area is set to 0; preprocess and data augment the annotated images, and divide them into training sets, validation sets, and test sets according to the ratio; Step S202: The cable recognition model is constructed using the U-Net architecture, which consists of three parts: an encoder, a bottleneck layer, and a decoder. The encoder part is composed of multiple DoubleConv modules alternating with maximum pooling layers. The DoubleConv module contains two convolutional layers, and each convolutional layer is connected to a normalization layer and a ReLU activation function. The encoder gradually extracts low-level to high-level features of the image through continuous alternating operations. The bottleneck layer is composed of a DoubleConv module to further extract high-level features of the image. The decoder part performs upsampling through the deconvolution layer to gradually restore the size of the feature map. The feature map of the corresponding layer of the encoder is spliced ​​with the feature map after deconvolution by using a jump connection method, and then passes through the DoubleConv module for feature extraction. Feature fusion; finally, the model converts the number of feature map channels to 1 through a 1x1 convolutional layer, and then uses the Sigmoid activation function to map the output value to between 0 and 1. The output value represents the probability that each pixel belongs to the cable area; the constructed model is trained using the training set, and the binary cross entropy loss function is selected to measure the difference between the model prediction result and the true mask. The Adam optimizer is used to adaptively adjust the learning rate of each parameter; in each training cycle, the model predicts the input image, calculates the loss between the prediction result and the true mask, calculates the gradient through the backpropagation algorithm, and then the optimizer updates the model parameters; during the training process, the validation set is used to regularly evaluate the model performance, monitor the accuracy, recall rate, and F1 value indicators, and the training is completed when the preset number of training cycles is reached; Step S203: Use the trained cable recognition model to predict the newly collected visible light image to obtain a predicted mask of the cable area in the image; binarize the predicted mask, set a threshold, set pixels with probability values ​​greater than the threshold to 1, and set pixels with probability values ​​less than the threshold to 0, to obtain the final cable area mask; process the binarized cable area mask through an edge detection algorithm to extract the boundary of the cable area, refine the extracted boundary, remove redundant points, and obtain the boundary coordinate set of the cable.

[0008] Furthermore, step S300 converts the acquired cable boundary coordinate set into the infrared image coordinate system through the homography matrix based on the spatial mapping relationship between the registered visible light image and the infrared image. For each coordinate point in the cable boundary coordinate set, its corresponding coordinate in the infrared image is calculated using the formula to determine the outline of the cable area in the infrared image: ; where (x v ,y v ) represents the pixel coordinates of the cable boundary coordinates in the visible light image coordinate system, (x m ,y m ) means (x v ,y v ) corresponds to the pixel coordinates in the infrared image coordinate system, H represents the homography matrix; from the temperature data matrix corresponding to the infrared image, the complete temperature data matrix T of the cable area is extracted, and the matrix element T mn The temperature value of the pixel at the mth row and nth column in the corresponding cable area; the extracted temperature data matrix T is interpolated and its resolution is adjusted to the same as the cable area mask of the visible light image using the bicubic interpolation algorithm.

[0009] Furthermore, step S400 includes: Step S401: Based on the temperature data matrix T processed in step S302, traverse all elements in the matrix and calculate the maximum temperature difference ΔTmax on the cable surface; at the same time, calculate the temperature gradient value of each pixel point and its eight neighboring pixels; for the pixel point with coordinates (i, j) in the matrix, the temperature gradient calculation formulas in the horizontal and vertical directions are: , ;in Indicates the temperature gradient value of pixel (i, j) in the horizontal direction, Represents the temperature gradient value of the pixel (i, j) in the vertical direction, T(i+1, j) represents the temperature value of the pixel at the coordinate (i+1, j) in the temperature matrix T, T(i-1, j) represents the temperature value of the pixel at the coordinate (i-1, j) in the temperature matrix T, T(i, j+1) represents the temperature value of the pixel at the coordinate (i, j+1) in the temperature matrix T, and T(i, j-1) represents the temperature value of the pixel at the coordinate (i, j-1) in the temperature matrix T; the comprehensive temperature gradient value G of the pixel (i, j) ij , according to the formula: ; By traversing all pixel points in the matrix, the temperature gradient matrix G of the entire cable area is obtained; Step S402: Compare the calculated maximum temperature difference ΔTmax with the preset maximum temperature difference threshold T t1 Compare and compare each element in the temperature gradient matrix G with the preset temperature gradient threshold Tt2 Compare; when ΔTmax>T is satisfied t1 and there are elements in G greater than T t2 When any one of the conditions is met, the corresponding area is determined as an abnormal area; mark the coordinate positions of the abnormal area in the infrared image and the visible light image, and generate an abnormal area mask. The pixel values of the abnormal area in the mask are set to 1, and the rest are 0.

[0010] Furthermore, step S500 includes: Step S501: Generate a three-layer spiral detection path centered on the coordinates of the abnormal point. After receiving the coordinate positions of the abnormal area marked in step S402, record the coordinates of the abnormal point in space as (x0, y0, z0). Use this point as the center of the three-layer spiral detection path and also as the starting point of the spiral detection path; set the initial radii of the first layer, the second layer, and the third layer of the spiral path as r1, r2, r3 respectively, and satisfy r1 < r2 < r3; for each layer of the spiral path, use the conversion method from polar coordinates to rectangular coordinates to generate path points: for the first layer of the detection path, in the polar coordinate system, the generation method of the spiral line is based on the formula: ρ = r1 + k1×θ; where ρ represents the polar radius, k1 represents the pitch coefficient of the first layer of the spiral line, controlling the increase of the polar radius when the spiral line rotates by a unit angle; θ represents the polar angle, which is the angle formed by rotating counterclockwise from the polar axis to the line segment connecting the pole and a certain point on the spiral line; set a series of angle values θ i , θ i ∈[0, 2aπ], a represents the number of turns of the first layer of the spiral line; convert the polar coordinates (ρ i , θ i ) to rectangular coordinates (x i , y i , z i ), where ρ i represents the polar radius corresponding to θ i , and through the formula: x i =x0 + ρ i ×cosθ i , y i =y0 + ρ i ×sinθ i , z i =z0; generate the coordinate points of the second layer and the third layer of the spiral path in the same way. By setting the corresponding angle value ranges respectively, and according to the above rectangular coordinate conversion formula, obtain the rectangular coordinates of each point on the second layer and the third layer of the spiral path, thus obtaining the complete three-layer spiral detection path; Step S502: Convert the generated three-layer spiral detection path into the motion instructions of the robotic arm. According to the kinematic model of the robotic arm, for each coordinate point (x i , y i , zi ) is converted into the angle values ​​of each joint of the robotic arm, and the inverse kinematics algorithm is used, combined with the connecting rod length and joint type of the robotic arm, to solve the angle that each joint needs to rotate, and these joint angle values ​​are arranged in sequence according to the order of the path points to generate the motion instruction sequence of the robotic arm; the motion instruction sequence is sent to the robotic arm motion controller to trigger the robotic arm to move according to the three-layer spiral detection path. During the movement, the robotic arm feeds back the current posture information in real time, and performs closed-loop control by comparing it with the preset path points; at the same time, when the robotic arm moves to each path point, the visible light camera and the infrared camera are synchronously triggered to collect dual-light images.

[0011] Step S503: Repeat the operation process of steps S200-S400 for the bi-optical image collected by the robot arm during the re-inspection process, use the trained cable recognition model to predict the visible light image, obtain the prediction mask of the cable area, and extract the cable boundary coordinate set through binarization processing and edge detection; then map the cable boundary to the infrared image, and extract the corresponding temperature data matrix; then calculate the maximum temperature difference and temperature gradient matrix of the cable surface, and compare them with the preset threshold to determine whether there is still an abnormal area; perform an abnormal warning based on the re-inspection result. If it is still determined that there is an abnormal area after the re-inspection, immediately start the sound and light alarm to send an alarm signal, and send abnormal information to the remote monitoring system at the same time. The abnormal information includes the coordinates, maximum temperature difference, and temperature gradient of the abnormal area; if the re-inspection result shows that the abnormal area disappears, record the abnormality as a misjudgment information, and store the relevant data in the historical database for record.

[0012] The dual-light fusion detection data management system at the end of the robotic arm includes a data acquisition module, a cable identification module, a data fusion module, an anomaly detection module, and a re-inspection module; The data acquisition module integrates a visible light camera and an infrared camera at the end of the robotic arm, sets discrete monitoring points according to the motion path of the robotic arm, and based on the posture data output by the motion controller, the robotic arm synchronously triggers dual-light image acquisition when the movement reaches the preset monitoring point, and performs spatiotemporal alignment on the collected data; The cable identification module performs semantic segmentation on the visible light image, filters the cable images from the visible light image data to construct a basic data set, and builds a cable identification model through the U-Net architecture to obtain the boundary coordinate set of the cable; The data fusion module performs feature matching between the cable boundary extracted from the visible light image and the infrared image, maps the cable area to the infrared image, and extracts the corresponding temperature data matrix; The anomaly detection module calculates the maximum temperature difference on the cable surface based on the mapped infrared temperature data matrix, and determines the abnormal area in combination with the preset temperature gradient threshold; The re-inspection module triggers the robotic arm to dynamically adjust the inspection path according to the abnormal detection results, focuses on the abnormal points for re-inspection, and issues abnormal warning prompts based on the re-inspection results.

[0013] Compared with the prior art, the present invention has the following beneficial effects: This invention integrates a visible light camera and an infrared camera at the end of the robotic arm, synchronously acquires dual-light images and aligns them in time and space, and combines this with a U-Net architecture to build a cable recognition model. This achieves high-precision cable area recognition and temperature data matching, enabling the robotic arm to efficiently perform cable inspections and provide analysis of temperature gradients and temperature differences, ensuring timely detection of cable anomalies. Through the spatial registration of dual-light images and the interpolation processing of infrared temperature data, the present invention can accurately obtain the temperature data of the cable area and determine the maximum temperature difference and temperature gradient thereof, thereby ensuring stability and accuracy in different environments and different detection conditions, and effectively improving the sensitivity and accuracy of cable anomaly detection; according to the abnormal detection results, the present invention triggers the mechanical arm to dynamically adjust the detection path, generates a three-layer spiral detection path and performs re-inspection, realizes the reconfirmation and re-inspection of the abnormal area, and issues abnormal warning prompts according to the re-inspection results, can automatically adapt to abnormal situations in the detection process, improve detection efficiency and accuracy, and realize timely warning of abnormal situations, ensuring the reliability and timeliness of the entire detection process. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] The accompanying drawings are used to provide a further understanding of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation of the present invention. In the accompanying drawings: Figure 1 It is a flow chart of the method for managing dual-light fusion detection data at the end of a robotic arm. DETAILED DESCRIPTION

[0015] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0016] See also Figure 1 The present invention provides a technical solution: a method for managing dual-light fusion detection data at the end of a robotic arm, comprising: Step S100: A visible light camera and an infrared camera are integrated at the end of the robotic arm. Discrete monitoring points are set according to the motion path of the robotic arm. Based on the posture data output by the motion controller, the robotic arm synchronously triggers dual-light image acquisition when the movement reaches the preset monitoring points, and the collected data is temporally and spatially aligned. Step S200: performing semantic segmentation on the visible light image, filtering cable images from the visible light image data to construct a basic data set, and building a cable recognition model through the U-Net architecture to obtain a cable boundary coordinate set; Step S300: performing feature matching on the cable boundary extracted from the visible light image and the infrared image, mapping the cable area to the infrared image, and extracting the corresponding temperature data matrix; Step S400: Calculate the maximum temperature difference on the cable surface based on the mapped infrared temperature data matrix, and determine the abnormal area in combination with a preset temperature gradient threshold; Step S500: According to the abnormality detection result, the robot arm is triggered to dynamically adjust the detection path, focus on the abnormal point for re-inspection, and issue an abnormality warning prompt based on the re-inspection result.

[0017] Step S100 includes: Step S101: A visible light camera, an infrared camera, and a dual-light acquisition trigger are installed at the end of the robotic arm. The posture data of the robotic arm motion controller is used as a synchronization signal. Discrete detection points are set according to the motion path of the robotic arm. Each detection point corresponds to a posture data range centered on the theoretical posture and including position tolerance and posture tolerance. The dual-light acquisition trigger continuously receives the posture data output by the robotic arm motion controller and matches the current posture data with the posture data range corresponding to the preset detection point. When the posture matching condition of a certain detection point is met, the trigger simultaneously sends a pulse signal to drive the visible light camera and the infrared camera to collect images at the same time. Step S102: A high checkerboard calibration plate is placed vertically at the center of the overlapping fields of view of the visible light camera and the infrared camera. The dual-light camera is driven by a robotic arm to perform multi-view motion, and n sets of dual-modal images at different orientations, pitches, and rotation angles are collected. The visible light image and the infrared image are subjected to single-target calibration using the relevant functions in the OpenCV library. For the visible light image sequence, the sub-pixel coordinates of the checkerboard corner points are extracted, and their internal parameter matrix and distortion coefficients are calculated. For the infrared image sequence, the internal parameter matrix is ​​also calculated. The stereoCalibrate function is used to combine the dual-modal image data to solve the external parameter matrix containing the rotation matrix and translation vector. Step S103: In the real-time detection stage, the homography matrix is ​​calculated based on the internal and external parameters obtained by calibration, the infrared image pixel coordinates are mapped to the visible light image coordinate system through the perspective transformation formula, and the infrared image is perspective transformed. The transformed pixel values ​​are filled in through the bilinear interpolation algorithm to achieve spatial registration of the dual-modal images; the registered visible light image is data-bound with the infrared temperature matrix at the corresponding moment, and a timestamp is added to each set of data and stored in the database.

[0018] Step S200 includes: Step S201: Filter images containing cables from the visible light image data as a basic dataset, annotate the cable areas in the dataset, create a corresponding pixel-level mask for each cable instance, and set the pixel values ​​in the mask to distinguish between cable areas and non-cable areas: the pixel value in the cable area is set to 1, and the pixel value in the non-cable area is set to 0; preprocess and data augment the annotated images, and divide them into training sets, validation sets, and test sets according to the ratio; Step S202: The cable recognition model is constructed using the U-Net architecture, which consists of three parts: an encoder, a bottleneck layer, and a decoder. The encoder part is composed of multiple DoubleConv modules alternating with maximum pooling layers. The DoubleConv module contains two convolutional layers, and each convolutional layer is connected to a normalization layer and a ReLU activation function. The encoder gradually extracts low-level to high-level features of the image through continuous alternating operations. The bottleneck layer is composed of a DoubleConv module to further extract high-level features of the image. The decoder part performs upsampling through the deconvolution layer to gradually restore the size of the feature map. The feature map of the corresponding layer of the encoder is spliced ​​with the feature map after deconvolution by using a jump connection method, and then passes through the DoubleConv module for feature extraction. Feature fusion; finally, the model converts the number of feature map channels to 1 through a 1x1 convolutional layer, and then uses the Sigmoid activation function to map the output value to between 0 and 1. The output value represents the probability that each pixel belongs to the cable area; the constructed model is trained using the training set, and the binary cross entropy loss function is selected to measure the difference between the model prediction result and the true mask. The Adam optimizer is used to adaptively adjust the learning rate of each parameter; in each training cycle, the model predicts the input image, calculates the loss between the prediction result and the true mask, calculates the gradient through the backpropagation algorithm, and then the optimizer updates the model parameters; during the training process, the validation set is used to regularly evaluate the model performance, monitor the accuracy, recall rate, and F1 value indicators, and the training is completed when the preset number of training cycles is reached; Step S203: Use the trained cable recognition model to predict the newly collected visible light image to obtain a predicted mask of the cable area in the image; binarize the predicted mask, set a threshold, set pixels with probability values ​​greater than the threshold to 1, and set pixels with probability values ​​less than the threshold to 0, to obtain the final cable area mask; process the binarized cable area mask through an edge detection algorithm to extract the boundary of the cable area, refine the extracted boundary, remove redundant points, and obtain the boundary coordinate set of the cable.

[0019] Step S300 uses the spatial mapping relationship between the registered visible light image and the infrared image to convert the acquired cable boundary coordinate set into the infrared image coordinate system using a homography matrix: for each coordinate point in the cable boundary coordinate set, its corresponding coordinate in the infrared image is calculated using the formula, thereby determining the outline of the cable area in the infrared image: ; where (x v ,y v ) represents the pixel coordinates of the cable boundary coordinates in the visible light image coordinate system, (x m ,y m ) means (x v ,y v ) corresponds to the pixel coordinates in the infrared image coordinate system, H represents the homography matrix; from the temperature data matrix corresponding to the infrared image, the complete temperature data matrix T of the cable area is extracted, and the matrix element T mn The temperature value of the pixel at the mth row and nth column in the corresponding cable area; the extracted temperature data matrix T is interpolated and its resolution is adjusted to the same as the cable area mask of the visible light image using the bicubic interpolation algorithm.

[0020] Step S400 includes: Step S401: Based on the temperature data matrix T processed in step S302, traverse all elements in the matrix and calculate the maximum temperature difference ΔTmax on the cable surface; at the same time, calculate the temperature gradient value of each pixel point and its eight neighboring pixels; for the pixel point with coordinates (i, j) in the matrix, the temperature gradient calculation formulas in the horizontal and vertical directions are: , ;in Indicates the temperature gradient value of pixel (i, j) in the horizontal direction, denotes the temperature gradient value of pixel (i, j) in the vertical direction. T(i + 1, j) represents the temperature value of the pixel at coordinate (i + 1, j) in the temperature matrix T. T(i - 1, j) represents the temperature value of the pixel at coordinate (i - 1, j) in the temperature matrix T. T(i, j + 1) represents the temperature value of the pixel at coordinate (i, j + 1) in the temperature matrix T. T(i, j - 1) represents the temperature value of the pixel at coordinate (i, j - 1) in the temperature matrix T; the comprehensive temperature gradient value G of pixel (i, j) ij , according to the formula: ; By traversing all pixels in the matrix, the temperature gradient matrix G of the entire cable area is obtained; Step S402: Compare the calculated maximum temperature difference value ΔTmax with the preset maximum temperature difference threshold T t1 and at the same time compare each element in the temperature gradient matrix G with the preset temperature gradient threshold T t2 ; When ΔTmax > T t1 or there is an element in G greater than T t2 , determine that the corresponding area is an abnormal area; Mark the coordinate positions of the abnormal area in the infrared image and the visible light image, and generate an abnormal area mask. The pixel values of the abnormal area in the mask are set to 1, and the rest are 0.

[0021] Step S500 includes: Step S501: Generate a three - layer spiral detection path centered on the coordinates of the abnormal point. After receiving the coordinate position of the abnormal area marked in step S402, record the coordinates of the abnormal point in space as (x0, y0, z0). Take this point as the center of the three - layer spiral detection path and also as the starting point of the spiral detection path; Set the initial radii of the first - layer, second - layer, and third - layer spiral paths as r1, r2, r3 respectively, and satisfy r1 < r2 < r3; For each layer of the spiral path, use the conversion method from polar coordinates to rectangular coordinates to generate path points: For the first - layer detection path, in the polar coordinate system, the generation method of the spiral line is according to the formula: ρ = r1 + k1×θ; where ρ represents the polar radius, k1 represents the pitch coefficient of the first - layer spiral line, controlling the increase of the polar radius when the spiral line rotates per unit angle; θ represents the polar angle, which is the angle formed by rotating counterclockwise from the polar axis to the line segment connecting the pole and a certain point on the spiral line; Set a series of angle values θ i , θ i ∈[0, 2aπ], a represents the number of turns of the first - layer spiral line; Convert the polar coordinates (ρ i , θ i ) to rectangular coordinates (x i , y i , z i ), where ρ iIndicates that it corresponds to θ i The polar diameter is calculated by the formula: i =x0+ρ i ×cosθ i ,y i =y0+ρ i ×sinθ i , z i = z0; Generate the coordinate points of the second and third spiral paths in the same way. By setting the corresponding angle value ranges respectively and following the above rectangular coordinate conversion formula, the rectangular coordinates of each point on the second and third spiral paths are obtained, thereby obtaining a complete three-layer spiral detection path; Step S502: Convert the generated three-layer spiral detection path into the motion instruction of the robot arm, and convert each coordinate point (x i ,y i ,z i ) is converted into the angle values ​​of each joint of the robotic arm, and the inverse kinematics algorithm is used, combined with the connecting rod length and joint type of the robotic arm, to solve the angle that each joint needs to rotate, and these joint angle values ​​are arranged in sequence according to the order of the path points to generate the motion instruction sequence of the robotic arm; the motion instruction sequence is sent to the robotic arm motion controller to trigger the robotic arm to move according to the three-layer spiral detection path. During the movement, the robotic arm feeds back the current posture information in real time, and performs closed-loop control by comparing it with the preset path points; at the same time, when the robotic arm moves to each path point, the visible light camera and the infrared camera are synchronously triggered to collect dual-light images.

[0022] Step S503: Repeat the operation process of steps S200-S400 for the bi-optical image collected by the robot arm during the re-inspection process, use the trained cable recognition model to predict the visible light image, obtain the prediction mask of the cable area, and extract the cable boundary coordinate set through binarization processing and edge detection; then map the cable boundary to the infrared image, and extract the corresponding temperature data matrix; then calculate the maximum temperature difference and temperature gradient matrix of the cable surface, and compare them with the preset threshold to determine whether there is still an abnormal area; perform an abnormal warning based on the re-inspection result. If it is still determined that there is an abnormal area after the re-inspection, immediately start the sound and light alarm to send an alarm signal, and send abnormal information to the remote monitoring system at the same time. The abnormal information includes the coordinates, maximum temperature difference, and temperature gradient of the abnormal area; if the re-inspection result shows that the abnormal area disappears, record the abnormality as a misjudgment information, and store the relevant data in the historical database for record.

[0023] Embodiments of the present invention: Taking substation cable inspection as an example, a visible light camera (resolution 1920×1080), an infrared camera (resolution 640×480), and a dual-light acquisition trigger are installed at the end of the robotic arm. Based on the layout and route of the substation cables, the robotic arm's motion path is planned, and discrete monitoring points are set along the path. Each monitoring point corresponds to a pose data range centered on the theoretical pose, with a position tolerance of ±5mm and an attitude tolerance of ±2°. The robotic arm moves along the preset path, and the dual-light acquisition trigger continuously receives pose data output by the robotic arm's motion controller. When the robotic arm reaches a monitoring point and the corresponding pose data falls within the pose data range of the corresponding monitoring point, the trigger simultaneously sends a pulse signal, driving the visible light camera and infrared camera to capture images at the same time. A 30mm×30mm tall checkerboard calibration plate was placed vertically at the center of the overlapping fields of view of the visible and infrared cameras. A robotic arm drove the dual-light camera to capture six sets of dual-modal images at different orientations, pitches, and rotations. Single-target calibration was performed on the visible and infrared images using OpenCV functions. For the visible image sequence, the sub-pixel coordinates of the checkerboard corners were extracted, and their internal parameter matrix and distortion coefficients were calculated. Similarly, the internal parameter matrix was calculated for the infrared image sequence. The stereoCalibrate function was used to combine the dual-modal image data to solve the external parameter matrix containing the rotation matrix and translation vector. During the real-time detection phase, the homography matrix H was calculated based on the internal and external parameters obtained from the calibration. The infrared image pixel coordinates were mapped to the visible image coordinate system using the perspective transformation formula. The infrared image was then perspective transformed, and the transformed pixel values ​​were filled in using the bilinear interpolation algorithm to achieve spatial registration of the dual-modal images. The registered visible image was then data-bound to the infrared temperature matrix at the corresponding time, and each data set was timestamped and stored in a database.

[0024] We selected 2,000 images containing cables from the collected visible light image data as the base dataset. We used the LabelMe tool to annotate the cable regions in the dataset, creating a corresponding pixel-level mask for each cable instance. We set the pixel values ​​in the mask to distinguish between cable areas (pixel values ​​set to 1) and non-cable areas (pixel values ​​set to 0). We then performed preprocessing and data augmentation operations on the annotated images, including random cropping, rotation, and brightness adjustment. The dataset was then divided into training, validation, and test sets in a 7:2:1 ratio. The cable recognition model is constructed using the U-Net architecture. The encoder part consists of four DoubleConv modules alternating with maximum pooling layers. The DoubleConv module contains two convolutional layers, and each convolutional layer is connected to a normalization layer and a ReLU activation function. The bottleneck layer consists of a DoubleConv module. The decoder part upsamples through the deconvolution layer and uses a jump connection method to splice the feature map of the corresponding layer of the encoder with the feature map after deconvolution, and then performs feature fusion through the DoubleConv module. Finally, the model converts the number of feature map channels to 1 through a 1x1 convolution layer, and then uses the Sigmoid activation function to map the output value to between 0 and 1. The constructed model is trained using the training set, and the binary cross entropy loss function is selected to measure the difference between the model prediction result and the true mask. The Adam optimizer is used, the initial learning rate is set to 0.001, and the learning rate of each parameter is adaptively adjusted. In each training cycle, the model predicts the input image, calculates the loss between the prediction result and the true mask, calculates the gradient through the back-propagation algorithm, and then the optimizer updates the model parameters. During the training process, the validation set is used to regularly evaluate the model performance, monitor the accuracy, recall rate, and F1 value indicators. When the preset number of training cycles is reached, the training is completed. The final model achieves an accuracy of 92%, a recall rate of 90%, and an F1 value of 91% on the test set. The trained cable recognition model is used to predict the newly collected visible light image to obtain the predicted mask of the cable area in the image. The predicted mask is binarized, and the threshold is set to 0.5. The pixels with probability values ​​greater than the threshold are set to 1, and the pixels with probability values ​​less than the threshold are set to 0 to obtain the final cable area mask. The binarized cable area mask is processed by the Canny edge detection algorithm to extract the boundary of the cable area. The extracted boundary is refined and redundant points are removed to obtain the boundary coordinate set of the cable. Through the spatial mapping relationship between the registered visible light image and the infrared image, the obtained cable boundary coordinate set is converted to the infrared image coordinate system through the homography matrix H. The complete temperature data matrix T of the cable area is extracted from the temperature data matrix corresponding to the infrared image. The extracted temperature data matrix T is interpolated and the bicubic interpolation algorithm is used to adjust its resolution to the same as the cable area mask of the visible light image, that is, 1920×1080. After receiving the coordinate position of the marked abnormal area, the coordinates of the abnormal point in space are marked as (x0, y0, z0). This point is used as the center of the three-layer spiral detection path and also as the starting point of the spiral detection path. The initial radii of the first, second and third layer spiral paths are set to r1=100mm, r2=200mm, and r3=300mm respectively. The number of turns of the first layer spiral is set to a=3, and the number of turns of the second and third layers of spirals are both set to 2. The detection path is generated and the generated three-layer spiral detection path is converted into the motion command of the robot arm. According to the kinematic model of the robot arm, each coordinate point (x i ,y i ,z i ) is converted into angle values ​​for each joint of the robotic arm. Using an inverse kinematics algorithm, combined with the robotic arm's link length and joint type, the required rotation angle of each joint is calculated. These joint angle values ​​are then arranged in the order of the path points to generate motion instructions for the robotic arm and send them to the robotic arm motion controller, triggering the robotic arm to move along the three-layer spiral detection path. Real-time feedback of the position information is used for closed-loop control, and dual-light image acquisition is simultaneously triggered. For the re-inspected dual-light images, cable identification, boundary extraction, and temperature data extraction are repeated. The maximum temperature difference and temperature gradient matrix on the cable surface are calculated, and compared with the preset threshold to determine abnormal areas. If the abnormal area still exists, the sound and light alarm is activated and the abnormal information (including the coordinates of the abnormal area, the maximum temperature difference, and the temperature gradient) is transmitted to the remote monitoring system. If the abnormal area disappears, it is recorded as a misjudgment and stored in the historical database.

[0025] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above and that the invention can be embodied in other specific forms without departing from the spirit or essential characteristics of the invention. Therefore, the embodiments should be considered in all respects as illustrative and non-restrictive, and the scope of the invention is defined by the appended claims, not the foregoing description, and all variations within the meaning and range of equivalents of the claims are intended to be included therein. Any reference sign in a claim should not be construed as limiting the claim to which it relates.

Claims

1. A method for managing dual-light fusion detection data at the end of a robotic arm, characterized by: The method comprises: Step S100: A visible light camera and an infrared camera are integrated at the end of the robotic arm. Discrete monitoring points are set according to the motion path of the robotic arm. Based on the posture data output by the motion controller, the robotic arm synchronously triggers dual-light image acquisition when the movement reaches the preset monitoring points, and the collected data is temporally and spatially aligned. Step S200: performing semantic segmentation on the visible light image, filtering cable images from the visible light image data to construct a basic data set, and building a cable recognition model through the U-Net architecture to obtain a cable boundary coordinate set; Step S300: performing feature matching on the cable boundary extracted from the visible light image and the infrared image, mapping the cable area to the infrared image, and extracting the corresponding temperature data matrix; Step S400: Calculate the maximum temperature difference on the cable surface based on the mapped infrared temperature data matrix, and determine the abnormal area in combination with a preset temperature gradient threshold; Step S500: According to the abnormality detection result, the robot arm is triggered to dynamically adjust the detection path, focus on the abnormal point for re-inspection, and issue an abnormality warning prompt based on the re-inspection result.

2. The method for managing dual-light fusion detection data at the end of a robotic arm according to claim 1, characterized in that: The step S100 includes: Step S101: A visible light camera, an infrared camera, and a dual-light acquisition trigger are installed at the end of the robotic arm. The posture data of the robotic arm motion controller is used as a synchronization signal. Discrete detection points are set according to the motion path of the robotic arm. Each detection point corresponds to a posture data range centered on the theoretical posture and including position tolerance and posture tolerance. The dual-light acquisition trigger continuously receives the posture data output by the robotic arm motion controller and matches the current posture data with the posture data range corresponding to the preset detection point. When the posture matching condition of a certain detection point is met, the trigger simultaneously sends a pulse signal to drive the visible light camera and the infrared camera to collect images at the same time. Step S102: A high checkerboard calibration plate is placed vertically at the center of the overlapping fields of view of the visible light camera and the infrared camera. The dual-light camera is driven by a robotic arm to perform multi-view motion, and n sets of dual-modal images at different orientations, pitches, and rotation angles are collected. The visible light image and the infrared image are subjected to single-target calibration using the relevant functions in the OpenCV library. For the visible light image sequence, the sub-pixel coordinates of the checkerboard corner points are extracted, and their internal parameter matrix and distortion coefficients are calculated. For the infrared image sequence, the internal parameter matrix is ​​also calculated. The stereoCalibrate function is used to combine the dual-modal image data to solve the external parameter matrix containing the rotation matrix and translation vector. Step S103: In the real-time detection stage, the homography matrix is ​​calculated based on the internal and external parameters obtained by calibration, the infrared image pixel coordinates are mapped to the visible light image coordinate system through the perspective transformation formula, and the infrared image is perspective transformed. The transformed pixel values ​​are filled in through the bilinear interpolation algorithm to achieve spatial registration of the dual-modal images; the registered visible light image is data-bound with the infrared temperature matrix at the corresponding moment, and a timestamp is added to each set of data and stored in the database.

3. The method for managing dual-light fusion detection data at the end of a robotic arm according to claim 1, characterized in that: The step S200 includes: Step S201: Filter images containing cables from the visible light image data as a basic dataset, annotate the cable areas in the dataset, create a corresponding pixel-level mask for each cable instance, and set the pixel values ​​in the mask to distinguish between cable areas and non-cable areas: the pixel value in the cable area is set to 1, and the pixel value in the non-cable area is set to 0; preprocess and data augment the annotated images, and divide them into training sets, validation sets, and test sets according to the ratio; Step S202: The cable recognition model is constructed using the U-Net architecture, which consists of three parts: an encoder, a bottleneck layer, and a decoder. The encoder part is composed of multiple DoubleConv modules alternating with maximum pooling layers. The DoubleConv module contains two convolutional layers, and each convolutional layer is connected to a normalization layer and a ReLU activation function. The encoder gradually extracts low-level to high-level features of the image through continuous alternating operations. The bottleneck layer is composed of a DoubleConv module to further extract high-level features of the image. The decoder part performs upsampling through the deconvolution layer to gradually restore the size of the feature map. The feature map of the corresponding layer of the encoder is spliced ​​with the feature map after deconvolution by using a jump connection method, and then passes through the DoubleConv module for feature extraction. Feature fusion; finally, the model converts the number of feature map channels to 1 through a 1x1 convolutional layer, and then uses the Sigmoid activation function to map the output value to between 0 and 1. The output value represents the probability that each pixel belongs to the cable area; the constructed model is trained using the training set, and the binary cross entropy loss function is selected to measure the difference between the model prediction result and the true mask. The Adam optimizer is used to adaptively adjust the learning rate of each parameter; in each training cycle, the model predicts the input image, calculates the loss between the prediction result and the true mask, calculates the gradient through the backpropagation algorithm, and then the optimizer updates the model parameters; during the training process, the validation set is used to regularly evaluate the model performance, monitor the accuracy, recall rate, and F1 value indicators, and the training is completed when the preset number of training cycles is reached; Step S203: Use the trained cable recognition model to predict the newly collected visible light image to obtain a predicted mask of the cable area in the image; binarize the predicted mask, set a threshold, set pixels with probability values ​​greater than the threshold to 1, and set pixels with probability values ​​less than the threshold to 0, to obtain the final cable area mask; process the binarized cable area mask through an edge detection algorithm to extract the boundary of the cable area, refine the extracted boundary, remove redundant points, and obtain the boundary coordinate set of the cable.

4. The method for managing dual-light fusion detection data at the end of a robotic arm according to claim 1, characterized in that: In step S300, the obtained cable boundary coordinate set is converted to the infrared image coordinate system through the homography matrix based on the spatial mapping relationship between the registered visible light image and the infrared image. For each coordinate point in the cable boundary coordinate set, its corresponding coordinate in the infrared image is calculated using the formula to determine the outline of the cable area in the infrared image: ; where (x v ,y v ) represents the pixel coordinates of the cable boundary coordinates in the visible light image coordinate system, (x m ,y m ) means (x v ,y v ) corresponds to the pixel coordinates in the infrared image coordinate system, H represents the homography matrix; from the temperature data matrix corresponding to the infrared image, the complete temperature data matrix T of the cable area is extracted, and the matrix element T mn The temperature value of the pixel at row m and column n in the corresponding cable area; The extracted temperature data matrix T is interpolated and its resolution is adjusted to the same as the cable area mask of the visible light image using the bicubic interpolation algorithm.

5. The method for managing dual-light fusion detection data at the end of a robotic arm according to claim 1, characterized in that: The step S400 includes: Step S401: Based on the temperature data matrix T processed in step S302, traverse all elements in the matrix and calculate the maximum temperature difference ΔTmax on the cable surface; at the same time, calculate the temperature gradient value of each pixel point and its eight neighboring pixels; for the pixel point with coordinates (i, j) in the matrix, the temperature gradient calculation formulas in the horizontal and vertical directions are: , ;in Indicates the temperature gradient value of pixel (i, j) in the horizontal direction, Represents the temperature gradient value of the pixel (i, j) in the vertical direction, T(i+1, j) represents the temperature value of the pixel at the coordinate (i+1, j) in the temperature matrix T, T(i-1, j) represents the temperature value of the pixel at the coordinate (i-1, j) in the temperature matrix T, T(i, j+1) represents the temperature value of the pixel at the coordinate (i, j+1) in the temperature matrix T, and T(i, j-1) represents the temperature value of the pixel at the coordinate (i, j-1) in the temperature matrix T; the comprehensive temperature gradient value G of the pixel (i, j) ij , according to the formula: ; By traversing all pixel points in the matrix, the temperature gradient matrix G of the entire cable area is obtained; Step S402: Compare the calculated maximum temperature difference ΔTmax with the preset maximum temperature difference threshold T t1 Compare and compare each element in the temperature gradient matrix G with the preset temperature gradient threshold T t2 For comparison; when ΔTmax>T t1 and there are elements in G greater than T t2 When any of the conditions is met, the corresponding area is determined to be an abnormal area; the coordinate position of the abnormal area in the infrared image and the visible light image is marked, and an abnormal area mask is generated, and the pixel value of the abnormal area in the mask is set to 1, and the rest are 0.

6. The method for managing dual-light fusion detection data at the end of a robotic arm according to claim 1, characterized in that: The step S500 includes: Step S501: Generate a three-layer spiral detection path centered on the abnormal point coordinates. After receiving the coordinates of the abnormal area marked in step S402, record the coordinates of the abnormal point in space as (x0, y0, z0). Take this point as the center of the three-layer spiral detection path and also as the starting point of the spiral detection path. Respectively set the initial radii of the first, second, and third layer spiral paths as r1, r2, r3, and satisfy r1 < r2 < r3. For each layer of the spiral path, use the conversion method from polar coordinates to rectangular coordinates to generate path points: For the first layer of the detection path, in the polar coordinate system, the generation method of the spiral line is according to the formula: ρ = r1 + k1×θ; where ρ represents the polar radius, k1 represents the pitch coefficient of the first layer of spiral line, which controls the increase of the polar radius when the spiral line rotates per unit angle; θ represents the polar angle, which is the angle formed by rotating counterclockwise from the polar axis to the line segment connecting the pole and a certain point on the spiral line. Set a series of angle values θ i , θ i ∈ [0, 2aπ], a represents the number of turns of the first layer of spiral line; convert the polar coordinates (ρ i , θ i ) to rectangular coordinates (x i , y i , z i ), where ρ i represents the polar radius corresponding to θ i , through the formula: x i = x0 + ρ i × cosθ i , y i = y0 + ρ i × sinθ i , z i = z0; Generate the coordinate points of the second and third layer spiral paths in the same way. By respectively setting the corresponding angle value ranges and according to the above rectangular coordinate conversion formula, obtain the rectangular coordinates of each point on the second and third layer spiral paths, so as to obtain a complete three-layer spiral detection path; Step S502: Convert the generated three-layer spiral detection path into the motion instruction of the robot arm, and convert each coordinate point (x i ,y i ,z i ) is converted into the angle values ​​of each joint of the robotic arm, and the inverse kinematics algorithm is used, combined with the connecting rod length and joint type of the robotic arm, to solve the angle that each joint needs to rotate, and these joint angle values ​​are arranged in sequence according to the order of the path points to generate the motion instruction sequence of the robotic arm; the motion instruction sequence is sent to the robotic arm motion controller to trigger the robotic arm to move according to the three-layer spiral detection path. During the movement, the robotic arm feeds back the current posture information in real time, and performs closed-loop control by comparing it with the preset path points; at the same time, when the robotic arm moves to each path point, the visible light camera and the infrared camera are synchronously triggered to collect dual-light images.

7. The method for managing dual-light fusion detection data at the end of a robotic arm according to claim 1, characterized in that: The step S500 further includes: Step S503: Repeat the operation process of steps S200-S400 for the bi-optical image collected by the robot arm during the re-inspection process, use the trained cable recognition model to predict the visible light image, obtain the prediction mask of the cable area, and extract the cable boundary coordinate set through binarization processing and edge detection; then map the cable boundary to the infrared image, and extract the corresponding temperature data matrix; then calculate the maximum temperature difference and temperature gradient matrix of the cable surface, and compare them with the preset threshold to determine whether there is still an abnormal area; perform an abnormal warning based on the re-inspection result. If it is still determined that there is an abnormal area after the re-inspection, immediately start the sound and light alarm to send an alarm signal, and send abnormal information to the remote monitoring system at the same time. The abnormal information includes the coordinates, maximum temperature difference, and temperature gradient of the abnormal area; if the re-inspection result shows that the abnormal area disappears, record the abnormality as a misjudgment information, and store the relevant data in the historical database for record.

8. The dual-light fusion detection data management system at the end of the robotic arm is characterized by: The system includes a data acquisition module, a cable identification module, a data fusion module, an anomaly detection module, and a re-inspection module; The data acquisition module integrates a visible light camera and an infrared camera at the end of the robotic arm, sets discrete monitoring points according to the motion path of the robotic arm, and synchronously triggers dual-light image acquisition when the robotic arm reaches the preset monitoring points based on the posture data output by the motion controller, and performs spatiotemporal alignment on the collected data. The cable identification module performs semantic segmentation on the visible light image, filters the cable images from the visible light image data to construct a basic data set, and builds a cable identification model through the U-Net architecture to obtain the boundary coordinate set of the cable; The data fusion module performs feature matching between the cable boundary extracted from the visible light image and the infrared image, maps the cable area to the infrared image, and extracts the corresponding temperature data matrix; The anomaly detection module calculates the maximum temperature difference on the cable surface based on the mapped infrared temperature data matrix, and determines the abnormal area in combination with the preset temperature gradient threshold; The re-inspection module triggers the robotic arm to dynamically adjust the inspection path according to the abnormal detection results, focuses on the abnormal points for re-inspection, and issues abnormal warning prompts based on the re-inspection results.

Images