Dual light fusion detection data management system and method for mechanical arm end

Abstract

The application discloses a mechanical arm end double-light fusion detection data management system and method, belongs to the technical field of industrial automation detection, integrates a visible light camera and an infrared camera at the end of a mechanical arm, sets discrete monitoring points according to the motion path of the mechanical arm, synchronously triggers double-light image acquisition when the mechanical arm reaches the preset monitoring points based on the pose data output by a motion controller, and performs space-time alignment on the collected data; performs semantic segmentation processing on the visible light image, filters out a cable image to construct a cable recognition model, obtains boundary coordinates of the cable, and performs feature matching on the infrared image, maps the cable area to the infrared image, extracts a corresponding temperature data matrix, calculates the maximum temperature difference value of the cable surface, determines an abnormal area in combination with a preset temperature gradient threshold value, triggers dynamic adjustment of a detection path of the mechanical arm according to an abnormal detection result, focuses on an abnormal point for re-inspection, and performs abnormal early warning and prompting according to a re-inspection result.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of industrial automation detection, in particular to a mechanical arm end dual-light fusion detection data management system and method. BACKGROUND

[0002] In the scenarios of power equipment, industrial facilities, etc., accurate monitoring of cable surface temperature is a key link for preventing fires and ensuring stable operation of equipment. Currently, the mainstream detection means relies on infrared thermal imaging technology and fixed temperature sensor networks. Specifically, an infrared camera obtains a device thermal distribution image through a non-contact method, which can quickly locate the abnormal temperature rise area; and a distributed sensor provides continuous temperature data collection. In recent years, some scenarios have introduced inspection mechanical arms to carry detection equipment to improve the detection coverage capability in complex environments. This method has solved the problems of low efficiency and high risk of manual inspection to a certain extent, and constitutes the technical basis for current cable temperature monitoring.

[0003] However, in actual application, although the infrared thermal imaging technology can capture temperature distribution, its image resolution is low, and it is difficult to accurately identify the fine structure of the cable surface, especially in the environment where cables are densely arranged or the light is complex, the temperature sampling area is easily shifted due to the blurred boundary; at the same time, although the mechanical arm expands the detection range, its movement path usually depends on the preset program and cannot be dynamically adjusted according to the real-time detection data. When a local temperature anomaly is detected, the mechanical arm still runs according to the original path, and it is difficult to focus on the problem area for re-inspection in time, resulting in an increased missed detection rate. SUMMARY

[0004] The purpose of the present application is to provide a mechanical arm end dual-light fusion detection data management system and method to solve the problems raised in the background art.

[0005] In order to solve the above technical problems, the present application provides the following technical scheme: a mechanical arm end dual-light fusion detection data management method, comprising:

[0006] Step S100: integrating a visible light camera and an infrared camera at the end of the mechanical arm, setting discrete monitoring points according to the movement path of the mechanical arm, and synchronously triggering dual-light image acquisition when the mechanical arm moves to the preset monitoring points based on the pose data output by the motion controller, and performing spatio-temporal alignment on the collected data;

[0007] Step S200: performing semantic segmentation processing on the visible light image, filtering the cable image from the visible light image data to construct a basic data set, and constructing a cable recognition model through a U-Net architecture to obtain a boundary coordinate set of the cable;

[0008] Step S300: Feature matching of the cable boundary extracted from the visible light image with the infrared image, mapping the cable area to the infrared image, and extracting the corresponding temperature data matrix;

[0009] Step S400: Based on the mapped infrared temperature data matrix, calculating the maximum temperature difference value of the cable surface, and combining the preset temperature gradient threshold to determine the abnormal area;

[0010] Step S500: According to the abnormal detection result, triggering the mechanical arm to dynamically adjust the detection path, focusing on the abnormal point for re-inspection, and according to the re-inspection result, triggering the abnormal pre-warning prompt.

[0011] Further, step S100 includes:

[0012] Step S101: A visible light camera, an infrared camera, and a dual-light acquisition trigger are installed at the end of the mechanical arm. The pose data of the mechanical arm motion controller is used as a synchronization signal. According to the motion path of the mechanical arm, a discrete detection point is set. Each detection point corresponds to a pose data range centered on the theoretical pose, including position tolerance and attitude tolerance. The dual-light acquisition trigger continuously receives the pose data output by the mechanical arm motion controller, matches the current pose data with the pose data range corresponding to the preset detection point, and when the pose matching condition of a certain detection point is met, the trigger sends a pulse signal at the same time to drive the visible light camera and the infrared camera to collect images at the same time.

[0013] Step S102: A high chessboard calibration board is vertically placed at the center of the overlapping field of view of the visible light camera and the infrared camera. The dual-light camera is moved by the mechanical arm to collect n groups of dual-mode images at different azimuth, pitch, and rotation angles. The relevant functions in the OpenCV library are used to perform single target calibration on the visible light image and the infrared image respectively. For the visible light image sequence, the chessboard corner sub-pixel coordinates are extracted, and the internal parameter matrix and distortion coefficient are calculated. For the infrared image sequence, the internal parameter matrix is also calculated. The stereoCalibrate function is used to jointly solve the external parameter matrix containing the rotation matrix and the translation vector based on the dual-mode image data.

[0014] Step S103: In the real-time detection stage, the homography matrix is calculated based on the internal and external parameters obtained by calibration, the infrared image pixel coordinates are mapped to the visible light image coordinate system through the perspective transformation formula, the infrared image is perspective transformed, the pixel values after transformation are filled through the bilinear interpolation algorithm, the spatial registration of the dual-mode images is realized, the registered visible light image and the corresponding infrared temperature matrix at the same time are data-bound, and a timestamp is added to each group of data for storage in the database.

[0015] Further, step S200 includes:

[0016] Step S201: Selecting images containing cables from the visible light image data as the base dataset, labeling the cable area in the dataset, creating a corresponding pixel-level mask for each cable instance, setting the pixel value in the mask to distinguish the cable area and the non-cable area: setting the cable area pixel value to 1 and the non-cable area pixel value to 0; performing preprocessing and data augmentation operations on the labeled images, and dividing the training set, validation set and test set according to the proportion;

[0017] Step S202: Building a cable recognition model using the U-Net architecture, which consists of an encoder, a bottleneck layer and a decoder; the encoder part is composed of multiple DoubleConv modules and max-pooling layers alternately; the DoubleConv module contains two convolution layers, and each convolution layer is connected with a normalization layer and a ReLU activation function; the encoder gradually extracts low-level to high-level features of the image through alternating operations; the bottleneck layer is composed of a DoubleConv module, which further extracts high-level features of the image; the decoder part restores the feature map size step by step through deconvolution; and the skip connection method is used to splice the feature map of the corresponding layer of the encoder and the feature map after deconvolution, and then the DoubleConv module is used for feature fusion; finally, the model converts the channel number of the feature map to 1 through a 1x1 convolution layer, and then maps the output value to between 0 and 1 through a Sigmoid activation function, wherein the output value represents the probability of each pixel belonging to the cable area; the training set is used to train the constructed model, a binary cross-entropy loss function is selected to measure the difference between the model prediction result and the true mask, an Adam optimizer is used to adaptively adjust the learning rate of each parameter; in each training period, the model predicts the input image, calculates the loss between the prediction result and the true mask, calculates the gradient through the back propagation algorithm, and then updates the model parameters by the optimizer; during the training process, the validation set is used to evaluate the model performance periodically, and the accuracy, recall rate and F1 value indicators are monitored; when the preset number of training periods is reached, the training is completed;

[0018] Step S203: Using the trained cable recognition model to predict the newly collected visible light image to obtain the prediction mask of the cable area in the image; performing binaryzation processing on the prediction mask, setting a threshold, setting the pixels with probability values greater than the threshold to 1 and the pixels with probability values less than the threshold to 0 to obtain the final cable area mask; performing processing on the binaryzation cable area mask through an edge detection algorithm, extracting the boundary of the cable area, and performing thinning processing on the extracted boundary to remove redundant points and obtain the boundary coordinate set of the cable.

[0019] Further, step S300 converts the obtained cable boundary coordinate set to the infrared image coordinate system through the spatial mapping relationship of the registered visible light image and infrared image: for each coordinate point in the cable boundary coordinate set, the corresponding coordinate of the coordinate point in the infrared image is calculated through the formula, and then the contour of the cable region in the infrared image is determined: ; wherein (x v ,y v ) represents the pixel coordinates of the coordinate set in the visible light image coordinate system, (x m ,y m ) represents the corresponding pixel coordinates of (x v ,y v ) in the infrared image coordinate system, and H represents the homography matrix; from the temperature data matrix corresponding to the infrared image, the complete temperature data matrix T of the cable region is extracted, and the matrix element T mn corresponds to the temperature value of the pixel point at the mth row and the nth column in the cable region; the extracted temperature data matrix T is subjected to interpolation processing, and a bicubic interpolation algorithm is used to adjust the resolution of the temperature data matrix T to be the same as the cable region mask of the visible light image.

[0020] Further, step S400 includes:

[0021] Step S401: based on the temperature data matrix T processed in step S300, all elements in the matrix are traversed, the maximum temperature difference value DTmax of the cable surface is calculated, and the temperature gradient value of each pixel point and its eight neighborhood pixel points is calculated; for the pixel point with coordinates (i, j) in the matrix, the temperature gradient calculation formula in the horizontal direction and the vertical direction is respectively: , ; wherein represents the temperature gradient value of the pixel point (i, j) in the horizontal direction, represents the temperature gradient value of the pixel point (i, j) in the vertical direction, T(i+1, j) represents the temperature value of the pixel point at coordinates (i+1, j) in the temperature matrix T, T(i-1, j) represents the temperature value of the pixel point at coordinates (i-1, j) in the temperature matrix T, T(i, j+1) represents the temperature value of the pixel point at coordinates (i, j+1) in the temperature matrix T, and T(i, j-1) represents the temperature value of the pixel point at coordinates (i, j-1) in the temperature matrix T; the comprehensive temperature gradient value G ij of the pixel point (i, j) is calculated according to the formula: ; by traversing all pixel points in the matrix, the temperature gradient matrix G of the entire cable region is obtained.

[0022] Step S402: the calculated maximum temperature difference value DTmax is compared with the preset maximum temperature difference threshold T t1Compare each element in the temperature gradient matrix G with a preset temperature gradient threshold T t2 Compare; when ΔTmax>T t1 and there is an element in G greater than T t2 , determine that the corresponding region is an abnormal region; mark the coordinate position of the abnormal region in the infrared image and the visible light image, and generate an abnormal region mask, and the pixel value of the abnormal region in the mask is set to 1 and the rest is 0.

[0023] Further, step S500 includes:

[0024] Step S501: generate a three-layer spiral detection path centered on the abnormal point coordinates; after receiving the abnormal region coordinate position marked in step S402, mark the coordinates of the abnormal point in space as (x0, y0, z0), and take this point as the center of the three-layer spiral detection path and also as the starting point of the spiral detection path; set the initial radii of the first layer, the second layer and the third layer spiral paths as r1, r2 and r3, respectively, and satisfy r1 i i ∈[0, 2aπ], a represents the number of turns of the first layer spiral line; convert the polar coordinates (ρ i , θ i ) into rectangular coordinates (x i , y i , z i ), where ρ i represents the polar radius corresponding to θ i , through the formula: x i =x0+ρ i ×cosθ i , y i =y0+ρ i ×sinθ i , z i =z0; generate the coordinate points of the second layer and the third layer spiral paths in the same way, set the corresponding angle value range respectively, and obtain the rectangular coordinates of each point on the second layer and the third layer spiral paths according to the above rectangular coordinate conversion formula, thereby obtaining the complete three-layer spiral detection path.

[0025] ​Step S502: Convert the generated three-layer spiral detection path into motion instructions of the robot arm, convert each coordinate point (x i ,y i ,z i ) on the path into angle values of each joint of the robot arm according to the kinematics model of the robot arm, use the inverse kinematics algorithm, combine the link length and joint type of the robot arm, solve the angle that each joint needs to rotate, arrange these joint angle values in order according to the sequence of the path points, and generate a motion instruction sequence of the robot arm; send the motion instruction sequence to the robot arm motion controller to trigger the robot arm to move according to the three-layer spiral detection path, and in the movement process, the robot arm feeds back the current pose information in real time, compares it with the preset path points, and performs closed-loop control; at the same time, when the robot arm moves to each path point, the visible light camera and the infrared camera are triggered to collect dual-light images synchronously.

[0026] Step S503: Repeat the operation procedures of steps S200-S400 for the dual-light images collected by the robot arm in the re-inspection process, use the trained cable recognition model to predict the visible light image to obtain the prediction mask of the cable area, and extract the cable boundary coordinate set through binaryzation processing and edge detection; then map the cable boundary to the infrared image to extract the corresponding temperature data matrix; then calculate the maximum temperature difference value and the temperature gradient matrix of the cable surface, compare them with the preset threshold value, and determine whether there is still an abnormal area; perform abnormal early warning according to the re-inspection result, if the re-inspection result still determines that there is an abnormal area, immediately start the sound and light alarm to issue an alarm signal, and send abnormal information to the remote monitoring system, the abnormal information including the coordinates of the abnormal area, the maximum temperature difference value, and the temperature gradient; if the re-inspection result shows that the abnormal area disappears, record this time of abnormality as a misjudgment information, and store the related data into a historical database for record.

[0027] The robot arm end dual-light fusion detection data management system comprises a data acquisition module, a cable recognition module, a data fusion module, an abnormality detection module, and a re-inspection module.

[0028] The data acquisition module integrates a visible light camera and an infrared camera at the end of the robot arm, sets discrete monitoring points according to the motion path of the robot arm, and synchronously triggers dual-light image collection when the robot arm moves to the preset monitoring points based on the pose data output by the motion controller, and performs spatio-temporal alignment on the collected data;

[0029] The cable recognition module performs semantic segmentation processing on the visible light image, filters the cable image from the visible light image data to construct a basic data set, and constructs a cable recognition model through a U-Net architecture to obtain the boundary coordinate set of the cable;

[0030] The data fusion module performs feature matching on the cable boundary extracted from the visible light image and the infrared image, maps the cable area to the infrared image, and extracts a corresponding temperature data matrix;

[0031] The abnormality detection module calculates a maximum temperature difference value of the cable surface based on the mapped infrared temperature data matrix, and determines an abnormal area in combination with a preset temperature gradient threshold value;

[0032] The re-inspection module triggers dynamic adjustment of a detection path of the mechanical arm according to the abnormality detection result, focuses on an abnormal point for re-inspection, and provides an abnormality warning prompt according to a re-inspection result.

[0033] Compared with the prior art, the present application has the following beneficial effects:

[0034] The present application integrates a visible light camera and an infrared camera at the end of the mechanical arm, synchronously collects and spatiotemporally aligns double light images, constructs a cable recognition model in combination with a U-Net architecture, realizes high-precision cable area recognition and temperature data matching, enables the mechanical arm to efficiently perform cable detection, and provides temperature gradient and temperature difference analysis, thereby ensuring timely discovery of abnormal problems of the cable.

[0035] The present application can accurately obtain temperature data of the cable area through spatial registration of double light images and interpolation processing of infrared temperature data, and determine a maximum temperature difference value and a temperature gradient, thereby ensuring stability and accuracy under different environments and different detection conditions, effectively improving sensitivity and accuracy of cable abnormality detection. BRIEF DESCRIPTION OF DRAWINGS

[0036] The accompanying drawings are included to provide a further understanding of the present application, and constitute a part of the specification, illustrate the present application together with the embodiments, and are used to explain the present application, and do not constitute a limitation on the present application.

[0037] Figure 1 is a method flowchart of a double light fusion detection data management method. DETAILED DESCRIPTION

[0038] With reference to the drawings of the embodiments of the present application, the technical solutions in the embodiments of the present application will be described clearly and completely. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments of the present application, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the scope of the present application.

[0039] Please refer to Figure 1 The present application provides a technical solution: a mechanical arm end double-light fusion detection data management method, comprising:

[0040] Step S100: integrating a visible light camera and an infrared camera at the end of a mechanical arm, setting discrete monitoring points according to the motion path of the mechanical arm, and triggering the collection of double-light images simultaneously when the mechanical arm reaches the preset monitoring points based on the pose data output by the motion controller, and performing spatio-temporal alignment on the collected data;

[0041] Step S200: performing semantic segmentation processing on the visible light image, filtering the cable image from the visible light image data to construct a basic data set, and constructing a cable recognition model through a U-Net architecture to obtain a boundary coordinate set of the cable;

[0042] Step S300: performing feature matching between the cable boundary extracted from the visible light image and the infrared image, mapping the cable region to the infrared image, and extracting the corresponding temperature data matrix;

[0043] Step S400: based on the mapped infrared temperature data matrix, calculating the maximum temperature difference value of the cable surface, and combining the preset temperature gradient threshold to determine the abnormal area;

[0044] Step S500: according to the abnormal detection result, triggering the mechanical arm to dynamically adjust the detection path, focusing on the abnormal point for re-inspection, and according to the re-inspection result, performing abnormal early warning prompt.

[0045] Step S100 includes:

[0046] Step S101: installing a visible light camera, an infrared camera, and a double-light collection trigger at the end of the mechanical arm, using the pose data of the mechanical arm motion controller as a synchronization signal, setting discrete detection points according to the motion path of the mechanical arm, each detection point corresponding to a pose data range centered on the theoretical pose, including position tolerance and attitude tolerance; the double-light collection trigger continuously receives the pose data output by the mechanical arm motion controller, matches the current pose data with the pose data range corresponding to the preset detection points, and when the pose matching condition of a certain detection point is met, the trigger sends a pulse signal at the same time, driving the visible light camera and the infrared camera to collect images at the same time;

[0047] Step S102: vertically place a high chessboard calibration board at the center position of the overlapping field of view of the visible light camera and the infrared camera, drive the dual optical camera to move in multiple views by the mechanical arm, and respectively collect n groups of dual-mode images under different azimuth, pitch and rotation angles; use the related functions in the OpenCV library to respectively calibrate single targets for the visible light image and the infrared image; for the visible light image sequence, extract the chessboard corner sub-pixel coordinates, calculate the internal parameter matrix and the distortion coefficient; for the infrared image sequence, also calculate the internal parameter matrix; use the stereoCalibrate function to jointly solve the external parameter matrix containing the rotation matrix and the translation vector by using the dual-mode image data;

[0048] Step S103: in the real-time detection stage, calculate the homography matrix according to the internal and external parameters obtained in the calibration, map the infrared image pixel coordinates to the visible light image coordinate system through the perspective transformation formula, and perform perspective transformation on the infrared image, fill the pixel values after transformation through the bilinear interpolation algorithm, realize the spatial registration of the dual-mode images; bind the registered visible light image and the corresponding infrared temperature matrix at the time, and add a time stamp to each group of data and store it in the database.

[0049] Step S200 includes:

[0050] Step S201: select the image containing the cable from the visible light image data as the basic data set, label the cable area in the data set, create a corresponding pixel-level mask for each cable instance, and set the pixel value in the mask to distinguish the cable area and the non-cable area: the pixel value of the cable area is set to 1, and the pixel value of the non-cable area is set to 0; perform preprocessing and data enhancement operations on the labeled image, and divide the training set, the validation set and the test set according to the proportion;

[0051] Step S202: a cable recognition model is constructed using a U-Net architecture, which is composed of an encoder, a bottleneck layer and a decoder; the encoder part is composed of multiple DoubleConv modules and max-pooling layers alternately; the DoubleConv module contains two convolution layers, and each convolution layer is connected with a normalization layer and a ReLU activation function; the encoder gradually extracts low-level to high-level features of the image through alternating operations; the bottleneck layer is composed of a DoubleConv module, which further extracts high-level features of the image; the decoder part is up-sampled through deconvolution layers, and gradually restores the feature map size; and a skip connection method is used to splice the feature map of the corresponding layer of the encoder and the feature map after deconvolution, and then the DoubleConv module is used for feature fusion; finally, the model converts the channel number of the feature map to 1 through a 1x1 convolution layer, and then maps the output value to between 0 and 1 through a Sigmoid activation function, wherein the output value represents the probability of each pixel belonging to the cable area; the training set is used to train the constructed model, a binary cross-entropy loss function is selected to measure the difference between the model prediction result and the real mask, an Adam optimizer is used to adaptively adjust the learning rate of each parameter; in each training period, the model predicts the input image, calculates the loss between the prediction result and the real mask, calculates the gradient through the back propagation algorithm, and then updates the model parameters through the optimizer; during the training process, the validation set is used to periodically evaluate the model performance, and the accuracy, recall rate and F1 value indicators are monitored; when the preset number of training periods is reached, the training is completed.

[0052] Step S203: using the trained cable recognition model to predict the newly collected visible light image to obtain the prediction mask of the cable area in the image; the prediction mask is binarized, a threshold is set, the pixels with a probability value greater than the threshold are set to 1, and the pixels with a probability value less than the threshold are set to 0, to obtain the final cable area mask; the binarized cable area mask is processed through an edge detection algorithm to extract the boundary of the cable area, and the extracted boundary is refined to remove redundant points to obtain the boundary coordinate set of the cable.

[0053] Step S300: through the spatial mapping relationship between the registered visible light image and infrared image, the obtained cable boundary coordinate set is converted to the infrared image coordinate system through a homography matrix: for each coordinate point in the cable boundary coordinate set, the corresponding coordinate in the infrared image is calculated through the formula, and then the outline of the cable area in the infrared image is determined: ; wherein (x v ,y v ) represents the pixel coordinates in the visible light image coordinate system in the cable boundary coordinate set, and (x m ,y m ) represents the pixel coordinates in the infrared image coordinate system. v v ​) corresponding pixel coordinates in the infrared image coordinate system, H represents a homography matrix; from the temperature data matrix corresponding to the infrared image, a complete temperature data matrix T of the cable region is extracted, and the matrix element T mn corresponds to the temperature value of the pixel point in the mth row and the nth column in the cable region; the extracted temperature data matrix T is subjected to interpolation processing, and a bicubic interpolation algorithm is used to adjust the resolution thereof to be the same as that of the cable region mask of the visible light image.

[0054] Step S400 comprises:

[0055] Step S401: based on the temperature data matrix T processed in step S300, all elements in the matrix are traversed, the maximum temperature difference value DTmax of the cable surface is calculated, and the temperature gradient value of each pixel point and its eight neighborhood pixel points is calculated; for the pixel point with coordinates (i, j) in the matrix, the temperature gradient calculation formula in the horizontal direction and the vertical direction is respectively: , ; wherein represents the temperature gradient value of the pixel point (i, j) in the horizontal direction, represents the temperature gradient value of the pixel point (i, j) in the vertical direction, T(i+1, j) represents the temperature value of the pixel point at coordinates (i+1, j) in the temperature matrix T, T(i-1, j) represents the temperature value of the pixel point at coordinates (i-1, j) in the temperature matrix T, T(i, j+1) represents the temperature value of the pixel point at coordinates (i, j+1) in the temperature matrix T, and T(i, j-1) represents the temperature value of the pixel point at coordinates (i, j-1) in the temperature matrix T; the comprehensive temperature gradient value G ij of the pixel point (i, j) is calculated according to the formula: ; by traversing all pixel points in the matrix, the temperature gradient matrix G of the entire cable region is obtained.

[0056] Step S402: the maximum temperature difference value DTmax calculated is compared with the preset maximum temperature difference threshold T t1 , and each element in the temperature gradient matrix G is compared with the preset temperature gradient threshold T t2 ; when DTmax>T t1 or there is an element greater than T t2 in G, it is determined that the corresponding region is an abnormal region; the coordinate position of the abnormal region in the infrared image and the visible light image is marked, and an abnormal region mask is generated, and the pixel value of the abnormal region in the mask is set to 1, and the rest is 0.

[0057] Step S500 comprises:

[0058] Step S501: generating a three-layer spiral detection path centered on the abnormal point coordinates, after receiving the abnormal region coordinate position marked in step S402, marking the coordinates of the abnormal point in space as (x0, y0, z0), taking this point as the center of the three-layer spiral detection path, and also as the starting point of the spiral detection path; setting the initial radii of the first layer, second layer and third layer spiral paths as r1, r2, r3 respectively, and satisfying r1 < r2 < r3; for each layer of spiral path, generating path points by adopting polar coordinate to rectangular coordinate conversion: for the first layer detection path, in the polar coordinate system, the generation mode of the spiral line is according to the formula: p = r1 + k1 x q; wherein p represents the polar radius, k1 represents the pitch coefficient of the first layer spiral line, controlling the increase amount of the polar radius when the spiral line rotates by a unit angle; q represents the polar angle, which is the angle formed by counterclockwise rotating from the polar axis to the line segment connecting the polar point and a certain point on the spiral line; set a series of angle values q i , q i e [0, 2a q], a represents the number of turns of the first layer spiral line; convert the polar coordinates (p i , q i ) into rectangular coordinates (x i , y i , z i ), wherein p i represents the polar radius corresponding to q i , through the formula: x i = x0 + p i x cos q i , y i = y0 + p i x sin q i , z i = z0; generate the coordinate points of the second layer and third layer spiral paths in the same way, by setting the corresponding angle value ranges respectively, according to the above rectangular coordinate conversion formula, to obtain the rectangular coordinates of each point on the second layer and third layer spiral paths, and thus obtain the complete three-layer spiral detection path;

[0059] Step S502: converting the generated three-layer spiral detection path into the motion instructions of the mechanical arm, according to the kinematic model of the mechanical arm, to obtain the rectangular coordinates of each coordinate point (x i , y i , z i) the angle values of each joint of the mechanical arm are converted, an inverse kinematics algorithm is used, the link length and joint type of the mechanical arm are combined, the angle that each joint needs to rotate is solved, the joint angle values are arranged in sequence according to the order of the path points, and the motion instruction sequence of the mechanical arm is generated; the motion instruction sequence is sent to the mechanical arm motion controller, the mechanical arm is triggered to move according to the three-layer spiral detection path, in the movement process, the mechanical arm feeds back the current pose information in real time, and the closed-loop control is performed by comparing with the preset path points; at the same time, when the mechanical arm moves to each path point, the visible light camera and the infrared camera are synchronously triggered to collect the dual-light images.

[0060] Step S503: for the dual-light images collected by the mechanical arm in the re-inspection process, the operation processes of steps S200-S400 are repeated, the trained cable recognition model is used to predict the visible light image, the prediction mask of the cable area is obtained, and the cable boundary coordinate set is extracted through binaryzation processing and edge detection; then the cable boundary is mapped to the infrared image, and the corresponding temperature data matrix is extracted; then the maximum temperature difference value and the temperature gradient matrix of the cable surface are calculated, and whether there is still an abnormal area is determined by comparing with the preset threshold; abnormal early warning is performed according to the re-inspection result, if it is still determined that there is an abnormal area after re-inspection, the sound and light alarm is immediately started to send an alarm signal, and abnormal information is sent to the remote monitoring system, the abnormal information includes the coordinates of the abnormal area, the maximum temperature difference value and the temperature gradient; if the re-inspection result shows that the abnormal area disappears, the abnormality this time is recorded as a misjudgment information, and the related data is stored into a historical database for record.

[0061] Embodiments of the application:

[0062] Taking the cable detection work of a transformer substation as an example, a visible light camera (resolution is 1920*1080), an infrared camera (resolution is 640*480) and a dual-light collection trigger are installed at the end of the mechanical arm; according to the layout and direction of the cables of the transformer substation, the motion path of the mechanical arm is planned, discrete monitoring points are set on the path, each monitoring point corresponds to a pose data range with a theoretical pose as the center, a position tolerance of ±5mm and an attitude tolerance of ±2°; the mechanical arm moves according to the preset path, the dual-light collection trigger continuously receives the pose data output by the mechanical arm motion controller, when the mechanical arm moves to a certain monitoring point, the corresponding pose data falls into the pose data range of the corresponding monitoring point, the trigger sends a pulse signal at the same time, and the visible light camera and the infrared camera are driven to collect images at the same time;

[0063] A high chessboard calibration board with a size of 30mm*30mm is vertically placed at the center of the overlapping field of view of the visible light camera and the infrared camera. The dual-mode image acquisition is carried out by the mechanical arm driving the dual light camera at 6 different azimuth, pitch and rotation angles. The relevant functions in the OpenCV library are used to perform single target calibration on the visible light image and the infrared image respectively. For the visible light image sequence, the chessboard corner sub-pixel coordinates are extracted, and the internal parameter matrix and the distortion coefficient are calculated. For the infrared image sequence, the internal parameter matrix is also calculated. The stereoCalibrate function is used to jointly solve the external parameter matrix containing the rotation matrix and the translation vector by using the dual-mode image data. In the real-time detection stage, the homography matrix H is calculated according to the internal and external parameters obtained by calibration, the infrared image pixel coordinates are mapped to the visible light image coordinate system through the perspective transformation formula, and the infrared image is subjected to perspective transformation. The pixel values of the transformed image are filled by using the bilinear interpolation algorithm, so as to realize the spatial registration of the dual-mode images. The registered visible light image and the infrared temperature matrix at the corresponding time are data-bound, and a time stamp is added to each group of data and stored in the database.

[0064] 2000 images containing cables are selected from the collected visible light image data as a basic data set. The cable area in the data set is labeled using the LabelMe tool, and a corresponding pixel-level mask is created for each cable instance. The pixel values in the mask are set to distinguish the cable area (pixel value set to 1) and the non-cable area (pixel value set to 0). The labeled images are preprocessed and data enhanced by random cropping, rotation, brightness adjustment, etc. The training set, validation set and test set are divided according to the ratio of 7:2:1.

[0065] The cable recognition model is constructed by using a U-Net architecture. The encoder part is alternately composed of four DoubleConv modules and a maximum pooling layer. The DoubleConv module contains two convolution layers, and each convolution layer is connected with a normalization layer and a ReLU activation function. The bottleneck layer is composed of a DoubleConv module. The decoder part is up-sampled by a deconvolution layer, and uses a skip connection method to splice the feature map of the corresponding layer of the encoder and the feature map after deconvolution, and then performs feature fusion through a DoubleConv module. Finally, the model converts the channel number of the feature map to 1 through a 1x1 convolution layer, and then maps the output value to between 0 and 1 through a Sigmoid activation function. The training set is used to train the constructed model, the binary cross-entropy loss function is selected to measure the difference between the model prediction result and the real mask, the Adam optimizer is used, the initial learning rate is set to 0.001, and the learning rate of each parameter is adaptively adjusted. In each training period, the model predicts the input image, calculates the loss of the prediction result and the real mask, calculates the gradient through the back propagation algorithm, and then updates the model parameters by the optimizer. During the training process, the validation set is used to evaluate the model performance regularly, and the accuracy, recall rate and F1 value indicators are monitored. When the preset number of 50 training periods is reached, the training is completed, and finally the model reaches an accuracy of 92%, a recall rate of 90% and an F1 value of 91% on the test set. The trained cable recognition model is used to predict the newly collected visible light image, and the prediction mask of the cable region in the image is obtained. The prediction mask is binarized by setting the threshold to 0.5, the pixels with a probability value greater than the threshold are set to 1, and the pixels with a probability value less than the threshold are set to 0, to obtain the final cable region mask. The binarized cable region mask is processed by the Canny edge detection algorithm, the boundary of the cable region is extracted, the extracted boundary is thinned to remove redundant points, and the boundary coordinate set of the cable is obtained.

[0066] The acquired cable boundary coordinate set is converted to the infrared image coordinate system through the homography matrix H by the spatial mapping relationship of the registered visible light image and infrared image, complete temperature data matrix T of the cable area is extracted from the temperature data matrix corresponding to the infrared image, interpolation processing is performed on the extracted temperature data matrix T, and the resolution is adjusted to the same as the visible light image cable area mask, that is, 1920*1080, by using a bicubic interpolation algorithm; after receiving the marked abnormal area coordinate position, the coordinates of the abnormal point in space are marked as (x0, y0, z0), the point is used as the center of the three-layer spiral detection path and also as the starting point of the spiral detection path; the initial radii of the first layer, the second layer and the third layer spiral paths are respectively set as r1=100mm, r2=200mm and r3=300mm; the number of turns of the first layer spiral line is set as a=3, and the number of turns of the second layer and the third layer spiral lines are both set as 2, a detection path is generated, the generated three-layer spiral detection path is converted into a motion instruction of the mechanical arm, each coordinate point (x i ,y i ,z i ) on the path is converted into an angle value of each joint of the mechanical arm according to the kinematic model of the mechanical arm, the angle value of each joint is solved by using an inverse kinematics algorithm combined with the length of the connecting rod and the type of the joint, the joint angle values are arranged in order according to the sequence of the path points, a motion instruction of the mechanical arm is generated and sent to a motion controller of the mechanical arm, the mechanical arm is triggered to move according to the three-layer spiral detection path, real-time feedback of the pose information is performed for closed-loop control, and dual-light image acquisition is triggered synchronously; for the dual-light images collected by rechecking, operations such as cable identification, boundary extraction and temperature data extraction are repeated, the maximum temperature difference value and the temperature gradient matrix of the cable surface are calculated, and the abnormal area is determined by comparison with a preset threshold. If the abnormal area still exists, an audible and visual alarm is started, and abnormal information (including abnormal area coordinates, maximum temperature difference value and temperature gradient) is sent to a remote monitoring system; if the abnormal area disappears, false information is recorded and stored in a historical database.

[0067] It is apparent to those skilled in the art that the present application is not limited to the details of the foregoing exemplary embodiments, and that the present application can be implemented in other particular forms without departing from the spirit or essential characteristics of the present application. Therefore, the embodiments should be considered in all respects as illustrative and not restrictive, and the scope of the present application should be defined by the appended claims rather than the foregoing description, and it is intended to encompass all changes falling within the meaning and scope of equivalents of the claims. Any reference signs in the claims should not be construed as limiting the claims to which the reference signs are directed.

Claims

1. A method for managing dual-light fusion detection data at the end effector of a robotic arm, characterized in that: The method includes: Step S100: Integrate a visible light camera and an infrared camera at the end effector of the robotic arm. Set discrete monitoring points according to the movement path of the robotic arm. Based on the pose data output by the motion controller, the robotic arm synchronously triggers dual-light image acquisition when the movement reaches the preset monitoring point, and performs spatiotemporal alignment on the acquired data. Step S100 includes: Step S101: Install a visible light camera, an infrared camera, and a dual-light acquisition trigger at the end of the robotic arm. Use the pose data from the robotic arm motion controller as a synchronization signal. Based on the movement path of the robotic arm, set discrete detection points. Each detection point corresponds to a pose data range centered on the theoretical pose and including position and attitude tolerances. The dual-light acquisition trigger continuously receives the pose data output by the robotic arm motion controller and matches the current pose data with the pose data range corresponding to the preset detection point. When the pose matching condition of a certain detection point is met, the trigger simultaneously sends a pulse signal to drive the visible light camera and the infrared camera to acquire images at the same time. Step S102: At the center of the overlapping field of view of the visible light camera and the infrared camera, a high checkerboard calibration plate is placed vertically. The dual-light camera is driven by a robotic arm to perform multi-view motion, acquiring n sets of dual-modal images under different azimuth, pitch, and rotation angles. The relevant functions in the OpenCV library are used to perform single-target calibration on the visible light image and the infrared image respectively: For the visible light image sequence, the sub-pixel coordinates of the checkerboard corner points are extracted, and its internal parameter matrix and distortion coefficients are calculated; for the infrared image sequence, its internal parameter matrix is ​​calculated in the same way; The stereoCalibrate function is used to combine the dual-modal image data to solve for the external parameter matrix containing the rotation matrix and translation vector. Step S103: In the real-time detection stage, the homography matrix is ​​calculated based on the endo-endo-parameters obtained from calibration. The pixel coordinates of the infrared image are mapped to the visible light image coordinate system through the perspective transformation formula. The infrared image is then subjected to perspective transformation, and the transformed pixel values ​​are filled using a bilinear interpolation algorithm to achieve spatial registration of the dual-modal images. The registered visible light image is then bound to the infrared temperature matrix at the corresponding time, and a timestamp is added to each set of data and stored in the database. Step S200: Perform semantic segmentation processing on the visible light images, filter cable images from the visible light image data to construct a basic dataset, and construct a cable recognition model through the U-Net architecture to obtain the boundary coordinate set of the cable; Step S300: Perform feature matching between the cable boundary extracted from the visible light image and the infrared image, map the cable area to the infrared image, and extract the corresponding temperature data matrix; Step S400: Based on the mapped infrared temperature data matrix, calculate the maximum temperature difference on the cable surface, and determine the abnormal area by combining it with the preset temperature gradient threshold. Step S500: Based on the abnormality detection results, trigger the robotic arm to dynamically adjust the detection path, focus on the abnormal point for re-inspection, and issue an abnormality warning based on the re-inspection results.

2. The method for managing dual-light fusion detection data at the end of a robotic arm according to claim 1, characterized in that: Step S200 includes: Step S201: Select images containing cables from the visible light image data as the basic dataset, label the cable regions in the dataset, create a corresponding pixel-level mask for each cable instance, and set the pixel values ​​in the mask to distinguish between cable regions and non-cable regions: set the pixel value of cable regions to 1 and the pixel value of non-cable regions to 0; perform preprocessing and data augmentation operations on the labeled images, and divide them into training set, validation set and test set according to the proportion; Step S202: A cable recognition model is constructed using the U-Net architecture, consisting of an encoder, a bottleneck layer, and a decoder. The encoder is composed of multiple DoubleConv modules alternating with max-pooling layers. Each DoubleConv module contains two convolutional layers, followed by a normalization layer and a ReLU activation function. The encoder extracts low-level to high-level features of the image through continuous alternating operations. The bottleneck layer consists of a DoubleConv module, which further extracts high-level features of the image. The decoder performs upsampling through deconvolutional layers to gradually restore the feature map size. A skip connection method is used to concatenate the feature map of the corresponding layer of the encoder with the deconvolutioned feature map, and then performs further processing through the DoubleConv module. Feature fusion is performed; finally, the model converts the number of feature map channels to 1 through a 1x1 convolutional layer, and then the Sigmoid activation function maps the output value to between 0 and 1, where the output value represents the probability that each pixel belongs to the cable region; the model is trained using the training set, and the binary cross-entropy loss function is selected to measure the difference between the model's prediction and the real mask. The Adam optimizer is used to adaptively adjust the learning rate of each parameter; in each training cycle, the model predicts the input image, calculates the loss between the prediction and the real mask, calculates the gradient through the backpropagation algorithm, and then the optimizer updates the model parameters; during training, the model performance is evaluated periodically using the validation set, and the accuracy, recall, and F1 score are monitored. Training is completed when the preset number of training cycles is reached; Step S203: Use the trained cable recognition model to predict the newly acquired visible light image to obtain the prediction mask of the cable region in the image; perform binarization on the prediction mask, set a threshold, set pixels with probability values ​​greater than the threshold to 1, and pixels with probability values ​​less than the threshold to 0, to obtain the final cable region mask; process the binarized cable region mask through an edge detection algorithm to extract the boundary of the cable region, refine the extracted boundary, remove redundant points, and obtain the boundary coordinate set of the cable.

3. The method for managing dual-light fusion detection data at the end of a robotic arm according to claim 2, characterized in that: Step S300 utilizes the spatial mapping relationship between the registered visible light image and the infrared image to transform the acquired cable boundary coordinate set to the infrared image coordinate system using a homography matrix: for each coordinate point in the cable boundary coordinate set, its corresponding coordinates in the infrared image are calculated using a formula, thereby determining the outline of the cable region in the infrared image. ;where (x v ,y v (x) represents the pixel coordinates of the cable boundary concentrated in the visible light image coordinate system. m ,y m ) represents (x v ,y v The pixel coordinates in the infrared image coordinate system, where H represents the homography matrix; from the temperature data matrix corresponding to the infrared image, extract the complete temperature data matrix T of the cable area, and the matrix elements T mn The temperature value of the pixel in the m-th row and n-th column within the corresponding cable area; The extracted temperature data matrix T is interpolated, and the resolution is adjusted to be the same as that of the cable region mask in the visible light image using a bicubic interpolation algorithm.

4. The method for managing dual-light fusion detection data at the end of a robotic arm according to claim 3, characterized in that: Step S400 includes: Step S401: Based on the temperature data matrix T processed in step S300, traverse all elements in the matrix and calculate the maximum temperature difference ΔTmax on the cable surface; simultaneously calculate the temperature gradient value between each pixel and its 8 neighboring pixels; for the pixel with coordinates (i,j) in the matrix, the formulas for calculating the temperature gradient in the horizontal and vertical directions are as follows: , ;in This represents the temperature gradient value of pixel (i,j) in the horizontal direction. Let T(i+1,j) represent the temperature gradient value of pixel (i,j) in the vertical direction, T(i+1,j) represent the temperature value of the pixel at coordinate (i+1,j) in temperature matrix T, T(i-1,j) represent the temperature value of the pixel at coordinate (i-1,j) in temperature matrix T, T(i,j+1) represent the temperature value of the pixel at coordinate (i,j+1) in temperature matrix T, and T(i,j-1) represent the temperature value of the pixel at coordinate (i,j-1) in temperature matrix T; and G represents the comprehensive temperature gradient value of pixel (i,j). ij According to the formula: By traversing all pixels in the matrix, the temperature gradient matrix G of the entire cable area is obtained. Step S402: Compare the calculated maximum temperature difference value ΔTmax with the preset maximum temperature difference threshold T. t1 The comparison is performed, and each element in the temperature gradient matrix G is compared with a preset temperature gradient threshold T. t2 Compare the results; when ΔTmax > T t1 And there exists an element in G that is greater than T. t2 If any one of the conditions is met, the corresponding region is determined to be an abnormal region; the coordinates of the abnormal region in the infrared and visible light images are marked, and an abnormal region mask is generated. The pixel value of the abnormal region in the mask is set to 1, and the rest are 0.

5. The method for managing dual-light fusion detection data at the end of a robotic arm according to claim 4, characterized in that: Step S500 includes: Step S501: Generate a three - layer spiral detection path centered on the coordinates of the abnormal point. After receiving the coordinates of the abnormal area marked in step S402, record the coordinates of the abnormal point in space as (x0, y0, z0). Use this point as the center of the three - layer spiral detection path and also as the starting point of the spiral detection path. Set the initial radii of the first, second, and third - layer spiral paths as r1, r2, and r3 respectively, and satisfy r1 < r2 < r3. For each layer of the spiral path, generate path points using the conversion method from polar coordinates to rectangular coordinates: For the first - layer detection path, in the polar coordinate system, the generation method of the spiral line is based on the formula: ρ = r1 + k1×θ; where ρ represents the polar radius, k1 represents the pitch coefficient of the first - layer spiral line, which controls the increase in the polar radius when the spiral line rotates by a unit angle; θ represents the polar angle, which is the angle formed by rotating counterclockwise from the polar axis to the line connecting the pole and a certain point on the spiral line. Set a series of angle values θ i , θ i ∈[0, 2aπ], a represents the number of turns of the first - layer spiral line; Convert the polar coordinates (ρ i , θ i ) to rectangular coordinates (x i , y i , z i ), where ρ i represents the polar radius corresponding to θ i . Through the formula: x i = x0 + ρ i ×cosθ i , y i = y0 + ρ i ×sinθ i , z i = z0; Generate the coordinate points of the second - layer and third - layer spiral paths in the same way. By setting the corresponding angle value ranges respectively, and according to the above rectangular - coordinate conversion formula, obtain the rectangular coordinates of each point on the second - layer and third - layer spiral paths, thus obtaining a complete three - layer spiral detection path; Step S502: Convert the generated three-layer spiral detection path into motion commands for the robotic arm. Based on the kinematic model of the robotic arm, assign coordinates (x, y, z) to each point on the path. i ,y i ,z i The values ​​are converted into angle values ​​for each joint of the robotic arm. Using an inverse kinematics algorithm, combined with the link length and joint type of the robotic arm, the required rotation angle of each joint is calculated. These joint angle values ​​are arranged sequentially according to the path points to generate a motion command sequence for the robotic arm. The motion command sequence is sent to the robotic arm motion controller to trigger the robotic arm to move along a three-layer spiral detection path. During the movement, the robotic arm provides real-time feedback on its current pose information and performs closed-loop control by comparing it with preset path points. Simultaneously, when the robotic arm moves to each path point, a visible light camera and an infrared camera are triggered to acquire dual-light images.

6. The method for managing dual-light fusion detection data at the end of a robotic arm according to claim 5, characterized in that: Step S500 further includes: Step S503: For the dual-light image acquired by the robotic arm during the re-inspection process, repeat the operation process of steps S200-S400. Use the trained cable recognition model to predict the visible light image to obtain the prediction mask of the cable area. After binarization and edge detection, extract the cable boundary coordinate set. Then, map the cable boundary onto the infrared image and extract the corresponding temperature data matrix. Next, calculate the maximum temperature difference and temperature gradient matrix of the cable surface and compare them with the preset threshold to determine whether there is still an abnormal area. Based on the re-inspection results, issue an abnormal warning. If an abnormal area is still determined to exist after re-inspection, immediately activate the audible and visual alarm to issue an alarm signal and send abnormal information to the remote monitoring system. The abnormal information includes the coordinates of the abnormal area, the maximum temperature difference, and the temperature gradient. If the re-inspection results show that the abnormal area has disappeared, record this abnormality as a misjudgment and store the relevant data in the historical database for recording.

7. A dual-light fusion detection data management system for the end effector of a robotic arm, characterized in that: The system includes a data acquisition module, a cable identification module, a data fusion module, an anomaly detection module, and a re-inspection module; The data acquisition module integrates a visible light camera and an infrared camera at the end of the robotic arm. It sets discrete monitoring points according to the movement path of the robotic arm. Based on the pose data output by the motion controller, the robotic arm synchronously triggers dual-light image acquisition when the movement reaches the preset monitoring point, and performs spatiotemporal alignment on the acquired data. The cable recognition module performs semantic segmentation processing on visible light images, filters cable images from visible light image data to construct a basic dataset, and constructs a cable recognition model through the U-Net architecture to obtain the boundary coordinate set of the cable. The data fusion module performs feature matching between the cable boundary extracted from the visible light image and the infrared image, maps the cable area to the infrared image, and extracts the corresponding temperature data matrix. The anomaly detection module calculates the maximum temperature difference on the cable surface based on the mapped infrared temperature data matrix, and determines the abnormal area by combining it with a preset temperature gradient threshold. The re-inspection module triggers the robotic arm to dynamically adjust the detection path based on the abnormal detection results, focuses on the abnormal points for re-inspection, and provides abnormal warning prompts based on the re-inspection results.

Images