Method and system for detecting state of visual sensor of inspection robot

Abstract

The invention discloses an inspection robot visual sensor state detection method and system, and is applied to the technical field of visual sensor state detection, and the method comprises the steps: constructing a frame loss recognition model based on a historical visual sensing data frame loss set in combination with the motion data of an inspection robot; constructing an anomaly recognition model based on the historical visual sensing data anomaly set in combination with communication perception and structure perception; constructing a distortion recognition model based on the historical visual sensing data distortion set in combination with image consistency analysis; fusing the frame loss identification model, the anomaly identification model and the distortion identification model to construct a state detection model; and outputting a state detection result of the inspection robot vision sensor according to the current inspection robot motion data, the collection data and the state detection model. The method has the beneficial effect that the accuracy of state detection of the visual sensor of the inspection robot is improved through multi-dimensional detection.

Description

Technical Field

[0001] The present application relates to the technical field of visual sensor state detection, and in particular to a method and system for detecting the state of a visual sensor of an inspection robot. Background Art

[0002] In industrial scenarios, visual sensor status detection often faces misjudgment issues in complex interference environments. Related technologies typically rely on single-dimensional metrics to detect the status of inspection robot visual sensors. For example, these techniques use fixed thresholds to determine image sequence continuity to identify frame drops, or assess sensor hardware status based on the image quality of a single modality.

[0003] Related technologies analyze the correlation between visual sensor data, device motion, and communication link parameters in isolation. This makes it difficult to accurately distinguish between image distortions such as blur and distortion caused by device jitter and rapid movement in high-speed motion scenarios, and frame drop caused by unstable communication links. Furthermore, these technologies rely too heavily on manually preset static thresholds, making them incapable of adapting to the complex and changing nature of industrial environments, significantly increasing false alarm rates.

[0004] The patent, "A Method for Calibrating Visual Sensor Image Quality Using Correlated Sensors," published with publication number CN119484807A and date of publication on February 18, 2025, specifically discloses the following steps: S1) evaluating the quality of images captured by the visual sensor; S2) using a deep learning algorithm to train a long short-term memory (LSTM) model by analyzing the correlation between relevant sensor data and image feature quality; S3) using the LSTM model to predict image feature quality; and S4) adjusting visual sensor parameters based on image feature quality to optimize image quality. Although this solution predicts image feature quality by analyzing the correlation between relevant sensor data and image feature quality and adjusts visual sensor parameters based on image feature quality, it cannot detect visual sensor status anomalies when communication anomalies exist. Summary of the Invention

[0005] This application addresses the technical problems in the existing technology of low accuracy and inability to distinguish specific causes of abnormalities in the visual sensor status detection of patrol robots. It provides a method and system for detecting the status of patrol robot visual sensors. By constructing a frame loss recognition model based on motion optical flow, an abnormality recognition model based on communication perception and structural perception, and a distortion recognition model based on image consistency analysis, the three models are used to detect the current visual sensor status respectively, and the detection results and the source of the abnormality are output. The accuracy of the patrol robot visual sensor status detection is improved through multi-dimensional detection.

[0006] To achieve the above-mentioned technical objectives, the present application provides a technical solution, which is a method for detecting the state of a visual sensor of an inspection robot, comprising the following steps: constructing a frame loss recognition model based on a set of historical visual sensor data frame losses combined with the inspection robot motion data; constructing an anomaly recognition model based on a set of historical visual sensor data anomalies combined with communication perception and structural perception; constructing a distortion recognition model based on a set of historical visual sensor data distortions combined with image consistency analysis; constructing a state detection model by fusing the frame loss recognition model, the anomaly recognition model and the distortion recognition model; and outputting the inspection robot visual sensor state detection result based on the current inspection robot motion data, collected data and the state detection model.

[0007] Furthermore, the frame loss recognition model constructed based on the frame loss set of historical visual sensor data combined with the motion data of the inspection robot includes: constructing a motion optical flow prediction model based on the historical inspection robot motion data and the theoretical optical flow; obtaining the theoretical optical flow data corresponding to the frame loss set of historical visual sensor data based on the motion optical flow prediction model; obtaining adjacent frame optical flow data based on the frame loss set of historical visual sensor data, and obtaining an abnormality threshold based on the adjacent frame optical flow data and the theoretical optical flow data; and constructing a frame loss recognition model based on the abnormality threshold and the motion optical flow prediction model.

[0008] Furthermore, the construction of a motion optical flow prediction model based on historical inspection robot motion data and theoretical optical flow includes: obtaining a motion prediction model based on the state transfer equation and historical inspection robot motion data; and using an extended Kalman filter to construct a motion optical flow prediction model based on the theoretical optical flow, the motion prediction model, the first motion data, and historical normal sampling images.

[0009] Furthermore, the method of obtaining an abnormality threshold based on adjacent frame optical flow data and theoretical optical flow data includes: calculating the similarity between the adjacent frame optical flow data and the theoretical optical flow data corresponding to the time sequence according to Euclidean calculation; and clustering the similarity using a clustering algorithm to obtain an abnormality threshold set corresponding to the state characteristics.

[0010] Furthermore, the construction of an anomaly recognition model based on an anomaly set of historical visual sensor data combined with communication perception and structural perception includes: obtaining frequency domain features of noise signals and time domain features of heartbeat packets based on an anomaly set of historical visual sensor data, and constructing a first anomaly recognition model corresponding to communication perception according to the frequency domain features of noise signals and the time domain features of heartbeat packets; obtaining image parameter anomaly features based on an anomaly set of historical visual sensor data, and constructing a second anomaly recognition model corresponding to structural perception according to the image parameter anomaly features.

[0011] Furthermore, the method of obtaining the frequency domain features of the noise signal and the time domain features of the heartbeat packet based on the anomaly set of historical visual sensor data includes: extracting the frequency domain features of the noise signal in the anomaly set of historical visual sensor data using spectrum analysis; and extracting the time domain features of the heartbeat packet in the anomaly set of historical visual sensor data based on interval jitter and response loss rate.

[0012] Furthermore, the method of obtaining abnormal features of image parameters based on an abnormal set of historical visual sensor data and constructing a second abnormality recognition model corresponding to structural perception according to the abnormal features of image parameters includes: obtaining vibration data, environmental data and optical parameters in the abnormal set of historical visual sensor data, and constructing a second abnormality recognition model corresponding to structural perception based on the vibration data, environmental data and optical parameters.

[0013] Furthermore, the second abnormality recognition model corresponding to structural perception is constructed based on vibration data, environmental data and optical parameters, including: obtaining mechanical stress according to vibration data and temperature and humidity data, constructing an offset influence relationship between mechanical stress and optical parameters according to mechanical stress and optical parameters, and constructing a second abnormality recognition model corresponding to structural perception based on the offset influence relationship.

[0014] Furthermore, the construction of a distortion recognition model based on the distortion set of historical visual sensor data combined with image consistency analysis includes: using Canny edge detection to obtain a real edge map based on the visible light image of the distortion set of historical visual sensor data; using the Sobel operator to obtain a real gradient map based on the infrared image of the distortion set of historical visual sensor data; using cross-modal conversion to obtain a simulated edge map and a simulated gradient map based on the distortion set of historical visual sensor data; calculating the edge similarity between the real edge map and the simulated edge map, and the gradient similarity between the real gradient map and the simulated gradient map, and obtaining the edge weight, gradient weight and comprehensive similarity threshold of the distortion recognition model based on the edge similarity and gradient similarity training.

[0015] Another technical solution provided by the present application is a patrol robot visual sensor status detection system for implementing the method as described above, including: a communication unit, which maintains communication and data interaction with the patrol robot through a wired or wireless network, and obtains the patrol robot motion data and collected data; a processing unit, which is used to construct a frame loss recognition model, an abnormality recognition model and a distortion recognition model based on historical visual sensor data, and perform patrol robot visual sensor status detection based on the current patrol robot motion data and collected data.

[0016] The beneficial effects of the present application are as follows: 1. Pre-process the historical visual sensor data according to the frame loss situation, abnormal situation and distortion situation to obtain the frame loss set, abnormal situation and distortion set. Build a frame loss recognition model, an abnormality recognition model and a distortion recognition model through the frame loss set, abnormality set and distortion set in the historical visual sensor data, and build a state detection model covering the entire fault chain of the visual sensor system by fusing the three independent recognition models of frame loss, abnormality and distortion. At the same time, combine motion data, communication perception, structural perception and image consistency to accurately distinguish different state abnormality types such as image blur caused by hardware jitter, network transmission frame loss, and physical damage to the sensor, thereby improving the accuracy of visual sensor state detection.

[0017] 2. A first anomaly recognition model is constructed based on the frequency domain characteristics of the clutter signal and the time domain characteristics of the heartbeat packet to determine communication anomalies. A second anomaly recognition model corresponding to structural perception is constructed based on vibration data, environmental data, and optical parameters to determine result anomalies. The accuracy of visual sensor anomaly recognition is improved by utilizing joint communication-structure diagnosis.

[0018] 3. Canny edge detection is used to extract a true edge map with geometric structure fidelity from visible light images, while the Sobel operator is used to capture a true gradient map of thermal radiation characteristics from infrared images. A cross-modal conversion network is constructed using CycleGAN to generate simulated gradient maps for the visible light modality and simulated edge maps for the infrared modality, respectively, overcoming the limitations of single-modal feature representation. The method then jointly calculates the edge structure similarity and gradient distribution similarity between the real and simulated images, and trains edge weights, gradient weights, and a comprehensive threshold based on an adaptive weighted fusion mechanism. This enables the distortion recognition model to accurately identify feature consistency deviations between cross-modal data, thereby performing distortion detection on visual sensors. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 This is a flow chart of the visual sensor status detection method for the inspection robot in this application.

[0020] Figure 2 This is a structural diagram of the visual sensor status detection system of the inspection robot in this application. DETAILED DESCRIPTION

[0021] In order to make the purpose, technical solutions and advantages of this application more clear, the application is further described in detail below with reference to the accompanying drawings and examples. It should be understood that the specific implementation method described here is only an optimal embodiment of this application, which is only used to explain this application and does not limit the scope of protection of this application. All other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0022] like Figure 1 As shown in the first embodiment of the present application, the inspection robot visual sensor state detection method includes the following steps: A frame loss recognition model is constructed based on the frame loss collection of historical visual sensor data and the motion data of the inspection robot; Anomaly recognition models are constructed based on anomaly collections of historical visual sensor data combined with communication perception and structural perception. A distortion recognition model is constructed based on the distortion set of historical visual sensor data combined with image consistency analysis; The state detection model is constructed by integrating the frame loss recognition model, the anomaly recognition model, and the distortion recognition model; The inspection robot visual sensor status detection results are output based on the current inspection robot motion data, collected data and status detection model.

[0023] In this embodiment, historical visual sensor data is preprocessed based on frame loss, anomalies, and distortion to obtain a set of frame loss, anomaly, and distortion data. A frame loss recognition model, anomaly recognition model, and distortion recognition model are constructed using these sets of frame loss, anomaly, and distortion data. By integrating these three independent recognition models, a state detection model covering the entire fault chain of the visual sensor system is constructed. Furthermore, motion data, communication perception, structural perception, and image consistency are combined to accurately distinguish between different state anomaly types, such as image blur caused by hardware jitter, network transmission frame loss, and physical sensor damage, thereby improving the accuracy of visual sensor state detection.

[0024] Specifically, the frame loss recognition model is constructed based on the historical visual sensor data frame loss set and the inspection robot motion data, including: Construct a motion optical flow prediction model based on historical inspection robot motion data and theoretical optical flow; Obtain theoretical optical flow data corresponding to the lost frame set of historical visual sensor data based on the motion optical flow prediction model; Obtain adjacent frame optical flow data based on the historical visual sensor data loss frame set, and obtain the abnormal threshold value based on the adjacent frame optical flow data and theoretical optical flow data; A frame loss recognition model is constructed based on abnormal threshold and motion optical flow prediction model.

[0025] By obtaining the theoretical changes of optical flow during the motion process based on historical inspection robot motion data and theoretical optical flow algorithm, and obtaining actual adjacent frame optical flow data based on the historical visual sensor data frame loss set, the anomaly threshold is obtained according to the difference between the adjacent frame optical flow data and the theoretical optical flow data, thereby associating the anomaly threshold with the inspection robot motion (the state of the visual sensor during the motion process), improving the adaptability of the anomaly threshold, and significantly enhancing the robustness of frame loss detection to motion blur.

[0026] The motion optical flow prediction model is constructed based on the historical inspection robot motion data and theoretical optical flow, including: Obtain motion prediction model based on state transition equation and historical inspection robot motion data; An extended Kalman filter is used to construct a motion optical flow prediction model according to the theoretical optical flow, the motion prediction model, the first motion data and the historical normal sampling images.

[0027] Specifically, the state vector of the inspection robot is defined as: X i =[x i ,y i ,θ i ,v i ,ω i ]; Among them, X i represents the motion state vector of the inspection robot at time i, x i represents the horizontal coordinate of the inspection robot at time i, y i represents the longitudinal coordinate of the inspection robot at time i, θ i represents the direction angle of the inspection robot at time i, v i represents the linear velocity of the inspection robot at time i, ω i represents the angular velocity of the inspection robot at time i.

[0028] The kinematic model is used to define the state transfer equation to obtain the motion prediction model: X i+1|i =f(X i ,u i ,dt); Among them, X i+1|i represents the predicted value of the inspection robot's motion state at time i for time i+1, u i represents the control input at time i, and dt represents the time step.

[0029] In this embodiment, the control input is obtained based on the acceleration data and the gyroscope data.

[0030] The Kalman filter is used to map the inspection robot's motion state prediction value to the optical flow space, and the motion optical flow prediction model is constructed as follows: Z i+1 =h(X i+1 )+δ i+1 ; Among them, Z i+1 represents the predicted optical flow at time i+1, h represents the observation function, δ i+1 represents the observation noise at time i+1.

[0031] The historical inspection robot motion data includes at least first motion data corresponding to historical normal sampling images and second motion data corresponding to a set of lost frames of historical visual sensor data. The historical normal sampling images are historical image samples collected when the visual sensor sampling is normal. The observation function and the observation noise are obtained by training based on the first motion data and the historical normal sampling images. The correlation between motion and optical flow is obtained based on the historical normal sampling conditions. It is understandable that when constructing the motion prediction model, all historical inspection robot motion data are used for training to enable a more accurate prediction of the motion of the inspection robot at the next moment, while when training the correlation between motion and optical flow, only the historical normal sampling images and the first motion data are used for training to avoid interference from abnormal data and optical flow data.

[0032] Furthermore, the theoretical optical flow data corresponding to the lost frame set of historical visual sensor data obtained based on the motion optical flow prediction model includes: Acquire second motion data from the historical inspection robot motion data according to the historical visual sensor data lost frame set; Theoretical optical flow data is obtained according to the second motion data and the motion optical flow prediction model.

[0033] According to the motion optical flow prediction model, the theoretical optical flow data at the moment of frame loss under normal circumstances is obtained and used as the abnormal comparison benchmark.

[0034] Obtaining adjacent frame optical flow data based on the historical visual sensor data loss frame set includes: Based on the optical flow algorithm, the optical flow data of adjacent frames is calculated according to the lost frame set of historical visual sensor data.

[0035] The optical flow algorithm is: O=[v l ,v h ] T ; Among them, O represents optical flow, v l Indicates the horizontal motion speed of the feature point on the image plane, v h It represents the vertical motion speed of the feature point on the image plane, and T represents the transpose of the matrix.

[0036] The actual optical flow data of adjacent frames is calculated based on the historical visual sensor data loss frame set, and the abnormal threshold is obtained based on the adjacent frame optical flow data and the theoretical optical flow data. Calculate the similarity between the optical flow data of adjacent frames corresponding to the time sequence and the theoretical optical flow data according to Euclidean method; Clustering algorithm is used to cluster similarities to obtain the abnormal threshold set corresponding to the state characteristics.

[0037] According to the Euclidean calculation time sequence, the similarity between the optical flow data of adjacent frames and the theoretical optical flow data is: Among them, τ represents the cosine similarity, O r Represents the adjacent frame optical flow data, O p Represents theoretical optical flow data, ||O r || represents the norm (Euclidean norm) of the optical flow data of adjacent frames, || O p || represents the norm of the theoretical optical flow data (Euclidean norm).

[0038] Then, the similarities of the same state features are clustered to obtain the abnormal threshold set of the corresponding state features. The state feature is the motion state of the inspection robot, which is represented by the state vector.

[0039] At this time, the frame loss recognition model based on the abnormal threshold and motion optical flow prediction model includes: Construct a dynamic recognition layer based on the maximum value of the anomaly threshold in the anomaly threshold set; A frame loss recognition model is constructed using the dynamic recognition layer and the motion optical flow prediction model.

[0040] The maximum value of the abnormal threshold in the abnormal threshold set is used as the dynamic abnormal threshold corresponding to the state feature, and the dynamic recognition layer is constructed with the dynamic abnormal threshold matched by all state features. When the motion optical flow prediction model outputs the optical flow prediction value and the motion state prediction value at the next moment, the dynamic recognition layer matches the dynamic abnormal threshold according to the motion state prediction value, calculates the similarity between the optical flow prediction value and the actual optical flow value at the next moment, and compares the similarity with the dynamic abnormal threshold. When the similarity is less than or equal to the dynamic abnormal threshold, it is considered that the current visual sensor has frame loss phenomenon. When the similarity is greater than the dynamic abnormal threshold, it is considered that the current visual sensor does not have frame loss phenomenon.

[0041] Building an anomaly recognition model based on an anomaly set of historical visual sensor data combined with communication perception and structure perception includes: obtaining frequency domain features of clutter signals and time domain features of heartbeat packets based on the anomaly set of historical visual sensor data, and building a first anomaly recognition model corresponding to communication perception based on the frequency domain features of clutter signals and the time domain features of heartbeat packets; Based on the anomaly set of historical visual sensor data, the abnormal features of image parameters are obtained, and a second anomaly recognition model corresponding to structure perception is constructed according to the abnormal features of image parameters.

[0042] Acquiring the frequency domain features of the clutter signal and the time domain features of the heartbeat packet based on the anomaly set of historical visual sensor data includes: extracting the frequency domain features of the clutter signal in the anomaly set of historical visual sensor data by using spectrum analysis; The time domain features of heartbeat packets in the anomaly set of historical visual sensor data are extracted based on interval jitter and response loss rate.

[0043] Therefore, the attention mechanism is used to construct the first anomaly model for communication perception based on the frequency domain characteristics of the clutter signal and the timing characteristics of the heartbeat packet.

[0044] Obtaining image parameter anomaly features based on the anomaly set of historical visual sensor data, and constructing a second anomaly recognition model corresponding to structural perception based on the image parameter anomaly features include: Vibration data, environmental data, and optical parameters in the anomaly set of historical visual sensing data are obtained, and a second anomaly recognition model corresponding to structural perception is constructed based on the vibration data, environmental data, and optical parameters.

[0045] In this embodiment, a first abnormality recognition model is constructed based on the frequency domain characteristics of the clutter signal and the time domain characteristics of the heartbeat packet to realize the judgment of communication abnormalities. A second abnormality recognition model corresponding to structural perception is constructed based on vibration data, environmental data and optical parameters to realize the judgment of result abnormalities. The joint diagnosis of communication and structure is utilized to improve the accuracy of abnormality recognition of visual sensors.

[0046] Among them, the first anomaly recognition model is trained based on the anomaly set of historical visual sensor data to obtain the influence weights of clutter signals and heartbeat packet anomalies on communication, so that in the subsequent anomaly recognition process, it is possible to identify whether the current visual sensor data has an anomaly based on the diagnosis of the communication protocol and the extraction of clutter signals.

[0047] In actual applications, vibration data is obtained according to the vibration sensor, and temperature and humidity data is obtained according to the temperature and humidity sensor. At this time, a second abnormality recognition model corresponding to structural perception is constructed based on the vibration data, environmental data and optical parameters, including: obtaining mechanical stress according to the vibration data and temperature and humidity data, constructing the offset influence relationship between mechanical stress and optical parameters according to the mechanical stress and optical parameters, and constructing the second abnormality recognition model corresponding to structural perception based on the offset influence relationship.

[0048] Vibration data, temperature and humidity data, and their effects on optical parameters are determined using vibration and temperature and humidity data. The optical parameter anomaly threshold is then determined by excluding the anomaly set of historical visual sensor data that indicates communication anomalies. When the optical parameters calculated based on the offset influence relationship meet the optical parameter anomaly threshold, the structure is considered abnormal. By comprehensively considering vibration data, temperature and humidity data, and optical parameters, the structure's status is monitored and analyzed from multiple perspectives. Compared to anomaly identification methods based on a single data source, this method comprehensively reflects the actual state of the structure, reducing misjudgments and missed detections. Furthermore, the source of the anomaly can be determined based on the offset influence relationship and communication perception, facilitating operator investigation.

[0049] The distortion recognition model is constructed based on the distortion set of historical visual sensor data combined with image consistency analysis, including: Using Canny edge detection to obtain the real edge map based on the visible light image of the distorted set of historical visual sensor data; The Sobel operator is used to obtain the true gradient map based on the infrared image of the distorted set of historical visual sensor data; Using cross-modal transformation to obtain simulated edge maps and simulated gradient maps based on the distortion set of historical visual sensor data; The edge similarity between the real edge map and the simulated edge map, and the gradient similarity between the real gradient map and the simulated gradient map are calculated. According to the edge similarity and gradient similarity training, the edge weight, gradient weight and comprehensive similarity threshold of the distortion recognition model are obtained.

[0050] For the set of historical visual sensor data distortions, a true edge map corresponding to the visible light image and a true gradient map corresponding to the infrared image are constructed. A cross-modal conversion network (which can adopt the CycleGAN architecture) is then used to construct a simulated gradient map for the visible light image and a simulated edge map for the infrared image. The true and simulated edge maps, as well as the true and simulated gradient maps, are compared in the same time series, and their similarity is calculated. Based on the set of historical visual sensor data distortions, the edge weights and gradient weights are jointly optimized using a backpropagation algorithm to calculate a comprehensive similarity. A comprehensive similarity threshold is obtained based on the overall comprehensive similarity of the historical data. This allows the comprehensive similarity to be calculated based on the edge weights of the true and simulated edge maps, and the gradient weights of the true and simulated gradient maps. When the comprehensive similarity meets the comprehensive similarity threshold, visual sensor distortion is considered to be present.

[0051] In this embodiment, by fusing the complementary characteristics of visible light and infrared dual-modal data, combining cross-modal generative adversarial networks with multi-dimensional image consistency analysis, highly robust distortion recognition is achieved. Specifically, Canny edge detection is used to extract a true edge map with geometric structure fidelity from the visible light image, while the Sobel operator is used to capture the true gradient map of thermal radiation characteristics from the infrared image. A cross-modal conversion network is constructed through CycleGAN to generate simulated gradient maps of the visible light modality and simulated edge maps of the infrared modality, respectively, breaking through the limitations of single-modal feature expression. Then, by jointly calculating the edge structure similarity and gradient distribution similarity between the real and simulated images, and training based on the adaptive weighted fusion mechanism to obtain edge weights, gradient weights and comprehensive thresholds, the distortion recognition model can accurately identify the feature consistency deviation between cross-modal data, thereby performing distortion detection on the visual sensor.

[0052] Based on the dynamic attention fusion mechanism, the state detection model is constructed by adjusting the fusion weights according to the real-time confidence of the frame loss recognition model, the anomaly recognition model, and the distortion recognition model. Furthermore, based on the current inspection robot motion data, collected data, and the state detection model, the inspection robot visual sensor state detection results are output, including: Obtain the real-time state vector, vibration data and environmental data of the inspection robot based on the current inspection robot motion data; Acquire real-time communication data and real-time image data based on the data collected by the current inspection robot; Based on the real-time state vector, vibration data, environmental data, real-time communication data and real-time image data of the inspection robot, the state detection results of the inspection robot's visual sensor are used to output the frame loss recognition results, anomaly recognition results and distortion recognition results.

[0053] In this embodiment, the difference between the optical flow prediction value and the actual optical flow value is obtained according to the real-time state vector of the inspection robot to determine whether there is frame loss, and whether there is communication abnormality is determined according to the noise and heartbeat packet conditions of the real-time communication signal. It is determined whether there is structural abnormality based on the image parameters, vibration data and environmental data of the real-time image data. It is determined whether there is distortion based on the consistency of the visible light and infrared dual-modal data of the real-time image data. The visual sensor state detection result is output according to the judgment result, and the corresponding abnormality source is output, so as to improve the accuracy of the visual sensor state detection by using multi-dimensional comprehensive detection.

[0054] As a second embodiment of the present application, the inspection robot visual sensor state detection method further includes: Obtain control host network bandwidth data and visual sensor exposure time; If the control host network bandwidth data and the visual sensor exposure time meet the preset conditions, the inspection robot visual sensor status detection result is output based on the current inspection robot motion data, collected data and status detection model; If the control host network bandwidth data and the visual sensor exposure time do not meet the preset conditions, a bandwidth abnormality or line frequency abnormality signal will be output.

[0055] Before outputting the inspection robot's visual sensor status detection results based on the current inspection robot's motion data, collected data, and status detection model, check whether the control host's network bandwidth is sufficient, and check whether the exposure time of each sensor in the configured task exceeds the set 1 / line frequency. If any conditions are not met, prompt the operator to make adjustments first, thereby saving visual sensor status detection time and improving detection efficiency.

[0056] Among them, the inspection robot visual sensor state detection method also includes: Obtain the data collected by the inspection robot, and determine whether there is image distortion abnormality based on the aspect ratio of the ROI area image in the collected data. If so, output an image distortion abnormality signal. If not, output the inspection robot visual sensor status detection result based on the current inspection robot motion data, collected data, and status detection model; Obtain the data collected by the inspection robot, and determine whether there is an image parameter abnormality based on the image contrast and image brightness in the collected data. If so, output the image parameter abnormality signal; if not, output the inspection robot visual sensor status detection result based on the current inspection robot motion data, collected data and status detection model.

[0057] When problems occur in the aspect ratio, image contrast, and image brightness of the image in the ROI area, the operator is prompted to adjust the light source controller and the fill light device.

[0058] As a third embodiment of the present application, a visual sensor state detection system for an inspection robot includes: The communication unit maintains communication and data exchange with the inspection robot through a wired or wireless network, obtains the inspection robot's motion data and collects data; The processing unit is used to build a frame loss recognition model, anomaly recognition model and distortion recognition model based on historical visual sensor data, and perform inspection robot visual sensor status detection based on the current inspection robot motion data and collected data.

[0059] In this embodiment, the processing unit is connected to the communication unit, obtains the inspection robot motion data output by the communication unit, and collects the data to perform status detection of the inspection robot visual sensor.

[0060] like Figure 2As shown, in other cases, the inspection robot visual sensor state detection system also includes: A timing unit, used to trigger the main control unit to perform visual sensor status detection; A storage unit, used for storing visual sensor status data; The main control unit is used to receive the timing information from the timing unit, control the processing unit to perform visual sensor status detection, and store the status detection result in the storage unit.

[0061] The timing unit, storage unit, processing unit and communication unit are all connected to the main control unit.

[0062] The specific implementation described above is a preferred implementation of the inspection robot visual sensor state detection method and system of this application, and is not intended to limit the specific implementation scope of this application. The scope of this application includes but is not limited to this specific implementation. Any equivalent changes made in accordance with the shape and structure of this application are within the scope of protection of this application.

Claims

1. A method for detecting the state of a visual sensor of an inspection robot, characterized in that: The steps include: A frame loss recognition model is constructed based on the frame loss collection of historical visual sensor data and the motion data of the inspection robot; Anomaly recognition models are constructed based on anomaly collections of historical visual sensor data combined with communication perception and structural perception. A distortion recognition model is constructed based on the distortion set of historical visual sensor data combined with image consistency analysis; The state detection model is constructed by integrating the frame loss recognition model, the anomaly recognition model, and the distortion recognition model; The inspection robot visual sensor status detection results are output based on the current inspection robot motion data, collected data and status detection model.

2. The inspection robot visual sensor state detection method according to claim 1, characterized in that: The frame loss recognition model constructed based on the frame loss set of historical visual sensor data combined with the inspection robot motion data includes: Construct a motion optical flow prediction model based on historical inspection robot motion data and theoretical optical flow; Obtain theoretical optical flow data corresponding to the lost frame set of historical visual sensor data based on the motion optical flow prediction model; Obtain adjacent frame optical flow data based on the historical visual sensor data loss frame set, and obtain the abnormal threshold value based on the adjacent frame optical flow data and theoretical optical flow data; A frame loss recognition model is constructed based on abnormal threshold and motion optical flow prediction model.

3. The inspection robot visual sensor state detection method according to claim 2, characterized in that: The construction of a motion optical flow prediction model based on historical inspection robot motion data and theoretical optical flow includes: Obtain motion prediction model based on state transition equation and historical inspection robot motion data; An extended Kalman filter is used to construct a motion optical flow prediction model according to the theoretical optical flow, the motion prediction model, the first motion data and the historical normal sampling images.

4. The inspection robot visual sensor state detection method according to claim 2, characterized in that: The obtaining of an abnormality threshold according to adjacent frame optical flow data and theoretical optical flow data includes: Calculate the similarity between the optical flow data of adjacent frames corresponding to the time sequence and the theoretical optical flow data according to Euclidean method; Clustering algorithm is used to cluster similarities to obtain the abnormal threshold set corresponding to the state characteristics.

5. The inspection robot visual sensor state detection method according to claim 1, characterized in that: The anomaly recognition model constructed based on the anomaly set of historical visual sensor data combined with communication perception and structure perception includes: Based on the anomaly set of historical visual sensor data, the frequency domain features of the clutter signal and the time domain features of the heartbeat packet are obtained, and a first anomaly recognition model corresponding to communication perception is constructed according to the frequency domain features of the clutter signal and the time domain features of the heartbeat packet; Based on the anomaly set of historical visual sensor data, the abnormal features of image parameters are obtained, and a second anomaly recognition model corresponding to structure perception is constructed according to the abnormal features of image parameters.

6. The inspection robot visual sensor state detection method according to claim 5, characterized in that: The acquisition of the frequency domain features of the clutter signal and the time domain features of the heartbeat packet based on the anomaly set of historical visual sensor data includes: Utilize spectrum analysis to extract frequency domain features of clutter signals from anomaly sets of historical visual sensor data; The time domain features of heartbeat packets in the anomaly set of historical visual sensor data are extracted based on interval jitter and response loss rate.

7. The inspection robot visual sensor state detection method according to claim 5, characterized in that: The method of obtaining an abnormal feature of image parameters based on an abnormal set of historical visual sensor data and constructing a second abnormality recognition model corresponding to structural perception according to the abnormal feature of image parameters comprises: Vibration data, environmental data, and optical parameters in the anomaly set of historical visual sensing data are obtained, and a second anomaly recognition model corresponding to structural perception is constructed based on the vibration data, environmental data, and optical parameters.

8. The inspection robot visual sensor state detection method according to claim 7, characterized in that: The second abnormality recognition model corresponding to the structural perception is constructed based on the vibration data, the environmental data and the optical parameters, including: Mechanical stress is obtained based on vibration data and temperature and humidity data, and the offset influence relationship between mechanical stress and optical parameters is constructed based on the mechanical stress and optical parameters. A second abnormality recognition model corresponding to structural perception is constructed based on the offset influence relationship.

9. The inspection robot visual sensor state detection method according to claim 1, characterized in that: The distortion recognition model constructed based on the historical visual sensor data distortion set combined with image consistency analysis includes: Using Canny edge detection to obtain the real edge map based on the visible light image of the distorted set of historical visual sensor data; The Sobel operator is used to obtain the true gradient map based on the infrared image of the distorted set of historical visual sensor data; Using cross-modal transformation to obtain simulated edge maps and simulated gradient maps based on the distortion set of historical visual sensor data; The edge similarity between the real edge map and the simulated edge map, and the gradient similarity between the real gradient map and the simulated gradient map are calculated. According to the edge similarity and gradient similarity training, the edge weight, gradient weight and comprehensive similarity threshold of the distortion recognition model are obtained.

10. A visual sensor state detection system for an inspection robot, configured to implement the method according to any one of claims 1 to 9, characterized in that: include: The communication unit maintains communication and data exchange with the inspection robot through a wired or wireless network, obtains the inspection robot's motion data and collects data; The processing unit is used to build a frame loss recognition model, anomaly recognition model and distortion recognition model based on historical visual sensor data, and perform inspection robot visual sensor status detection based on the current inspection robot motion data and collected data.

Images