A deep-sea heavy-duty operation robot for autonomous fault detection based on image recognition

Abstract

The present invention discloses a deep-sea heavy-duty operation robot based on autonomous fault detection by image recognition. The robot includes the following modules: an image acquisition and processing module; a device driving module; a calculation module; a fault recognition module. Among them, in the present invention, the neural network model of the image acquisition and processing module is used to identify whether there is a probability of environmental intervention fault in the marine environment around the robot, and based on the image recognition, the stroke and rotation angle of the driving device itself under different time-driven actions are obtained. Further, the device driving module and the calculation module are used to judge the stroke and rotation angle of the driving device itself based on the motor current of the driving device. By synthesizing the above various reference data, the final fault type of the robot is obtained. The present invention can significantly improve the accuracy of fault recognition by introducing the stroke and rotation angle of environmental intervention fault and motor current integral mapping processing.

Description

Technical Field

[0001] The present invention relates to the technical field of deep - sea robot, and particularly to a deep - sea heavy - duty operation robot based on autonomous fault detection by image recognition. Background Art

[0002] At present, due to the influence of the extreme deep - sea environment, deep - sea operation robots are prone to various faults. The common fault types of deep - sea operation robots include the following: mechanical structure faults, environmental intervention faults, drive device faults, electrical circuit faults, energy system faults, sensor faults, software communication faults, etc. Among them, the most common faults faced by deep - sea operation robots are mainly mechanical structure faults, environmental intervention faults, drive device faults, electrical circuit faults, and sensor faults. Due to the complex and extreme deep - sea environment, the above - mentioned fault types will seriously affect the operation ability of deep - sea operation robots, and there is no technical solution in the prior art to accurately identify the above - mentioned fault types of deep - sea operation robots and give solutions corresponding to the fault types. Summary of the Invention

[0003] One object of the present invention is to provide a deep - sea heavy - duty operation robot based on autonomous fault detection by image recognition. The robot uses an image recognition module to perform image recognition in multiple directions and angles around the deep - sea robot, and performs image stitching processing on the recognized images, including but not limited to, to obtain a panoramic image around the deep - sea robot, and uses the panoramic image as a fault reference factor for image feature recognition to identify whether there may be environmental intervention faults for the robot, and serves as reference data for subsequent analysis of fault types.

[0004] One object of the present invention is to provide a deep - sea heavy - duty operation robot based on autonomous fault detection by image recognition. The robot uses the image recognition module to identify and obtain the action mechanism images of the deep - sea operation robot, where the action mechanism images include a first action image and a second action image after driving. Based on the first action image and the second action image and combined with the panoramic image of the environment around the deep - sea robot captured and recognized by the panoramic camera, it is distinguished and judged whether there are mechanical mechanism faults or drive device faults at present, and combined with the current state data of the drive motor of the corresponding drive device in different second action images, it is further distinguished and judged whether there is a drive device fault state.

[0005] One of the invention objects of the present invention is to provide a deep - sea heavy - duty operation robot for autonomous fault detection based on image recognition. During the fault detection process, the robot uses the state data of the corresponding drive devices under drive signals at different time series. The state data includes the rotation angle and the traveled distance. The time integral processing is performed on the state data of the drive devices at the corresponding detection time series, and at the same time, the time integral processing of the corresponding current is performed according to the mapping rule of the motor current of the drive device. The difference between the distance integral result and the mapped current integral result based on the action image frame is calculated by selecting the action reference points of any two time series, and the maximum value, average value, and sum value of the differences between the two integral results of any two time series are obtained as the judgment reference data for the faults of the drive device. Thus, the accuracy of fault judgment during the entire driving process of the drive device is improved.

[0006] In order to achieve at least one of the above - mentioned invention objects, the present invention provides a deep - sea heavy - duty operation robot for autonomous fault detection based on image recognition. The robot includes:

[0007] An image acquisition and processing module;

[0008] A device drive module;

[0009] A calculation module;

[0010] A fault recognition module;

[0011] The image acquisition and processing module acquires the images of the ocean environment around the robot and performs image processing on the images of the ocean environment around the robot to obtain a panoramic image of the ocean environment around the robot;

[0012] The device drive module is used to generate the first action signal and the second action signal for the corresponding drive device. The image recognition module intercepts the image information of the corresponding drive device according to the time series of the first action signal and the second action signal;

[0013] The image acquisition and processing module inputs the panoramic ocean images of the corresponding time series into a neural network model based on the first action signal and the second action signal for training to obtain an environmental intervention fault recognition model for identifying the probability of environmental intervention faults around the robot;

[0014] The calculation module acquires the current data of the first action signal and the second action signal of the device drive module at the corresponding time series, and performs time - integral mapping processing on the current of the corresponding drive device according to the time series based on the motor dynamic speed equation and the motor torque equation; performs feature analysis including travel and angle on the intercepted image information of the corresponding drive device according to the same time series;

[0015] The fault identification module determines the type of fault existing in the current robot based on the identification result of the environmental intervention fault identification model, the time integral mapping processing result of the motor dynamic speed equation and the motor torque equation, and the feature analysis result of the stroke and angle of the corresponding driving device image information.

[0016] According to one preferred embodiment of the present invention, the image acquisition and processing module respectively acquires multiple marine environment images with edge overlapping areas around the robot under the time series of the first action signal and the second action signal, uses the ORB algorithm to extract features from multiple marine environment images with edge overlapping areas respectively, obtains the key points of the marine environment images, performs key point matching on the extracted features, constructs BRIEF descriptors invariant to the rotation of the corresponding key points, calculates the similarity values of the BRIEF descriptors invariant to the rotation of the key points of different marine environment images, and if the similarity value is greater than a preset threshold, performs feature stitching and fusion based on the similar key points of the different marine environment images to obtain the panoramic marine environment image.

[0017] According to another preferred embodiment of the present invention, the image recognition and processing module identifies and labels the key points of the corresponding panoramic marine environment image according to the ORB algorithm, obtains the depth information of each key point, performs feature transformation processing on the key point pixel features and the depth information of each key point to obtain standardized panoramic image features, and inputs the standardized panoramic image features into the trained environmental intervention fault identification model to identify the environmental intervention fault probability of each key point. If there is at least one key point with an environmental intervention fault probability greater than a preset probability threshold, the position coordinates of the corresponding key point in the panoramic image are output, and the type of the corresponding environmental intervention fault probability is output.

[0018] According to another preferred embodiment of the present invention, the device drive module generates a first action signal and a second action signal, and the calculation module respectively obtains the time series t1 of the first action signal and the time series t2 of the second action signal. Among them, according to the time series t1 and the time series t2, the motor instantaneous currents I t1 and I t2 of the corresponding driving device are respectively obtained. The calculation module calculates the first stroke I1 or the first rotation angle θ1 experienced by the corresponding driving device from the time series t1 to the time series t2 according to the motor torque equation H = K H *I t and the motor dynamic speed equation according to the following formula: , where D ∈ [t1, t2], where K H is the torque constant, * is the product calculation, I t is the instantaneous current, D is the integration domain, J is the moment of inertia, and λ is the stroke transmission coefficient or the angle transmission coefficient.

[0019] According to another preferred embodiment of the present invention, after the image acquisition and processing module intercepts the image information of the corresponding driving device at the time series t1 and the time series t2, it uses a target detection model to detect the travel I2 or the rotation angle θ2 of the corresponding driving device on the image interface from the time series t1 to the time series t2, and calculates the absolute value of the travel difference and the absolute value of the rotation angle difference , and based on the absolute value of the travel difference , the absolute value of the rotation angle difference and the comprehensive analysis of the key point environmental intervention failure probability of the corresponding driving device to judge the current failure type.

[0020] In order to achieve at least one of the above-mentioned invention purposes, the present invention provides a method for autonomous fault detection of a deep-sea heavy-duty operation robot based on image recognition, and the method includes:

[0021] S01. Collect the ocean environment image around the robot, and perform image processing on the surrounding ocean environment image to obtain the panoramic image of the robot's ocean environment;

[0022] S02. Generate a first action signal and a second action signal, and obtain the time series of the first action signal and the second action signal. Intercept the image information and the instantaneous current of the driving device according to the time series of the first action signal and the time series of the second action signal respectively;

[0023] S03. Perform time integral mapping processing on the instantaneous current of the driving device according to the motor dynamic speed equation and the motor torque equation for the time series of the corresponding action signal, and obtain the first travel or the first rotation angle of the driving device based on the current;

[0024] S04. Input the ocean panoramic images corresponding to the time series of the first action signal and the second action signal into a neural network model for training to obtain an environmental intervention fault recognition model for identifying the probability of environmental intervention faults around the robot;

[0025] S05. Perform feature analysis including travel and angle on the image information of the corresponding driving device intercepted corresponding to the time series of the first action signal and the second action signal, and judge the current fault type of the robot according to the recognition result of the environmental intervention fault recognition model, the first travel or the first rotation angle of the driving device based on the current, and the feature analysis results of the travel and angle of the image information of the corresponding driving device.

[0026] According to another preferred embodiment of the present invention, multiple marine environment images with edge overlapping regions around the robot are respectively collected under the time series of the first motion signal and the second motion signal. The ORB algorithm is used to extract features from multiple marine environment images with edge overlapping regions to obtain key points of the marine environment images, and the extracted features are subjected to key point matching to construct BRIEF descriptors invariant to the rotation of the corresponding key points. The similarity values of the BRIEF descriptors invariant to the rotation of the key points of different marine environment images are calculated. If the similarity value is greater than a preset threshold, the panoramic marine environment image is obtained by feature stitching and fusion based on the similar key points of the different marine environment images.

[0027] According to another preferred embodiment of the present invention, the key points of the corresponding panoramic marine environment image are identified and labeled according to the ORB algorithm, and the depth information of each key point is obtained. The pixel features of the key points and the depth information of each key point are subjected to feature transformation processing to obtain standardized panoramic image features. The standardized panoramic image features are input into a trained environmental intervention fault recognition model to identify the environmental intervention fault probability of each key point. If there is at least one key point with an environmental intervention fault probability greater than a preset probability threshold, the position coordinates of the corresponding key point in the panoramic image are output, and the type of the corresponding environmental intervention fault probability is output.

[0028] According to another preferred embodiment of the present invention, a first motion signal and a second motion signal are generated, and the time series t1 of the first motion signal and the time series t2 of the second motion signal are obtained. Among them, the instantaneous currents I t1 and I t2 of the corresponding drive devices are respectively obtained according to the time series t1 and the time series t2. According to the motor torque equation H = K H *I t and the motor dynamic speed equation The first stroke I1 or the first rotation angle θ1 experienced by the corresponding drive device from the time series t1 to the time series t2 is calculated according to the following formula: , where D ∈ [t1, t2], where K H is the torque constant, * is the product calculation, I t is the instantaneous current, D is the integration domain, J is the moment of inertia, and λ is the stroke transmission coefficient or the angle transmission coefficient.

[0029] According to another preferred embodiment of the present invention, after the image information of the corresponding drive device intercepted in the time series t1 and the time series t2, the target detection model is used to detect the stroke I2 or the rotation angle θ2 of the corresponding drive device on the image interface from the time series t1 to the time series t2, and the absolute value of the stroke difference and the absolute value of the rotation angle difference , and based on the absolute value of the travel difference , the absolute value of the rotation angle difference and the comprehensive analysis of the environmental intervention failure probability of the key points of the corresponding drive device to determine the current failure type.

[0030] The present invention provides a computer-readable storage medium storing a computer program, and the computer program is executed by a processor to implement the above-mentioned method for autonomous fault detection of a deep-sea heavy-duty operation robot based on image recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 It shows a schematic diagram of the modules of a deep-sea heavy-duty operation robot for autonomous fault detection based on image recognition according to the present invention;

[0032] Figure 2 It shows a schematic flowchart of a method for autonomous fault detection of a deep-sea heavy-duty operation robot based on image recognition according to the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0033] The following description is used to disclose the present invention so that those skilled in the art can implement the present invention. The preferred embodiments described below are only examples, and those skilled in the art can think of other obvious variations. The basic principles defined in the following description can be applied to other embodiments, variations, improvements, equivalent embodiments, and other technical solutions without departing from the spirit and scope of the present invention.

[0034] It can be understood that the term "one" should be understood as "at least one" or "one or more". That is, in one embodiment, the number of an element can be one, and in other embodiments, the number of the element can be multiple. The term "one" cannot be understood as a limitation on the number.

[0035] Please refer to Figure 1 - Figure 2, the present invention provides a deep-sea heavy-duty operation robot based on image recognition for autonomous fault detection, and also provides an autonomous fault detection method for the robot. The robot includes the following parts: an image acquisition and processing module; a device driving module; a calculation module; a fault identification module. The image acquisition and processing module includes processors such as a camera, a CPU, and a GPU, which are used to collect, identify, and process image information. The image acquisition and processing module has a built-in memory and is configured with a trained neural network model to identify the probability of environmental intervention faults in the marine environment image around the robot. The device driving module includes, but is not limited to, an MCU chip, a MOSFET (metal-oxide-semiconductor field-effect transistor), a DSP (digital signal processing chip), etc. The MCU chip is used to generate action signals for driving the device, and the MOSFET generates corresponding motor drive currents according to the action signals. The device driving module also includes a current sensor for detecting the instantaneous current of the motor corresponding to the driving device. The DSP is used to process digital signals including, but not limited to, images and action signals. The DSP is configured with a working clock and can intercept the clock information of the corresponding action signal as the time series of the corresponding action signal. The image acquisition and processing module is also configured with a model including target recognition and feature analysis to identify the travel and rotation angle features of the corresponding driving device in the panoramic image of the marine environment around the robot at different time series. The target recognition and feature analysis model can be a neural network model or a model based on the target pixel distance feature and angle feature algorithm. The present invention obtains the travel and rotation angle of the driving device based on current through integral mapping and solution according to the corresponding time series based on the instantaneous current of the driving device and the relevant motor dynamic speed equation and motor torque equation. The present invention compares the above two different travel and rotation angles and combines the probability of the corresponding driving environment intervention fault to determine the current robot fault type.

[0036] Specifically, since deep-sea operation robots are vulnerable to the influence of marine organisms such as kelp and coral, as well as seabed reefs and suspended garbage, etc., including but not limited to, the actions of deep-sea heavy-duty operation robots will be affected by the above-mentioned intervening factors in the marine environment, resulting in malfunctions that are not caused by the robots themselves. Therefore, the present invention needs to utilize an existing mature image recognition model to determine whether there are any environmental intervening factors in the current marine environment image around the robot that may affect the failure of the current robot's driving device. It should be noted that from a practical perspective, due to the influence of the fluidity of suspended substances and water bodies in the marine environment, the images of marine organisms such as kelp and coral, as well as seabed reefs and suspended garbage taken above may not necessarily affect the actions of the current robot. Therefore, the present invention needs to perform feature analysis including travel and rotation angle on the actions of the driving device at different time series by means of an image recognition model; and further comprehensively compare the travel and rotation angle of the driving device obtained by integrating the current characteristics of the corresponding driving device motor current at the same time series to determine whether the images of marine organisms such as kelp and coral, as well as seabed reefs and suspended garbage taken above actually affect the actual actions of the robot, so that the present invention can more efficiently and accurately determine the failure type of the deep-sea operation robot.

[0037] The image acquisition and processing module is configured to include multiple cameras for acquiring marine environment images at different positions around the robot, and further performing feature fusion and stitching on the marine environment images at different positions to obtain the panoramic image of the marine environment around the robot. The method for generating the panoramic image of the marine environment around the robot includes: acquiring marine environment images around the robot captured by multiple cameras in the same time series, where there are overlapping images between multiple adjacent marine environment images around the robot. Further, the ORB algorithm (Oriented FAST and Rotated BRIEF) is used for key point detection. The key point detection method includes: selecting a central pixel point p from the marine environment image around the robot, taking the central pixel point p as the center of a circle, and making a detection circle neighborhood with a radius r of 3 pixel distances. If there are N consecutive pixels (N can be 8 or 9) in the detection circle neighborhood whose pixel intensity is significantly higher than that of the central pixel point p, then the current pixel point is defined as a key point (corner point). The pixel intensity being significantly higher than the central pixel point p can be screened by setting a certain threshold, which is not elaborated in detail in the present invention. The key points can be screened by non-maximum suppression to select representative key points, and the directions of the key points are determined, where the directions of the key points can be given rotation-invariant directions by using the intensity centroid method. Further, the present invention generates a rotation-invariant BRIEF descriptor for the key points according to the ORB algorithm. The rotation-invariant BRIEF descriptor for the key points is a binary descriptor obtained by comparing the pixel intensities in the neighborhood of the key points. When the intensity of the current pixel point is higher than that of the subsequent pixel point, the binary value of the current pixel point in the neighborhood of the key point is 1, otherwise it is 0. Therefore, the binary features of the rotation-invariant BRIEF descriptor for the key points have the characteristics of high efficiency, light weight and real-time performance, and are particularly suitable for the recognition of the features of the marine environment images around the robot in the deep-sea environment.

[0038] Since there are overlapping parts in the multiple images of the ocean environment around the robots, there are key points with a very high similarity and overlap in the ocean environments around different robots. The present invention performs feature stitching and fusion of the images of the ocean environment around different robots based on the overlapping key points. After obtaining the key points of different images of the surrounding ocean environment, the present invention calculates the Hamming distance between the key points of different images of the surrounding ocean environment of the robot and sets a Hamming distance threshold. If the calculated Hamming distance is less than the preset Hamming distance threshold, it indicates that the key points of the current different images of the ocean environment around the robot are overlapping key points for feature stitching and fusion. The feature stitching and fusion can adopt methods including but not limited to weighted average fusion of pixel points of overlapping key points and Poisson Blending of other non-key point pixel points, etc., so as to have obvious features at the key points and retain certain texture features at the non-key points, adapting to complex lighting environments. Thus, the panoramic image of the ocean environment around the robot is obtained. In one preferred embodiment of the present invention, it is necessary to establish a coordinate system for the panoramic image of the ocean environment around the robot. The coordinate system can be a planar coordinate system or a spherical coordinate system, and the key points are marked with, including but not limited to, position marking and driving device marking.

[0039] It should be noted that in the present invention, the key points obtained based on the ORB algorithm are generally the edge features at the intersection of the robot body, the mechanical structure of the relevant driving device, and the ocean environment. Therefore, the corresponding driving device can be relatively accurately marked based on the key points. The present invention further performs standardized feature conversion on the panoramic image of the ocean environment around the robot and the corresponding key points and inputs them as samples into a neural network model for training. The neural network model is used for training the probability of environmental intervention faults corresponding to the key points, and an environmental intervention fault recognition model is obtained. It should be noted that the present invention also obtains the depth information of the corresponding key points of the panoramic image of the ocean environment around the robot as an input sample parameter of the environmental intervention fault recognition model, so as to improve the training effect of the neural network model. The present invention can adopt network models including but not limited to FNN (feedforward neural network model), CNN (convolutional neural network model), DNN (deep neural network model), etc. as training models for training. The training methods of the above network models are prior arts, and the present invention does not improve the training methods, so the present invention will not elaborate on this.

[0040] When the trained environmental intervention fault recognition model processes the panoramic image of the marine environment around the robot and the corresponding key-point features input through the corresponding time series, it can output the probability of environmental intervention faults at the key points on the corresponding starting device. For example, through the environmental intervention fault recognition model, it can be recognized that the probability of an environmental intervention fault at the key point of the robot drive device with kelp entanglement is 80%. Therefore, this 80% probability of environmental intervention fault is used as reference data for the true fault type of the robot. In one preferred embodiment of the present invention, an environmental intervention fault probability threshold can be set to screen out key points with a relatively high probability of environmental intervention faults, where the key points correspond to the types of drive devices.

[0041] It is worth mentioning that the present invention analyzes the stroke and rotation angle of the target drive device using an image recognition model, and compares them with the stroke and rotation angle of the target drive device obtained by integrating the current of the drive device in the time domain. Combining the probability of environmental intervention faults, it can accurately analyze whether there are mechanical faults in the action mechanism or motor drive device faults. Specifically, the device drive module generates a first action signal and a second action signal and obtains corresponding time series t1 and time series t2 according to the working clock. Further, according to the time series t1 and time series t2, the instantaneous motor currents I t1 and I t2 of the corresponding drive device are obtained, where the time interval for each detection of the instantaneous motor current is further obtained according to the frequency of the working clock , and integral processing in the time domain is performed according to the time interval of the instantaneous motor current. Specifically, in the present invention, after obtaining the instantaneous motor currents I t1 and I t2 , it is necessary to solve the stroke of the drive device motor according to the motor torque equation H = K H *I t and the motor dynamic speed equation . Among them, in the motor torque equation H = K H *I t , K H is the torque constant, * is the product calculation, I t is the instantaneous current, and the drive device motor dynamic speed equation performs discrete integral processing in the time domain according to the time interval obtained according to the frequency of the motor working clock. In one preferred embodiment of the present invention, the discrete integral can be approximated by summing the discrete products of the time interval and the corresponding instantaneous current I t . For example, in the time domain from t1 to t2, there are m time intervals And the instantaneous current corresponding to each time interval is I tm , then the integral value of the sum in the current time domain can be approximately expressed as . It should be noted that the above motor dynamic speed equation After further performing integral processing on the time domain of the motor dynamic speed , the total rotation angle of the corresponding motor can be obtained.

[0042] It is worth mentioning that since there is a certain linear relationship between the motor rotation angle or number of turns and the stroke of the corresponding motor action mechanism, the more the number of turns the motor rotates, the greater the corresponding motor stroke. When the action mechanism corresponding to the motor is a rotating mechanism, the more the motor rotation angle or number of turns, the greater the corresponding angular stroke. Since the type of action mechanism connected to the motor involves the corresponding mechanical structure, the specific numerical relationship is also related to the model of the motor itself. Therefore, according to the above motor torque equation H = K H *I t and the motor dynamic speed equation , equation substitution is performed. Substitute the motor torque equation H = K H *I t into the motor dynamic speed equation to obtain the following motor dynamic speed equation: , further perform integral processing on the time domain of the motor dynamic speed equation to obtain the following equation . Since there is a linear relationship between the motor rotation angle or number of turns and the corresponding stroke, the present invention sets the stroke transmission coefficient or the angle transmission coefficient λ of the driving device, and further substitutes it into the above formula to obtain the following first stroke I1 or first rotation angle θ1 based on the integral processing of the motor instantaneous current: , where D ∈ [t1, t2], D is the integration domain, and J is the moment of inertia. It is worth mentioning that according to Newton's second law, the rotational relationship between the above torque H and the action mechanism of the corresponding driving device is , where B represents the viscous coefficient, ω represents the motor speed, and H L represents the external load torque. In the present invention, the viscous coefficient includes the sum of the mechanical friction coefficient and the seawater resistance coefficient, and the above coefficients can be detected in advance, and the present invention will not elaborate on this. In addition, since there is generally no external load when the driving device in the present invention performs fault detection, therefore, the external load torque H L in the present invention can be set to 0 or a small constant value. In the present invention, the rotational relationship between the above torque H and the action mechanism of the corresponding driving device can be substituted into the formula is used as a constraint equation. Thus, the first stroke I1 or the first rotation angle θ1 obtained based on the current integration is the first stroke I1 or the first rotation angle θ1 considering the water damping and the no-load state of the driving device.

[0043] Furthermore, the present invention needs to rely on the actual movement conditions of the driving devices corresponding to the key points in the panoramic ocean image around the actual robot. Therefore, after intercepting the image information of the corresponding driving devices in the time series t1 and the time series t2, the present invention uses an object detection model to detect the stroke I2 or the rotation angle θ2 of the corresponding driving devices on the image interface from the time series t1 to the time series t2. The object detection model may include but is not limited to the YOLO model. Using the YOLO model, the corresponding driving device types and the stroke or rotation angle between the above time series t1 and time series t2 can be identified. The YOLO model is an example of the prior art, and how the present invention trains the YOLO model will not be elaborated here.

[0044] Furthermore, the present invention needs to calculate the absolute value of the stroke difference and the absolute value of the rotation angle difference , and based on the absolute value of the stroke difference , the absolute value of the rotation angle difference and the comprehensive analysis of the key point environmental intervention failure probability of the corresponding driving device, the current failure type is judged. Since the first stroke I1 and the first rotation angle θ1 experienced by the driving device obtained based on the current integration in the present invention are the first stroke I1 and the first rotation angle θ1 considering the water damping and the no-load state, when the absolute value of the stroke difference or the absolute value of the rotation angle difference is small, it can be considered that the current driving device itself executes the action smoothly, and thus the environmental intervention failure can be excluded. When the absolute value of the stroke difference or the absolute value of the rotation angle difference is greater than the preset threshold, and combined with the key point environmental intervention failure probability, when the corresponding key point environmental intervention probability is also greater than the preset threshold, it is determined that the current driving device failure is an environmental intervention failure. When the absolute value of the stroke difference or the absolute value of the rotation angle difference is greater than the preset threshold, and combined with the key point environmental intervention failure probability, when the corresponding key point environmental intervention probability is less than the preset threshold, it is determined that the current driving device failure is a mechanical structure failure. When the relevant driving signal of the driving device or the detected motor current is abnormal, it can be directly determined as a driving device system failure.

[0045] In another preferred embodiment of the present invention, in addition to the above-mentioned detection of environmental intervention faults, mechanical mechanism faults, and drive device system faults. The present invention can also obtain two integral results based on current obtained from any two time series based on image features, and calculate the maximum value, average value, and sum value of the experienced stroke or rotation difference as the judgment reference data for the drive device fault, and can detect faults including but not limited to motor jitter, motor overtravel, and delay.

[0046] In the embodiments disclosed by the present invention, the processes described above with reference to the flowcharts can be implemented as computer software programs. The embodiments disclosed by the present invention include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program contains program codes for executing the methods shown in the flowcharts. In such an embodiment, the computer program can be downloaded and installed from the network through the communication part, and / or installed from a removable medium. When the computer program is executed by a central processing unit (CPU), it executes the above-mentioned functions not limited in the method of the present application. It should be noted that the above-mentioned computer-readable medium in the present application can be a computer-readable signal medium, a computer-readable storage medium, or any combination of the two. The computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or component, or any combination of the above. More specific examples of the computer-readable storage medium can include, but are not limited to: an electrical connection with one or more wire segments, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present application, the computer-readable storage medium can be any tangible medium that contains or stores a program, and the program can be used by or combined with an instruction execution system, device, or component. In the present application, the computer-readable signal medium can include a data signal propagated in a baseband or as part of a carrier wave, which carries the computer-readable program code. Such a propagated data signal can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. The computer-readable signal medium can also be any computer-readable medium other than the computer-readable storage medium, and the computer-readable medium can send, propagate, or transmit a program for use by or combined with an instruction execution system, device, or component. The program code contained on the computer-readable medium can be transmitted by any suitable medium, including but not limited to: wireless, wire segments, optical cables, RF, etc., or any suitable combination of the above.

[0047] The flowcharts and block diagrams in the accompanying drawings illustrate the possible architectures, functions, and operations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in the flowchart or block diagram may represent a module, a segment of a program, or a portion of code that contains one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions noted in the blocks may occur in a different order than that noted in the accompanying drawings. For example, two consecutive blocks shown may actually be executed substantially in parallel, and they may sometimes be executed in the reverse order, depending on the functions involved. It should also be noted that each block in the block diagram and / or flowchart, and combinations of blocks in the block diagram and / or flowchart, can be implemented by a dedicated hardware-based system that performs the specified functions or operations, or can be implemented by a combination of dedicated hardware and computer instructions.

[0048] Those skilled in the art should understand that the embodiments of the present invention described above and shown in the accompanying drawings are only examples and do not limit the present invention. The objectives of the present invention have been fully and effectively achieved. The functions and structural principles of the present invention have been demonstrated and explained in the embodiments. Without departing from the said principles, the embodiments of the present invention may have any variations or modifications.

Claims

1. A deep-sea heavy-duty operation robot based on autonomous fault detection by image recognition, characterized in that, The robot includes: An image acquisition and processing module; A device driving module; A calculation module; A fault identification module; Wherein the image acquisition and processing module acquires an image of the marine environment around the robot, and performs image processing on the image of the marine environment around the robot to obtain a panoramic image of the marine environment around the robot; The device driving module is used to generate a first action signal and a second action signal for corresponding driving devices. The image acquisition and processing module intercepts image information of the corresponding driving devices according to the time series of the first action signal and the second action signal. The image acquisition and processing module inputs the features after processing the marine panoramic image into a neural network model for training to obtain an environmental intervention fault identification model for identifying the probability of environmental intervention faults; The calculation module acquires the current data of the first action signal and the second action signal of the device driving module, and performs time integral mapping processing on the current of the corresponding driving device according to the motor dynamic speed equation and the motor torque equation; performs feature analysis including stroke and angle on the intercepted image information of the corresponding driving device according to the same time series; The fault identification module determines the type of fault existing in the current robot according to the identification result of the environmental intervention fault identification model, the time integral mapping processing result of the motor dynamic speed equation and the motor torque equation, and the feature analysis result of the stroke and angle of the image information of the corresponding driving device detected by using the target detection model; The image acquisition and processing module respectively acquires multiple marine environment images with edge overlapping regions around the robot under the time series of the first action signal and the second action signal, uses the ORB algorithm to extract features from the multiple marine environment images with edge overlapping regions respectively to obtain key points of the marine environment images, and performs key point matching on the extracted features to construct BRIEF descriptors invariant to the rotation of the corresponding key points, calculates the similarity values of the BRIEF descriptors invariant to the rotation of the key points of different marine environment images. If the similarity value is greater than a preset threshold, feature stitching and fusion are performed based on the similar key points of the different marine environment images to obtain the marine environment panoramic image; The image acquisition and processing module identifies and labels the key points of the corresponding marine environment panoramic image according to the ORB algorithm, and acquires the depth information of each key point, performs feature conversion processing on the pixel features of the key points and the depth information of each key point to obtain standardized panoramic image features, and inputs the standardized panoramic image features into the trained environmental intervention fault identification model to identify the probability of environmental intervention faults for each key point. If the probability of environmental intervention faults of at least one key point is greater than a preset probability threshold, the position coordinates of the corresponding key point in the panoramic image are output, and the type of the corresponding environmental intervention fault probability is output; The device driver module generates a first action signal and a second action signal, and the computing module respectively obtains the time series t1 of the first action signal and the time series t2 of the second action signal, wherein the motor instantaneous current I including the corresponding drive device is respectively obtained according to the time series t1 and the time series t2 t1 and I t2 , and the computing module calculates the first stroke l1 or the first rotation angle θ1 experienced by the corresponding drive device from the time series t1 to the time series t2 according to the motor torque equation H = K H *I t and the motor dynamic speed equation according to the following formula: , where D ∈ [t1, t2], where K H is the torque constant, * is the product calculation, I t is the instantaneous current, D is the integration domain, J is the moment of inertia, and λ is the stroke transmission coefficient or the angle transmission coefficient.

2. The deep-sea heavy-duty operation robot for autonomous fault detection based on image recognition according to claim 1, wherein After intercepting the image information of the corresponding drive device in the time series t1 and the time series t2, the image acquisition and processing module uses a target detection model to detect the travel l2 or the rotation angle θ2 of the corresponding drive device on the image interface from the time series t1 to the time series t2, and calculates the absolute value of the travel difference. and the absolute value of the rotation angle difference , and based on the absolute value of the travel difference , the absolute value of the rotation angle difference and the key point environment intervention failure probability of the corresponding drive device are comprehensively analyzed to determine the current failure type; when the absolute value of the travel difference or the absolute value of the rotation angle difference is greater than the preset threshold, and combined with the key point environment intervention failure probability, when the corresponding key point environment intervention probability is also greater than the preset threshold, it is determined that the current drive device failure is an environment intervention failure; when the absolute value of the travel difference or the absolute value of the rotation angle difference is greater than the preset threshold, and combined with the key point environment intervention failure probability, when the corresponding key point environment intervention probability is less than the preset threshold, it is determined that the current drive device failure is a mechanical structure failure; when the relevant drive signal of the drive device or the detected motor current is abnormal, it is directly determined as a drive device system failure.

3. An autonomous fault detection method for a deep - sea heavy - duty operation robot based on image recognition, characterized in that, The method includes: S01. Acquire an image of the marine environment around the robot, and perform image processing on the image of the marine environment around to obtain a panoramic image of the marine environment around the robot; S02. Generate the first action signal and the second action signal, obtain the time series of the first action signal and the second action signal, and respectively intercept the image information and instantaneous current of the driving device according to the time series of the first action signal and the time series of the second action signal; S03. Perform time integral mapping processing on the instantaneous current of the driving device according to the motor dynamic speed equation and the motor torque equation for the corresponding action signal time series to obtain the first stroke or the first rotation angle of the driving device based on the current; S04. Input the ocean panoramic images corresponding to the time series of the first action signal and the second action signal into a neural network model for training to obtain an environmental intervention fault recognition model for identifying the probability of environmental intervention faults around the robot; S05. Perform feature analysis including stroke and angle on the image information of the corresponding driving device intercepted by the time series of the first action signal and the second action signal, and judge the type of fault existing in the current robot according to the recognition result of the environmental intervention fault recognition model, the first stroke or the first rotation angle of the driving device based on the current, and the feature analysis results of the stroke and angle of the image information of the corresponding driving device detected by using the target detection model; The method for generating the panoramic image of the ocean environment around the robot includes: respectively collecting multiple ocean environment images with edge overlapping regions around the robot under the time series of the first action signal and the second action signal, using the ORB algorithm to extract features from multiple ocean environment images with edge overlapping regions to obtain the key points of the ocean environment images, performing key point matching on the extracted features, constructing BRIEF descriptors invariant to key point rotation, calculating the similarity values of the BRIEF descriptors invariant to key point rotation of different ocean environment images, and if the similarity value is greater than a preset threshold, performing feature stitching and fusion based on the similar key points of the different ocean environment images to obtain the panoramic image of the ocean environment around the robot; Identify and label the key points of the corresponding ocean environment panoramic image according to the ORB algorithm, obtain the depth information of each key point, perform feature conversion processing on the key point pixel features and the depth information of each key point to obtain panoramic image features, input the panoramic image features into the trained environmental intervention fault recognition model to identify the probability of environmental intervention faults for each key point, and if the probability of environmental intervention faults for at least one key point is greater than the preset probability threshold, output the position coordinates of the corresponding key point in the panoramic image and output the type of the corresponding environmental intervention fault probability; Generate a first motion signal and a second motion signal, and obtain the time series t1 of the first motion signal and the time series t2 of the second motion signal, wherein the motor instantaneous currents I of the corresponding drive devices are respectively obtained according to the time series t1 and the time series t2 t1 and I t2 , according to the motor torque equation H = K H *I t and the motor dynamic speed equation Calculate the first stroke l1 or the first rotation angle θ1 experienced by the corresponding drive device from the time series t1 to the time series t2 according to the following formula: , where D ∈ [t1, t2], where K H is the torque constant, * is the product calculation, I t is the instantaneous current, D is the integration domain, J is the moment of inertia, and λ is the stroke transmission coefficient or the angle transmission coefficient.

4. The autonomous fault detection method for a deep-sea heavy-duty operation robot based on image recognition according to claim 3, wherein After intercepting the image information of the corresponding drive device at the time series t1 and the time series t2, use the target detection model to detect the stroke l2 or the rotation angle θ2 of the corresponding drive device on the image interface from the time series t1 to the time series t2, and calculate the absolute value of the stroke difference and the absolute value of the rotation angle difference , and based on the absolute value of the stroke difference , the absolute value of the rotation angle difference and the comprehensive analysis of the key point environment intervention failure probability of the corresponding drive device to judge the current failure type; when the absolute value of the stroke difference or the absolute value of the rotation angle difference is greater than the preset threshold, and combined with the key point environment intervention failure probability, when the corresponding key point environment intervention probability is also greater than the preset threshold, it is determined that the current drive device failure is an environment intervention failure; when the absolute value of the stroke difference or the absolute value of the rotation angle difference is greater than the preset threshold, and combined with the key point environment intervention failure probability, when the corresponding key point environment intervention probability is less than the preset threshold, it is determined that the current drive device failure is a mechanical structure failure; when the relevant drive signal of the drive device or the detected motor current is abnormal, it is directly determined as a drive device system failure.

5. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, and the computer program is executed by a processor to implement a method for autonomous fault detection of a deep-sea heavy-duty operation robot based on image recognition according to any one of claims 3-4.

Images