Fault diagnosis system and method for autonomous berthing and leaving near-distance target and obstacle sensing system

Abstract

The invention discloses a fault diagnosis system and method for an autonomous berthing and departing short-distance target and obstacle sensing system, and the system comprises a ship-borne equipment fault diagnosis subsystem and a fault diagnosis operation and maintenance platform, and the ship-borne equipment fault diagnosis subsystem is in communication connection with the fault diagnosis operation and maintenance platform. The ship-borne equipment fault diagnosis subsystem is provided with a plurality of sensor modules, can sense the surrounding environment in all directions, and transmits the collected data to the ship-borne edge computer for real-time processing. According to the invention, through the fault diagnosis system, full-process automatic management from equipment fault monitoring, fault diagnosis, information transmission, personnel scheduling to fault maintenance can be realized, and the operation and maintenance efficiency and safety of shipborne equipment are greatly improved; the system not only can improve the accuracy of equipment fault diagnosis, but also can ensure the efficient management and data tracking of the whole maintenance process through the coordination and recording functions of the operation and maintenance platform, and provides powerful technical guarantee for the reliable operation of ship equipment.

Description

Technical Field

[0001] The present invention relates to a fault diagnosis system and method, and in particular to a fault diagnosis system and method for an autonomous berthing and unberthing close-range target and obstacle perception system. Background Art

[0002] In recent years, with the maturity of artificial intelligence and sensor technology, the intelligence level of ships has been greatly improved, and the autonomous berthing and unberthing technology of intelligent ships has also developed rapidly. In the process of autonomous berthing and unberthing of ships, the primary problem is perception, especially the perception of close-range targets and obstacles. The autonomous berthing and unberthing system uses sensors such as visible light cameras, lidar, and millimeter-wave radar to comprehensively realize the perception of the surrounding environment. Due to the complexity of the port environment, especially when operating at close range, the perception accuracy and reliability of the sensor are crucial.

[0003] In the existing autonomous berthing and unberthing solutions, the amount of visual information is large, and the camera is relatively cheap and mature in technology, so it is widely used; millimeter-wave radar is reliable and accurate in ranging, and is often used together with cameras to form a multi-sensor perception solution. During autonomous berthing and unberthing operations, the failure of any perception system may cause the ship to collide with obstacles, posing a safety hazard. Compared with millimeter-wave and other sensors, cameras are relatively less reliable and have typical functional failures: missing pixels lead to missing row and column information, pixel offset leads to blurred images, and missing target colors lead to image rendering errors. Its fault characterization is difficult to extract through conventional fault diagnosis methods, and diagnosis is difficult to automate in real time. Summary of the invention

[0004] Purpose of the invention: The purpose of the present invention is to propose a fault diagnosis system and method for an autonomous berthing and unberthing close-range target and obstacle perception system, which improves the operational safety of intelligent ships in complex port environments and reduces the risk of collision accidents.

[0005] Technical solution: The present invention includes a shipborne equipment fault diagnosis subsystem and a fault diagnosis operation and maintenance platform. The shipborne equipment fault diagnosis subsystem is communicatively connected to the fault diagnosis operation and maintenance platform. The shipborne equipment fault diagnosis subsystem is equipped with a variety of sensor modules, which can perceive the surrounding environment in all directions and transmit the collected data to the shipborne edge computer for real-time processing.

[0006] The shipborne equipment fault diagnosis subsystem comprises a communication module, a data acquisition module, a data processing module, a data storage module and a fault diagnosis module.

[0007] The fault diagnosis and operation and maintenance platform includes a network receiver, a data center, a monitoring center, a dispatching center and a message center.

[0008] The fault diagnosis module will identify potential problems when abnormal equipment operation occurs through the built-in diagnostic algorithm, and send alarm information to the fault diagnosis operation and maintenance platform.

[0009] After receiving the alarm information, the dispatch center of the fault diagnosis and maintenance platform matches maintenance personnel according to the type and location of the fault, records the entire maintenance process, and stores relevant information after the fault is resolved.

[0010] A fault diagnosis method for an autonomous berthing and unberthing close-range target and obstacle perception system comprises the following steps:

[0011] S1. For each shipborne perception module, extract the image data acquired by the camera;

[0012] S2. For each frame of the image, use a neural network to detect typical obstacles.

[0013] S3, after obtaining the obstacle type in each frame image, use the typical obstacle database to retrieve the typical size of the obstacle by obstacle type;

[0014] S4, performing spatiotemporal fusion of the above data with the data frame of the millimeter wave radar, including spatial fusion of the millimeter wave radar and the camera visual information and temporal fusion of the millimeter wave radar and the camera visual information;

[0015] S5, after data fusion, the closest obstacle is extracted from the fused data frame for comparison and judgment.

[0016] The spatial fusion method of the millimeter wave radar and camera visual information is as follows:

[0017] The two-dimensional information scanned by a single frame of millimeter-wave radar is converted from the radar coordinate system coordinates to the camera coordinate system coordinates, and the radar data (x r ,y r ) to image coordinates (u, v):

[0018]

[0019] in, is the transformation matrix, assuming that the corresponding correction coordinate point pair is and (u j , v j ):

[0020]

[0021] The temporal fusion method of the millimeter wave radar and camera visual information is as follows: for the radar data frame at time t0, assuming that the camera operating frequency is n fps, if If there is a camera data frame from time t to time t0, the radar data is fused with the camera data of that frame; otherwise, the radar data frame waits for the next camera data frame.

[0022] The comparison and judgment method is: calculating the Euclidean distance of the geometric centers of obstacles sensed by the two sensors, recorded as τ, and comparing it with the typical size w of the obstacle;

[0023] If the fusion data fails, the judgment system is judged to have serious failure and the user is warned;

[0024] If the fusion data is successful and τ≥0.3w, the camera of the perception module has pixel loss or serious pixel offset, and the system failure affects the safety of docking and undocking, and the user is warned;

[0025] If the fusion data is successful and 0.1w≤τ<0.3w, the camera of the perception module has slight pixel loss, pixel offset or target color loss, and the system failure may affect the safety of berthing and unberthing, and the user is prompted;

[0026] If the fusion data is successful and r≤0.1w, the perception module of the autonomous docking and undocking close-range target and obstacle perception system works normally, with no failure or a minor failure.

[0027] The shipborne sensing modules are deployed at different positions of the intelligent ship, and multiple sensing modules constitute an autonomous berthing and unberthing close-range target and obstacle sensing system.

[0028] Beneficial effects: The present invention can realize the automated management of the whole process from equipment fault monitoring, fault diagnosis, information transmission, personnel dispatch to fault repair through the fault diagnosis system, which greatly improves the operation and maintenance efficiency and safety of shipborne equipment; the system can not only improve the accuracy of equipment fault diagnosis, but also ensure the efficient management and data tracking of the entire maintenance process through the coordination and recording functions of the operation and maintenance platform, providing a strong technical guarantee for the reliable operation of ship equipment. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 is a flow chart of the fault diagnosis method of the present invention;

[0030] Figure 2 This is a block diagram of the fault diagnosis system of the present invention. DETAILED DESCRIPTION

[0031] The present invention will be further described below in conjunction with the accompanying drawings.

[0032] like Figure 2As shown, the fault diagnosis system of the autonomous berthing and unberthing close-range target and obstacle perception system of the present invention includes a shipboard equipment fault diagnosis subsystem and a fault diagnosis operation and maintenance platform. The system is connected via a wireless network, so that the shipboard equipment fault diagnosis subsystem and the fault diagnosis operation and maintenance platform can communicate in real time. The shipboard equipment fault diagnosis subsystem includes a communication module, a data acquisition module, a data processing module, a data storage module and a fault diagnosis module. These modules work together, wherein the data acquisition module collects various operating data of the shipboard equipment by connecting sensors, and transmits the collected data to the data processing module for processing and analysis. The data processing module compares the operating status of the equipment with the diagnostic standard through calculation and analysis; the fault diagnosis module determines whether the target has a fault and stores the results of the analysis process in the local database. Subsequently, the communication module transmits the fault information to the fault diagnosis operation and maintenance platform in real time through the wireless network, thereby realizing real-time monitoring and fault diagnosis of the equipment operating status.

[0033] The fault diagnosis and maintenance platform includes a network receiver, a data center, a monitoring center, and a dispatch center. First, the network receiver receives data information from the fault diagnosis subsystem and transmits it to the data center for synchronization and storage. After receiving the new data, the data center will compare the fault information with the existing data. If a fault risk or abnormal data is found, the monitoring center will update the equipment status in real time and mark the fault on the interface. At the same time, the platform will send an alarm message to the dispatch center, which will match the most suitable operation and maintenance personnel based on the fault type, severity, and equipment location information. The operation and maintenance personnel will receive the task message and handle the equipment failure to ensure the normal operation of the equipment. The above processing flow will be performed synchronously and stored in the local database.

[0034] The shipboard equipment fault diagnosis subsystem is equipped with a variety of sensor modules in terms of perception devices, such as visible light cameras, lidars, and millimeter wave radars. These sensors can perceive the surrounding environment in all directions, and the collected data is transmitted to the shipboard edge computer for real-time processing. When the equipment is operating abnormally, the fault diagnosis module will identify potential problems through the built-in diagnostic algorithm and immediately send the alarm information to the fault diagnosis operation and maintenance platform. After receiving the alarm information, the dispatch center of the operation and maintenance platform will match the appropriate maintenance personnel according to the type and location of the fault, record the entire maintenance process, and store relevant information after the fault is resolved. If the diagnostic results show that the equipment is operating normally, the system will generate a monitoring log and store it in the local database, and send the information to the operation and maintenance platform for recording and archiving.

[0035] The autonomous berthing and unberthing close-range target and obstacle perception system targeted by the present invention is composed of a plurality of perception modules deployed at different positions of the intelligent ship, and each module includes a single camera and a single millimeter-wave radar deployed closely as sensors.

[0036] like Figure 1 As shown, the fault diagnosis method of the autonomous docking and undocking close-range target and obstacle perception system of the present invention comprises the following steps:

[0037] S1, for each shipborne perception module, extract the image data obtained by the camera.

[0038] S2, for each frame of the image, uses the yolo neural network to detect typical obstacle targets, and outputs the rectangular bounding box of the typical obstacle and the corresponding typical obstacle type. The yolo neural network used is trained with historical experimental data, uses pre-processed images of uniform size as input, and uses manually annotated obstacle types and bounding boxes as labels.

[0039] S3, after obtaining the obstacle type in each frame image, use the typical obstacle database to retrieve the typical size of the obstacle by obstacle type. And for each obstacle bounding box, use the midpoint of the rectangular bounding box as the geometric center of the obstacle.

[0040] S4, the above data is spatially and temporally fused with the data frame of the millimeter-wave radar, including the spatial fusion of the millimeter-wave radar and the camera visual information, and the temporal fusion of the millimeter-wave radar and the camera visual information.

[0041] The spatial fusion method of millimeter wave radar and camera visual information is as follows:

[0042] The coordinates of the two-dimensional information scanned by a single-frame millimeter-wave radar are converted from the radar coordinate system coordinates to the camera coordinate system coordinates, so as to be integrated with the single-frame camera visual information.

[0043] The camera outputs images in the form of two-dimensional coordinate data, and the scanning plane of the millimeter-wave radar is two-dimensional, so the output data is also two-dimensional coordinates. Therefore, the conversion from millimeter-wave radar coordinates to camera image coordinates can be regarded as a transformation between two two-dimensional coordinate systems.

[0044] Use the following formula to convert radar data (x r ,y r ) to image coordinates (u, v):

[0045]

[0046] in, is the transformation matrix, which is obtained by prior calibration. The coefficients of the matrix are calibrated by at least three calibration points and confirmed by the least square method. Assume that the corresponding correction coordinate point pair is and (u j , v j ):

[0047]

[0048] The temporal fusion method of millimeter wave radar and camera visual information is as follows:

[0049] The typical operating frame rate of a shipborne camera is 30-60 frames per second, while that of a millimeter-wave radar is 15-20 frames per second. The data of a millimeter-wave radar is of lower frequency. The present invention uses the millimeter-wave radar data frame with a lower frequency as a reference to align the camera data with the millimeter-wave radar data. Specifically, for the radar data frame at time t0, assuming that the camera operating frequency is nfps, if If there is a camera data frame from time t to time t0, the radar data is fused with the camera data of that frame. Otherwise, the radar data frame waits for the next camera data frame.

[0050] S5, after data fusion, the closest obstacle is extracted from the fused data frame for comparison and judgment. At this time, the data frame contains information such as the geometric center of the obstacle identified by vision, the geometric center of the obstacle identified by millimeter wave radar, and the typical size of the obstacle.

[0051] The comparison method is: calculate the Euclidean distance between the geometric centers of obstacles perceived by the two sensors, denoted as τ, and compare it with the typical size w of the obstacle.

[0052] If the fusion data fails, the judgment system is judged to have serious failure and the user is warned;

[0053] If the fusion data is successful and τ≥0.3w, the camera of the perception module has pixel loss or serious pixel offset, and the system failure affects the safety of docking and undocking, and the user is warned;

[0054] If the fusion data is successful and 0.1w≤τ<0.3w, the camera of the perception module has slight pixel loss, pixel offset or target color loss, and the system failure may affect the safety of berthing and unberthing, and the user is prompted;

[0055] If the fusion data is successful and τ≤0.1w, the perception module of the autonomous docking and undocking close-range target and obstacle perception system works normally, with no failure or a minor failure.

Claims

1. A fault diagnosis system for an autonomous docking and undocking close-range target and obstacle perception system, characterized in that: include: A shipboard equipment fault diagnosis subsystem and a fault diagnosis operation and maintenance platform, wherein the shipboard equipment fault diagnosis subsystem is communicatively connected with the fault diagnosis operation and maintenance platform, and the shipboard equipment fault diagnosis subsystem is equipped with a variety of sensor modules, which can perceive the surrounding environment in all directions and transmit the collected data to the shipboard edge computer for real-time processing.

2. The fault diagnosis system for the autonomous berthing and near-range target and obstacle perception system according to claim 1, characterized in that: The shipborne equipment fault diagnosis subsystem comprises a communication module, a data acquisition module, a data processing module, a data storage module and a fault diagnosis module.

3. The fault diagnosis system for the autonomous berthing and near-range target and obstacle perception system according to claim 1, characterized in that: The fault diagnosis and operation and maintenance platform includes a network receiver, a data center, a monitoring center, a dispatching center and a message center.

4. The fault diagnosis system for the autonomous berthing and near-range target and obstacle perception system according to claim 2, characterized in that: The fault diagnosis module will identify potential problems when abnormal equipment operation occurs through the built-in diagnostic algorithm, and send alarm information to the fault diagnosis operation and maintenance platform.

5. The fault diagnosis system for the autonomous berthing and near-range target and obstacle perception system according to claim 3 is characterized in that: After receiving the alarm information, the dispatching center of the fault diagnosis and maintenance platform performs maintenance.

6. A fault diagnosis method for an autonomous docking and undocking close-range target and obstacle perception system, characterized in that: The following steps are involved: S1. For each shipborne perception module, extract the image data acquired by the camera; S2. For each frame of the image, use a neural network to detect typical obstacles. S3, after obtaining the obstacle type in each frame image, use the typical obstacle database to retrieve the typical size of the obstacle by obstacle type; S4, performing spatiotemporal fusion of the above data with the data frame of the millimeter wave radar, including spatial fusion of the millimeter wave radar and the camera visual information and temporal fusion of the millimeter wave radar and the camera visual information; S5, after data fusion, the closest obstacle is extracted from the fused data frame for comparison and judgment.

7. The fault diagnosis method for the autonomous berthing and unberthing close-range target and obstacle perception system according to claim 6, characterized in that: The spatial fusion method of the millimeter wave radar and camera visual information is as follows: The two-dimensional information scanned by a single frame of millimeter-wave radar is converted from the radar coordinate system coordinates to the camera coordinate system coordinates, and the radar data (x r ,y r ) to image coordinates (u, v): in, is the transformation matrix, assuming that the corresponding correction coordinate point pair is and (u j , v j ):

8. The fault diagnosis method for the autonomous berthing and unberthing close-range target and obstacle perception system according to claim 6, characterized in that: The temporal fusion method of the millimeter wave radar and camera visual information is as follows: for the radar data frame at time t0, assuming that the camera operating frequency is nfps, if If there is a camera data frame from time t to time t0, the radar data is fused with the camera data of that frame; otherwise, the radar data frame waits for the next camera data frame.

9. The fault diagnosis method for the autonomous berthing and unberthing close-range target and obstacle perception system according to claim 6, characterized in that: The comparison and judgment method is: calculating the Euclidean distance of the geometric centers of obstacles sensed by the two sensors, recorded as τ, and comparing it with the typical size w of the obstacle; If the fusion data fails, the judgment system is judged to have serious failure and the user is warned; If the fusion data is successful and τ≥0.3w, the camera of the perception module has missing pixels or serious pixel offset, and the system failure affects the safety of berthing and unberthing, and the user is warned; If the fusion data is successful and 0.1w≤τ<0.3w, the camera of the perception module has slight pixel loss, pixel offset or target color loss, and the system failure may affect the safety of berthing and unberthing, and the user is prompted; If the fusion data is successful and τ≤0.1w, the perception module of the autonomous docking and undocking close-range target and obstacle perception system works normally, with no failure or a minor failure.

10. The fault diagnosis method for the autonomous berthing and unberthing close-range target and obstacle perception system according to claim 6, characterized in that: The shipborne sensing modules are deployed at different positions of the intelligent ship, and multiple sensing modules constitute an autonomous berthing and unberthing close-range target and obstacle sensing system.

Images