A dynamic video monitoring method for shipborne retractable equipment based on machine vision
Through a dynamic video surveillance method based on machine vision, the ship's on-board collection and playback equipment is monitored in real time using lidar and cameras, solving the problem of manual judgment dependence and improving the intelligence level and emergency response capabilities of the equipment.
Patent Information
- Application Number
- CN202111599690.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-12-24
- Publication Date
- 2024-11-26
- Estimated Expiration
- 2041-12-24
AI Technical Summary
The existing ship-mounted storage equipment relies on manual judgment in terms of status monitoring, which leads to heavy burden on operators and insufficient intelligence of the system, making it impossible to effectively monitor the movement status of key components of the equipment.
Using a dynamic video surveillance method based on machine vision, data is collected through lidar and cameras, and combined with supporting software to identify and analyze, real-time monitoring of the collection and playback equipment and fault alarms are realized, and the intelligence level of the equipment is improved.
Real-time monitoring and automatic identification of ship-mounted and retrieval equipment is realized, reducing the pressure on operators, and significantly improving the intelligence level and emergency response capabilities of the equipment.
Smart Images

Figure CN114449217B_ABST
Abstract
Description
Technical Field
[0001] The invention belongs to the field of image processing and moving target recognition, and in particular relates to a dynamic video monitoring method of shipborne retractable equipment based on machine vision. Background Art
[0002] At present, shipborne equipment is widely used in marine resource exploration, such as imaging sonar. Among them, the retractable equipment is responsible for the deployment and recovery of the detection device, and is an important part of the shipborne detection system. At present, most of the shipborne retractable equipment has realized automatic control, but in terms of status monitoring, it relies more on manual judgment, system built-in sensors, video monitoring, etc. The manual judgment method not only has high requirements for personnel, but also invisibly increases the workload of operators. The system built-in sensors are mostly used for displacement, speed, pressure, flow and other parameter monitoring, and cannot monitor the motion status of key components of the equipment (especially some flexible bodies). In terms of video monitoring, the current application is to provide video images for operators, and judgment and decision-making still rely on operators, resulting in high requirements for users and insufficient equipment intelligence. Summary of the invention
[0003] In order to solve the above problems, the present invention proposes a dynamic video monitoring method for shipborne retractable equipment based on machine vision. By identifying and analyzing the collected radar point cloud data, video picture data, etc., the dynamic monitoring, fault alarm and other functions of the shipborne retractable equipment are realized, thereby improving the reliability and intelligence level of the equipment.
[0004] The present invention provides the following technical solutions:
[0005] A method for dynamic video monitoring of shipborne retractable equipment based on machine vision, wherein the retractable equipment comprises a laser radar, a monitoring kit, a network switch, a host and supporting software, wherein the host communicates with a control system arranged inside the retractable equipment in real time, shares image recognition results and provides the control system of the retractable equipment with the results; the monitoring kit comprises a camera and an NVR recorder, and the supporting software is used to process and analyze the point cloud data provided by the laser radar and the image data provided by the camera, and generate the detection results required by the user, and provide a human-computer interaction interface; the overall architecture of the supporting software is divided into four levels, which are divided from bottom to top into: physical sensor, software function drive, software function realization, and human-computer interaction, and the laser radar, monitoring kit and host are respectively connected to the network switch through a network cable;
[0006] The method includes a method for detecting a person entering by mistake, a method for detecting an abnormal appearance of a transmission chain, a method for detecting a chain state at a sprocket, and a method for detecting an abnormal state of a key component;
[0007] The method for detecting the mistaken entry of personnel is to arrange a camera in the dangerous area of the equipment, use the image data of the camera as input, and realize the mistaken entry detection of personnel by presetting the ROI area;
[0008] The transmission chain appearance anomaly detection method uses the point cloud data of the laser radar as input, and realizes chain appearance anomaly detection by extracting the contour and width of the preset ROI area;
[0009] The chain state detection method at the sprocket uses the point cloud data of the laser radar as input, and can monitor the meshing state of the chain and the sprocket in real time;
[0010] The method for detecting abnormal state of key components is to detect through a preset ROI area when a component is detected to arrive at a specified position, so as to determine whether the current state of the component is a normal state.
[0011] In general, the above technical solutions conceived by the present invention can achieve the following beneficial effects compared with the prior art:
[0012] 1. The present invention can monitor the shipborne retractable equipment in real time and automatically identify key components and personnel who have strayed into dangerous areas. The detection results can be used as input signals for the control system of the shipborne retractable equipment to improve the emergency response capability of the equipment.
[0013] 2. By building a test system in the laboratory, it was found through the experimental process that the present invention can significantly reduce the operator's pressure of use, reduce the number of people required when using the retractable equipment, and improve the intelligence level of the retractable equipment. BRIEF DESCRIPTION OF THE DRAWINGS
[0014] Figure 1 is a system structure diagram of the present invention;
[0015] Figure 2 It is a software architecture diagram of the present invention;
[0016] Figure 3 is a flow chart of chain appearance abnormality detection of the present invention;
[0017] Figure 4 It is a flow chart of the detection of the primitive flip board leakage operation of the present invention;
[0018] Figure 5 It is a flow chart of chain state detection at the sprocket of the present invention. DETAILED DESCRIPTION
[0019] The following is a clear and complete description of the technical solutions in the embodiments of the present application in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of them. Based on the embodiments, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0020] In this embodiment, ROI means "region of interest, which is divided into ROI (2D block) in the image coordinate system and ROI (3D block) in the laser coordinate system", image ROI means "the appearance of a preset color within the ROI range is a trigger", laser ROI means "the appearance of more than a preset number of reflection points within the ROI range is a trigger", and personnel ROI means "the appearance of a person within the ROI range is a trigger", and laser to image ROI means projecting the laser ROI onto the image plane.
[0021] The shipborne launching and retracting equipment described in this embodiment is a large-scale launching and retracting system, which uses a chain as a transmission carrier and a hydraulic motor as a driving device. It is responsible for placing the primitives underwater and recovering the primitives. The maximum load capacity of the launching and retracting equipment is 50 tons.
[0022] In this embodiment, the dynamic video monitoring method of shipborne retractable equipment based on machine vision mainly includes: image ROI detection method, laser ROI detection method, laser to image ROI detection method, and personnel ROI detection method.
[0023] The image ROI detection method monitors the ROI area in the image and sets the HSV color space threshold. If a preset color appears, it is triggered, otherwise it is silent.
[0024] The laser ROI detection method monitors the ROI area in the three-dimensional space output by the laser radar, sets the point cloud reflection intensity threshold and the trigger threshold number of points. If a reflection point appears to be greater than the threshold and the reflection point meets the reflection intensity constraint, it is triggered, otherwise it is silent.
[0025] The laser-to-image ROI detection method monitors the ROI area in the three-dimensional space output by the laser radar, sets the point cloud reflection intensity threshold and the trigger threshold number of points, and reduces the three-dimensional point cloud data to the image plane.
[0026] The personnel ROI detection method monitors the ROI area in the image and triggers if a person target appears, otherwise it is silent.
[0027] As attached Figure 1 As shown, the device for dynamic video monitoring of shipborne retractable and deployable equipment based on machine vision in this embodiment includes a laser radar, a monitoring kit, a network switch, a host and supporting software.
[0028] The laser radar adopts the domestically produced livox series non-repetitive scanning laser radar, which has a non-repetitive scanning function. It can perform dense three-dimensional reconstruction of the target area and obtain centimeter-level point cloud data. The communication interface is Ethernet, and the laser radar has a ranging capacity of 260m@80% reflectivity, a distance accuracy of 2cm, an angle accuracy of 0.1°, and a laser wavelength of 905nm. The laser radar is installed near a specific position of the chain and primitive running path. After the laser radar is set with a static IP address, it is connected to the network switch via a network cable.
[0029] The camera and NVR recorder adopt the domestic Hikvision NVR series monitoring kit, which can conveniently store and access data, play back data, and access camera data in real time. The communication interface is Ethernet, the camera main stream resolution is 50Hz, the maximum frame rate is 20fps, the sub-stream resolution is 50Hz, the maximum frame rate is 25fps, and the communication interface is adaptive Ethernet. The camera is connected to the NVR recorder, and the NVR recorder is connected to the switch via a network cable.
[0030] The network switch uses a domestically produced H3C 5-port Gigabit Ethernet switch, which is used to connect the host with the laser radar and camera.
[0031] The host adopts a commercial PC with a hardware environment of 32G memory, 2 RTC2080 graphics cards, and a software environment of Ubuntu18.04 operating system with CUDA10.1 installed. The host communicates with the control system of the retractable device in real time, shares image recognition results and provides them for use by the control system of the retractable device.
[0032] The supporting software architecture is as shown in the attached Figure 2 As shown, it is used to process and analyze the point cloud data provided by the lidar and the image data provided by the camera, generate the detection results required by the user, and provide a human-computer interaction interface.
[0033] The overall architecture of the supporting software is divided into four levels, from bottom to top: physical sensors, software function drivers, software function implementation, and human-computer interaction; the four levels are connected to the data of the levels above and below them respectively and cannot be accessed across levels.
[0034] The physical sensor layer is a data producer that ensures a stable and reliable data source, mainly including lidar data and image data. Lidar data can perceive three-dimensional space with centimeter-level accuracy, but does not have the function of collecting color information. It can only obtain the reflection intensity of the target object and can be used to distinguish objects with different reflection loudness. Image data has rich color information, but cannot detect the three-dimensional space distance of the target object. It can be used to detect targets with obvious colors.
[0035] The software function driver layer processes the data generated by the hardware sensor to generate custom ROI signals and local point clouds and image block data. This layer abstracts the sensor data to generate the data required by the upper system.
[0036] The software function implementation layer receives the required ROI signals and local point clouds and image block data, and performs secondary processing according to the different trigger logics of each module to generate detection result signals that meet the requirements of this system, including personnel misentry detection, chain appearance abnormality detection, and key component abnormal state detection.
[0037] The human-computer interaction layer receives the recognition results transmitted from the lower layer, performs graphic display rendering on different detection items, and has certain interactive functions.
[0038] A dynamic video monitoring method for shipborne launching and retracting equipment based on machine vision, the method comprising a method for detecting stray entry of personnel, a method for detecting abnormal appearance of a transmission chain, a method for detecting the state of a chain at a sprocket, and a method for detecting abnormal state of key components.
[0039] The method for detecting the mistaken entry of personnel adopts the personnel ROI detection method, arranges a camera in the dangerous area of the equipment, inputs image data through the camera, and monitors the ROI area in the image. If a human target appears, it is triggered, otherwise it is silent, thereby realizing the mistaken entry detection of personnel.
[0040] As attached Figure 3 As shown, the transmission chain appearance abnormality detection method adopts the laser-to-image ROI detection method. This method reduces the 3D point cloud data of the laser radar to the image plane by monitoring the ROI area in the three-dimensional space, forming the ROI area point cloud, ROI area binary image, and ROI area pseudo-RGB image. When the laser radar is deployed, a position facing the chain is selected to monitor the chain. When the chain is stationary, the collection is triggered by personnel or the control system, the system enters the integration state, maintains for 3-5 seconds, and then automatically jumps out of the integration state, and performs contour detection and width extraction on the preset area, and finally obtains the chain width. If there is no target in the preset area, it means that the appearance contour of the chain is abnormal.
[0041] As attached Figure 5 As shown, the chain state detection method at the sprocket adopts the laser image ROI detection method, which is a normal detection. When the program is deployed, the point cloud data at the sprocket is intercepted by the laser radar, the point cloud is adjusted to the front view, and the 3D point cloud is drawn on the 2D image for visualization, where the color is in pseudo RGB form and mapped according to the reflection intensity. In the deployment stage, the standard convex hull data of the sprocket needs to be collected, and the convex hull line segment will be monitored in real time during the operation stage to realize real-time monitoring of the meshing state of the chain and the sprocket. If the convex hull line segment is significantly different from the standard situation, an abnormality report will be made.
[0042] The key component abnormal state detection method is as follows Figure 4 As shown, it is specifically the detection of missed operation of primitive flap. The primitive is the underwater part of the detection device, and the wing is a spoiler installed on the primitive. When the primitive is released to a specified position, the operator manually flips the wing up (rotates a certain angle) before it can continue to be deployed. In order to detect the situation where the flap operation is not performed, the laser ROI detection method is adopted. This detection is based on the detection triggered by the primitive. When the appearance of the primitive is detected, the laser ROI of the flap area is detected. If there is a flap signal in the area, it is considered to be missed and an alarm message is generated. On the contrary, if there is a primitive signal but no flap signal during the detection process, the result of this detection is considered to be normal, and the standby state is entered after the primitive disappears.
[0043] The above description is only a preferred implementation manner of the present invention, but the protection scope of the present invention is not limited thereto, and any modification and replacement based on the technical solution and inventive concept provided by the present invention should be included in the protection scope of the present invention.
Claims
1. A dynamic video monitoring method for shipborne retractable equipment based on machine vision, Features: The retractable device includes a laser radar, a monitoring kit, a network switch, a host and supporting software. The host communicates with the control system set inside the retractable device in real time, shares image recognition results and provides them for use by the control system of the retractable device; the monitoring kit includes a camera and an NVR recorder, and the supporting software is used to process and analyze the point cloud data provided by the laser radar and the image data provided by the camera, and generate the detection results required by the user, and provide a human-computer interaction interface; the overall architecture of the supporting software is divided into four levels, from bottom to top: physical sensor, software function drive, software function implementation, and human-computer interaction. The laser radar, monitoring kit and host are respectively connected to the network switch through a network cable; The method includes a method for detecting a person entering by mistake, a method for detecting an abnormal appearance of a transmission chain, a method for detecting a chain state at a sprocket, and a method for detecting an abnormal state of a key component; The method for detecting the mistaken entry of personnel is to arrange a camera in the dangerous area of the equipment, use the image data of the camera as input, and realize the mistaken entry detection of personnel by presetting the ROI area; The transmission chain appearance anomaly detection method uses the point cloud data of the laser radar as input, and realizes chain appearance anomaly detection by extracting the contour and width of the preset ROI area; The chain state detection method at the sprocket uses the point cloud data of the laser radar as input, and can monitor the meshing state of the chain and the sprocket in real time; The method for detecting abnormal state of key components is to detect through a preset ROI area when a component is detected to arrive at a specified position, so as to determine whether the current state of the component is a normal state.
