Automatic inspection and identification system and method for sewage outfalls along shoreline

Abstract

The present invention relates to the technical field of environment monitoring, and specifically to an automatic inspection and identification system and method for sewage outfalls along a shoreline. The system comprises a carrier, a four-view camera, a side-scan sonar and an intelligent inspection integrated machine. An on-bridge water drain opening identification model, an overwater sewage outfall detection model, an underwater sewage outfall detection model, and a personal posture detection model are carried in the intelligent inspection integrated machine. The on-bridge water drain opening identification model analyzes a riverway environment collected by a front camera to identify an on-bridge water drain opening; the overwater sewage outfall detection mode identifies an overwater sewage outfall according to image information collected by two side cameras, i.e. a left side camera and a right side camera; the underwater sewage outfall detection model identifies an underwater sewage outfall according to underwater scene information collected by the side-scan sonar; the personal posture detection model determines whether an operator is in a safe operation state according to image information collected by a rear camera. The present invention can improve accuracy and efficiency in sewage outfall inspection processes.

Description

A system and method for automatically checking and identifying shoreline sewage outlets Technical Field

[0001] The present invention relates to the technical field of environmental monitoring, and more particularly to a system and method for automatically checking and identifying shoreline sewage outlets. Background Art

[0002] Sewage outlets along river and lake banks play a crucial role in urban wastewater discharge. However, due to management issues or illegal discharge, these outlets can pollute and harm the environment. Therefore, it is crucial to develop a system and method that can automatically detect and identify river outlets.

[0003] Traditional sewage outlet inspection methods are usually summarized as a three-level inspection model of "drone patrol + on-site inspection on foot + quality control." In the first-level inspection, satellite remote sensing and drones are mainly used to identify suspected sewage outlets and suspicious areas through remote sensing map images for subsequent targeted inspections. The second-level inspection is a manual on-foot inspection, which requires inspectors to conduct a carpet-style, full-type inspection along the coastline, focusing on on-site confirmation of relevant information on suspected sewage outlets and historical sewage outlets into rivers in industrial parks, densely populated areas, and environmentally sensitive areas, and to find sewage outlets missed by remote sensing images. The third-level inspection is based on the above and organizes experts to conduct information synthesis, detailed inspections of key areas, and omission detection for problematic sewage outlets, focusing on difficult-to-identify sewage outlets and strengthening quality control.

[0004] Although traditional inspection methods are relatively mature, they still have some significant shortcomings. First, drones, the primary imaging equipment, have insufficient battery life, resulting in a short effective working time for primary inspections and increased costs. Second, due to the difficulty in visualizing the location of sewage outlets and the angle of remote sensing imagery, many outlets cannot be accurately identified, which increases the burden of subsequent manual inspections and increases labor costs. Therefore, traditional inspection methods are both inefficient and costly.

[0005] Therefore, how to improve the accuracy and efficiency of the first-level screening process is an urgent problem that technical personnel in this field need to solve.

[0006] Summary of the Invention

[0007] In view of this, the present invention provides a system and method for automatically checking and identifying shoreline sewage outlets, which can improve the accuracy and efficiency of the sewage outlet inspection process.

[0008] In order to achieve the above object, the present invention adopts the following technical solutions:

[0009] In a first aspect, the present invention provides a shoreline sewage outlet automatic inspection and identification system, comprising: a carrier, a camera module, a side-scan sonar, and an intelligent inspection all-in-one machine;

[0010] The camera module includes four cameras with different viewing angles, which are mounted on the front, back, left, and right sides of the carrier. The front camera is used to record the river environment; the left and right cameras are used to record the environment on both sides of the shoreline; and the rear camera is used to record the posture of the operator.

[0011] The side scan sonar is mounted on both sides of the carrier to scan the underwater scenes on both sides of the coastline;

[0012] The intelligent inspection all-in-one machine is equipped with a bridge drain outlet recognition model, an above-water sewage outlet detection model, an underwater sewage outlet detection model, and a personnel posture detection model;

[0013] The bridge drain outlet recognition model is used to analyze the river environment captured by the front camera and identify the bridge drain outlet; the water sewage outlet detection model is used to identify the water sewage outlet based on the image information captured by the left and right cameras; the underwater sewage outlet detection model is used to identify the underwater sewage outlet based on the underwater scene information captured by the side-scan sonar; the personnel posture detection model is used to determine whether the operating personnel are in a safe working state based on the image information captured by the rear camera.

[0014] Furthermore, the left camera and the right camera use a 60-78 degree automatic zoom module or a 36-42 degree automatic zoom module. When the width of the river bank is less than 60 meters, the 60-78 degree automatic zoom module is used for recording. When the width of the river bank is greater than or equal to 60 meters, the 36-42 degree automatic zoom module is used for recording. A camera adjustment area is also provided on the working interface of the intelligent inspection all-in-one machine for zooming and adjusting the cameras on the left and right sides.

[0015] Furthermore, the intelligent inspection all-in-one machine is also integrated with a GPS module and an IMU posture module; the GPS module is used to collect current position, speed and direction information; the IMU posture module is used to collect current acceleration, angular velocity and geomagnetic direction information;

[0016] The intelligent inspection integrated machine determines whether the confidence level of the GPS module meets the requirements based on a preset precision factor threshold. When the confidence level is less than or equal to the preset precision factor threshold, it is considered that the confidence level of the GPS module meets the requirements, and the GPS module is used to solve the current position, and the information collected by the GPS module is used to correct the IMU posture module;

[0017] When the precision factor is greater than the preset threshold, it is considered that the confidence of the GPS module does not meet the requirements, and the IMU posture module is used to calculate the current position.

[0018] In a second aspect, the present invention provides a method for automatically checking and identifying shoreline sewage outlets, comprising the following steps:

[0019] The front camera based on the carrier records the river environment;

[0020] The left and right cameras based on the carrier record the environment on both sides of the shoreline;

[0021] The rear camera of the carrier records the posture of the operator;

[0022] The side-scan sonars on the left and right sides of the carrier scan the underwater scenes on both sides of the coastline;

[0023] Based on the pre-trained bridge sluice hole recognition model, the river environment captured by the front camera is analyzed to identify the bridge sluice hole;

[0024] The image information collected by the left and right cameras is analyzed based on the pre-trained water sewage outlet detection model to identify the water sewage outlet;

[0025] Identify underwater sewage outlets based on underwater scene information collected by side-scan sonar using a pre-trained underwater sewage outlet detection model;

[0026] Based on the pre-built personnel posture detection model, the image information collected by the rear camera is analyzed to determine whether the operator is in a safe working state.

[0027] Furthermore, the training process of the water sewage outlet detection model includes the following steps:

[0028] Randomly select video data from rural areas, industrial areas, and built-up areas for multiple days, extract a key frame every second from the video data for multiple days to form the initial image set R;

[0029] Filter out the image set R with sewage outlets from the image set R A and the image set R without sewage outlet B ;

[0030] R A The image set is annotated with a rectangular frame to obtain the annotated data set R Am ; For R B The data set is randomly sampled at a ratio of 0.1 to obtain R Bm ;

[0031] Combination R Am and R Bm , get the initial training data set Rit 0 ;

[0032] Based on the initial training dataset R it 0 Train the constructed model to obtain the initial model M it 0 ;

[0033] Based on the initial training dataset R it 0 and the initial model M it 0 Perform iterative optimization of the regional FeedbackLoop to obtain the final water sewage outlet detection model.

[0034] Furthermore, the initial training data set R it 0 and the initial model M it 0 Performing iterative optimization involves the following steps:

[0035] Select multiple days of video frame data from rural areas, industrial areas, and built-up areas for the new operation and perform the i-th training data iteration;

[0036] The model M of the i-1th iteration it i-1 Predict the video frame data for these days and get the dataset R with sewage outlets A i and the dataset R without sewage outlet B i ;

[0037] From the dataset R A i Filter out the dataset R with completely correct predictions and confidence score >= 0.6 A_less i and datasets with prediction errors or prediction confidence scores < 0.6 A_more i ; Dataset R A_less i Random sampling is performed at a ratio of 0.3 and compared with the data set R A_more i Combined together, based on the model prediction frame, fine-tuned through the annotation tool and added to this iteration dataset R it_delta i R A_less Indicates that the dataset is not valuable, R A_more Indicates that the dataset has high value, i indicates the i-th iteration;

[0038] From the dataset R Bi Filter out the dataset with sewage outlets that the model missed R B_more i , R B_more i The dataset is annotated based on the annotation tool and added to the dataset R of this iteration. it_delta i middle;

[0039] The dataset R it_delta i and the training data R of the i-1th iteration it i-1 Combine to get the training data R of the i-th iteration it i ;

[0040] Based on R it i Dataset pair M it i-1 Perform training to obtain the model M after the i-th iteration it i .

[0041] Furthermore, the step of analyzing the working status of the operator by the personnel posture detection model includes:

[0042] Inspect all parts of the operator and mark the key points;

[0043] Based on the distance relationship between key points, determine whether the operator is squatting, sitting, standing, walking or lying down;

[0044] If no key points of the human body are detected and no standing or walking postures have been detected before, or only a small number of key points are detected, it is determined that the operator on the ship has accidentally fallen into the water;

[0045] When the human body posture is determined to be squatting, sitting, standing, or walking, it is determined that the sailors on board are in a normal working state;

[0046] When it is determined that the operator is in a lying position or has fallen into the water, a one-button alarm is triggered and the alarm information is sent to the terminals of other staff on site. The alarm information carries the name, phone number, location of the operator in the abnormal working state and the time when the alarm was first triggered.

[0047] Furthermore, during the inspection of sewage outlets by ship, there are two situations:

[0048] Case 1: The image frame captured by the camera contains sewage outlets, and the model detects all sewage outlets in the image frame;

[0049] Case 2: The image frame captured by the camera contains sewage outlets, but the model does not detect all of them;

[0050] For scenario 1, a lifecycle sequence with an ID is maintained for each sewage outlet. Based on optical flow tracking, sewage outlets with the same ID detected in each frame are added to the corresponding sequence. When the sewage outlet leaves the camera's field of view, the sequence lifecycle is stopped, and the frame with the sewage outlet in the middle is selected as the best shot of the sewage outlet for upload.

[0051] For the second situation, assume that the sewage outlet is correctly detected in the image frames corresponding to times t0 and t2, but the sewage outlet at the middle time t1 is not correctly detected, t0<t1<t2; at this time, the possible location information of the sewage outlet at time t1 is calculated based on the current navigation speed, optical flow direction and the position of the sewage outlet detected in the previous frame, and the calculated predicted location information is added to the life cycle of the sewage outlet; when the sewage outlet or the calculated predicted location information exceeds the camera field of view, the life cycle of the sewage outlet is stopped, and the frame with the sewage outlet in the middle is selected in the actual detected sequence as the best shot for upload.

[0052] Furthermore, the updating process of the life cycle sequence includes:

[0053] The current image frame captured by the camera j The first time a new sewage outlet was detected i j When , create d i j Life cycle sequence l i , and d i j And GNSS position information is stored in sequence l i ;

[0054] For the next consecutive image frame j+1 , based on optical flow tracking to predict the sewage outlet d i j The predicted position d i j+1 _track, while viewing the model in the frame j+1 Sewage outlet detection results on the frame;

[0055] If the predicted position d i j+1 If _track is outside the camera's field of view, the detection life cycle of the sewage outlet is considered to be completed, and the life cycle sequence is terminated. i , and select l i The frame with the sewage outlet in the center of the image is taken as the best shooting frame and the result is uploaded;

[0056] If the model is in the image frame j+1 The sewage outlet d k j+1 _pred, and the sewage outlet d k j+1 _pre and d i j+1 _track intersects, it is considered that in the image frame frame j+1 Detected sewage outlet d k j+1 _pred and in the image frame frame j The detected d i j It is the same sewage outlet, and the predicted location d k j+1 _pred as d i j+1 The sewage outlet is added to l i In the life sequence;

[0057] If the model is in the image frame j+1 No sewage outlet was detected. i j+1 _track intersects, the image frame is considered j+1 The life cycle sequence l i The identity of the sewage outlet is not recognized by the model, and d i j+1 _track as d i j+1 The sewage outlet is added to l i in the life sequence.

[0058] Furthermore, it also includes: performing navigation position calculation based on the GPS module and IMU posture module, specifically including:

[0059] Evaluate the confidence accuracy of the GPS module using the horizontal component DOP and the vertical component DOP;

[0060] When the horizontal component precision coefficient and the vertical component precision coefficient are both less than or equal to the preset threshold, it is considered that the confidence level of the GPS module meets the requirements, the GPS module is used to solve the current position, and the IMU posture module is corrected using the information collected by the GPS module;

[0061] When the horizontal component precision dilution or the vertical component precision dilution is greater than the preset precision dilution threshold, it is considered that the confidence of the GPS module does not meet the requirements, and the IMU posture module is used to solve the current position;

[0062] When the vehicle travels under the bridge, it is assumed that the vehicle maintains a straight line under the bridge to obtain an ideal trajectory; the IMU posture module is used to solve the current position to obtain IMU-solved trajectory information; the ideal trajectory is fitted with the IMU-solved trajectory information to obtain the actual GNSS position information.

[0063] It can be seen from the above technical solutions that compared with the prior art, the present invention has the following beneficial effects:

[0064] 1. This invention installs cameras in four directions, front, back, left, and right, on the carrier, creating a four-view camera system. Based on pre-built models, it can simultaneously identify the bridge's sluice gates, the left and right shoreline water outlets, and the posture of crew members on board. This improves the efficiency and accuracy of outlet detection and ensures that crew members on board maintain a safe working posture. Furthermore, side-scan sonars are installed on both sides of the carrier to effectively identify underwater concealed outlets, increasing the probability of detecting both surface and underwater outlets.

[0065] 2. The present invention installs two cameras with different field of view angles on the left and right sides of the ship to adapt to different river widths and can achieve zooming of image information. It can perform highly accurate intelligent identification of sewage outlets in rivers with a narrowest of 5 meters and a widest of 100 meters, and has a wide range of application scenarios.

[0066] 3. The present invention integrates the GPS module and the IMU posture module, which can accurately fit and calculate the GNSS position information of the ship in scenes with weak or no satellite positioning, such as bridge tunnels.

[0067] 4. The present invention proposes a method for deduplicating and uploading sewage outlets based on a life cycle sequence with an identity ID and a predicted position approximate to the detected position, which can identify the sewage outlets with high accuracy and select the best shooting data for uploading.

[0068] 5. The method for rapid iterative enrichment of training data for the regional FeedbackLoop proposed in this invention can specifically address the problem that models with complex environment data that are inconvenient to expand in large quantities at one time are difficult to quickly reach productization level. Through rapid and purposeful data expansion, the model can expand the scope of new scenarios while the production line is working. BRIEF DESCRIPTION OF THE DRAWINGS

[0069] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are merely embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying any creative work.

[0070] FIG1 is a schematic structural diagram of a system for automatically checking and identifying shoreline sewage outlets provided by the present invention;

[0071] FIG2 is a flow chart of the method for automatically checking and identifying shoreline sewage outlets provided by the present invention;

[0072] FIG3 is a flowchart of staff posture detection and alarm provided by the present invention;

[0073] FIG4 is a schematic diagram of a model for detecting a sewage outlet according to the second situation provided by the present invention;

[0074] FIG5 is a schematic diagram of route solution when driving under a bridge provided by the present invention. DETAILED DESCRIPTION

[0075] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0076] As shown in FIG1 , an embodiment of the present invention discloses an automatic inspection and identification system for shoreline sewage outlets, comprising: a carrier, a camera module, a side-scan sonar, and an intelligent inspection all-in-one machine;

[0077] The camera module includes four cameras with different viewing angles, mounted on the front, back, left, and right sides of the carrier. The front camera is used to record the river environment; the left and right cameras are used to record the environment on both sides of the shoreline; and the rear camera is used to record the posture of the operator.

[0078] Side scan sonars are mounted on both sides of the carrier to scan underwater scenes on both sides of the coastline; one side scan sonar can cover both left and right scans;

[0079] The intelligent inspection all-in-one machine is equipped with a bridge drain outlet recognition model, a water sewage outlet detection model, an underwater sewage outlet detection model, and a personnel posture detection model;

[0080] The bridge drain outlet recognition model is used to analyze the river environment captured by the front camera and identify the bridge drain outlet; the water sewage outlet detection model is used to identify the water sewage outlet based on the image information collected by the cameras on the left and right sides; the underwater sewage outlet detection model is used to identify the underwater sewage outlet based on the underwater scene information collected by the side-scan sonar; the personnel posture detection model is used to determine whether the operating personnel are in a safe working state based on the image information collected by the rear camera.

[0081] This embodiment of the present invention uses a kayak as a carrier. An intelligent water inspection system is mounted at the bow. Cameras on both sides of the system record riverbank structures and identify sewage outlets using a water-based sewage outlet detection model. A front camera records the river ahead and identifies bridge outlets using a bridge outlet recognition model. A rear camera records the actions of crew members and uses a posture detection model to identify human postures and issue warnings. The center of the kayak is the crew's work area, while the stern houses the propellers, which control the boat's direction and provide propulsion. Underwater side-scan sonars are mounted on the sides of the kayak to scan and identify underwater scenes, enabling the identification of concealed pipes. The intelligent inspection system reads data, performs image recognition, and uploads the identified data in real time, enabling the real-time identification of concealed pipes and the construction and upload of result information.

[0082] In one embodiment, the left and right cameras use either a 60-78 degree automatic zoom module or a 36-42 degree automatic zoom module. When the riverbank width is less than 60 meters, the 60-78 degree automatic zoom module is used for recording. At this time, due to the relatively close distance, this FOV range allows for a relatively wide field of view, better capturing various information and details in the scene. When the riverbank width is 60 meters or greater, the 36-42 degree automatic zoom module is used for recording. At this time, due to the relatively long distance, more focused capture of the riverbank area is required. This camera allows for better capture of the riverbank, minimizes image distortion, and better captures detailed information about the embankment scene.

[0083] The present invention adopts different cameras to shoot rivers of different widths, thereby improving the accuracy of detection and making the pictures shot by the cameras more targeted.

[0084] Since the actual river width is continuously variable, in order to make the present invention better applicable to rivers of various widths, a camera adjustment area is also provided on the working interface of the intelligent inspection all-in-one machine, which is used to zoom and adjust the left and right viewing angle cameras. By smoothly and purposefully controlling the zoom of the left and right cameras on the working interface of the intelligent inspection all-in-one machine, the images taken by the left and right cameras can be adjusted to focus on the embankment area, thereby ensuring the adaptability of the images.

[0085] The intelligent inspection all-in-one machine also integrates a GPS module and an IMU posture module; the GPS module is used to collect current position, speed and direction information; the IMU posture module is used to collect current acceleration, angular velocity and geomagnetic direction information;

[0086] The intelligent inspection machine determines whether the confidence level of the GPS module meets the requirements based on the preset precision factor threshold. If the confidence level is less than or equal to the preset precision factor threshold, it is considered that the confidence level of the GPS module meets the requirements, and the GPS module is used to calculate the current position. The information collected by the GPS module is used to correct the IMU posture module.

[0087] When the precision factor is greater than the preset threshold, the confidence of the GPS module is considered to be unsatisfactory, and the IMU posture module is used to calculate the current position.

[0088] As shown in FIG2 , the present invention also provides a method for automatically checking and identifying shoreline sewage outlets, comprising the following steps:

[0089] The front camera based on the carrier records the river environment;

[0090] The left and right cameras based on the carrier record the environment on both sides of the shoreline;

[0091] The rear camera of the carrier records the posture of the operator;

[0092] The side-scan sonars on the left and right sides of the carrier scan the underwater scenes on both sides of the coastline;

[0093] Based on the pre-trained bridge sluice hole recognition model, the river environment captured by the front camera is analyzed to identify the bridge sluice hole;

[0094] The image information collected by the left and right cameras is analyzed based on the pre-trained water sewage outlet detection model to identify the water sewage outlet;

[0095] Identify underwater sewage outlets based on underwater scene information collected by side-scan sonar using a pre-trained underwater sewage outlet detection model;

[0096] Based on the pre-built personnel posture detection model, the image information collected by the rear camera is analyzed to determine whether the operator is in a safe working state.

[0097] The following is a further explanation of the construction and training process of the above three detection models.

[0098] 1. Water sewage outlet detection model.

[0099] The training process of the water outfall detection model includes the following steps:

[0100] 1) Randomly select multiple days of video data from rural areas, industrial areas, and built-up areas (for example, 5 days of video data for each area, totaling 15 days). Extract one key frame per second from the multiple days of video data to form an initial image set R. The number of images in the initial image data set R is more than 200,000.

[0101] 2) Filter out the image set R with sewage outlet from the image set R A and the image set R without sewage outlet B .

[0102] 3) R A The image set is annotated with a rectangular frame to obtain the annotated data set R Am ; For R B The data set is randomly sampled at a ratio of 0.1 to obtain R Bm .

[0103] 4) Combination R Am and R Bm , get the initial training data set R it 0 .

[0104] 5) Based on the initial training data set R it 0 Train the constructed model to obtain the initial model M it 0 .

[0105] 6) Based on the initial training data set R it 0 and the initial model M it 0 Perform iterative optimization of the regional FeedbackLoop to obtain the final water sewage outlet detection model.

[0106] Among them, based on the initial training data set R it 0 and the initial model M it 0 Perform iterative optimization, which includes the following steps:

[0107] a. Select multiple days of video frame data from the rural, industrial, and built-up areas of the new operation (for example, 2 days of video frame data for each new operation area, a total of 6 days), and perform the i-th training data iteration.

[0108] b. The model M of the i-1th iteration it i-1 Predict the video frame data for these days and get the dataset R with sewage outlets A i and the dataset R without sewage outlet B i .

[0109] c. From the dataset R A i Filter out the dataset R with completely correct predictions and confidence score >= 0.6 A_lessi and datasets with prediction errors or prediction confidence scores < 0.6 A_more i ; Dataset R A_less i Random sampling is performed at a ratio of 0.3 and compared with the data set R A_more i Combined together, based on the model prediction frame, fine-tuned through the annotation tool and added to this iteration dataset R it_delta i R A_less Indicates that the dataset is not valuable, R A_more Indicates that the dataset is of high value, and i indicates the i-th iteration.

[0110] d. From the dataset R B i Filter out the dataset with sewage outlets that the model missed R B_more i , R B_more i The dataset is annotated based on the annotation tool and added to the dataset R of this iteration. it_delta i middle.

[0111] e. Data set R it_delta i and the training data R of the i-1th iteration it i-1 Combine to get the training data R of the i-th iteration it i .

[0112] f. Based on R it i Dataset pair M it i-1 Perform training to obtain the model M after the i-th iteration it i .

[0113] To achieve real-time and high-accuracy detection of water-based sewage outlets on both sides of the riverbank, this paper uses the YoloV5large model as the water-based sewage outlet detection model, ensuring both accuracy and real-time detection. TensorRT is a high-performance neural network inference engine developed by NVIDIA that provides maximum throughput and efficiency. This paper uses TensorRT to deploy the trained YoloV5 model and uses half-precision floating-point numbers (FP16) to reduce the model's computational complexity while maintaining accuracy. With GPU edge computing power, a prediction frame rate of 20fps can be achieved.

[0114] 2. Underwater sewage outlet detection model.

[0115] This method selected underwater sewage outlets from different regions, at different times, and with varying sizes and materials to construct a dataset, ultimately creating a dataset of over 10,000 images. Different regions are affected by varying water quality, riverbank environments, and riverbed environments, which can affect side-scan sonar results. Different times are affected by varying operating methods and water surface variations within the same area. The selection of different sizes and materials of sewage outlets aims to enrich the dataset's target diversity. This acquisition method enriches and completes both the background environment and the targets of the data.

[0116] The present invention uses the yolov5 model yolov5l version as the underwater sewage outlet detection model. The input is the image collected by the side-scan sonar. The yolov5 model has a faster detection speed and can meet the real-time requirements in actual work. The yolov5 model is small in size, has low deployment cost, and is easy to train.

[0117] 3. Personnel posture detection model

[0118] The present invention can monitor the working status of the crew on board in real time through the rear camera of the water intelligent inspection all-in-one machine, and analyze the sailors' behavior through the behavior status analysis algorithm to determine whether the operation is safe and whether an early warning is needed.

[0119] Specifically, OpenPose is used to identify human movements and postures. OpenPose detects various parts of the human body and marks their positions, thus forming the human posture. By analyzing the positional relationship of these key points, the current human posture can be determined.

[0120] Specifically, the steps of analyzing the working status of the operator by the personnel posture detection model include:

[0121] 1) Inspect all parts of the operator and mark the key points.

[0122] 2) Based on the distance relationship between the key points, determine whether the operator is squatting, sitting, standing, walking or lying down. Specifically:

[0123] a. When the knee key point and the foot key point are approximately on the same vertical line, and the horizontal position difference between the knee key point and the hip key point is close, it can be determined that the staff member is in a sitting state.

[0124] b. When the key points of the head, knees, and feet are approximately on the same horizontal plane perpendicular to the hull, it can be determined that the staff member is in a standing posture.

[0125] c. When the coordinates of the foot and knee data points are detected to be in the same straight line, and the knee data point coordinates are higher than the foot data point coordinates, it can be determined that the worker is walking. At the same time, the movement of the arms and legs is detected to determine whether the worker is walking. If the gait movement of the legs and the natural swing of the arms are detected at the same time, the worker's walking state can be determined more accurately.

[0126] d. When it is detected that the data point coordinates of the head and feet are on the same straight line, and the data point coordinates of the hips are between the data point coordinates of the head and feet, it can be determined that the staff member is in a lying state.

[0127] The steps for early warning based on the staff's posture are shown in Figure 3, specifically:

[0128] When no key points of the human body are detected and the above-mentioned standing and walking postures have not been detected before (that is, the previous posture is sitting or squatting), or only a small number of key points are detected and the duration is greater than or equal to 5 seconds, a voice inquiry is issued to ask whether it is normal. If an abnormal response or no response is received, it is determined that the operator on the ship has accidentally fallen into the water.

[0129] When the posture of the human body is determined to be squatting, sitting, standing and walking, it is determined that the sailors on the ship are in a normal working state.

[0130] When it is determined that the operator is squatting and the duration is greater than or equal to 90 seconds, a voice inquiry is issued to ask whether it is normal. If an abnormal response or no response is received, a one-button alarm is triggered. Otherwise, the system returns to continue posture detection.

[0131] When it is determined that the operator is in a lying position or has fallen into the water, a one-button alarm is triggered and the alarm information is sent to the terminals of other staff on site. The alarm information carries the name, phone number, location of the operator in the abnormal working state and the time when the alarm was first triggered.

[0132] In a specific embodiment, it is necessary to deduplicate and upload the sewage outlet detection results. During the ship's navigation, when detecting sewage outlets based on the water sewage outlet detection model, there are two situations:

[0133] Case 1: The image frame captured by the camera contains sewage outlets, and the model detects all sewage outlets in the image frame;

[0134] Case 2: The image frame captured by the camera contains sewage outlets, but the model does not detect all of them;

[0135] For scenario 1, a lifecycle sequence with an ID is maintained for each sewage outlet. Based on optical flow tracking, sewage outlets with the same ID detected in each frame are added to the corresponding sequence. When the sewage outlet leaves the camera's field of view, the sequence lifecycle is stopped, and the frame with the sewage outlet in the middle is selected as the best shot of the sewage outlet for upload.

[0136] For the second situation, as shown in Figure 4, it is assumed that the sewage outlet is correctly detected in the image frames corresponding to times t0 and t2, but the sewage outlet at the middle time t1 is not correctly detected, t0<t1<t2; at this time, the possible location information of the sewage outlet at time t1 is calculated based on the current navigation speed, optical flow direction and the position of the sewage outlet detected in the previous frame, and the calculated predicted location information is added to the life cycle of the sewage outlet; when the sewage outlet or the calculated predicted location information exceeds the camera field of view, the life cycle of the sewage outlet is stopped, and the frame with the sewage outlet in the middle is selected in the actual detected sequence as the best shot for upload.

[0137] Specifically, the update process of the lifecycle sequence includes:

[0138] The current image frame captured by the camera j The first time a new sewage outlet was detected i j When , create d i j Life cycle sequence l i , and d i j And GNSS position information is stored in sequence l i ;

[0139] For the next consecutive image frame j+1 , based on optical flow tracking to predict the sewage outlet d i j The predicted position d i j+1 _track, while viewing the model in the frame j+1 Sewage outlet detection results on the frame;

[0140] If the predicted position d i j+1 If _track is outside the camera's field of view, the detection life cycle of the sewage outlet is considered to be completed, and the life cycle sequence is terminated. i , and select l i The frame with the sewage outlet in the center of the image is taken as the best shooting frame and the result is uploaded;

[0141] If the model is in the image frame j+1 The sewage outlet dk j+1 _pred, and the sewage outlet d k j+1 _pre and d i j+1 _track intersects, it is considered that in the image frame frame j+1 Detected sewage outlet d k j+1 _pred and in the image frame frame j The detected d i j It is the same sewage outlet, and the predicted location d k j+1 _pred as d i j+1 The sewage outlet is added to l i In the life sequence;

[0142] If the model is in the image frame j+1 No sewage outlet was detected. i j+1 _track intersects, the image frame is considered j+1 The life cycle sequence l i The identity of the sewage outlet is not recognized by the model, and d i j+1 _track as d i j+1 The sewage outlet is added to l i in the life sequence.

[0143] In another embodiment, the present invention uses a GPS module (Beidou 3 navigation module) and an IMU posture module to form an inertial navigation module. The Beidou 3 navigation module can obtain information such as the device's position, speed, and direction, and the IMU posture module can obtain an object's posture information such as acceleration, angular velocity, and geomagnetic direction. The IMU posture module can perform range estimation and GNSS position calculation based on the currently obtained speed, acceleration, geomagnetic direction, and other information when the satellite positioning signal is poor.

[0144] The present invention is based on the GPS module and the IMU posture module to jointly perform navigation position solution, specifically including:

[0145] When a ship passes under a bridge, the satellite signal may be interfered with, resulting in inaccurate position information. The horizontal dilution of precision (HDOP) and vertical dilution of precision (VDOP) are used to evaluate the confidence accuracy of the GPS module.

[0146] When both the horizontal dilution of precision (HDOP) and the vertical dilution of precision (VDOP) are less than or equal to the preset thresholds, the GPS module's confidence level is considered to meet the requirements, and the GPS module is used to calculate the current position. The optimal HDOP and VDOP thresholds are 5 (the optimal thresholds may vary for different GPS modules; please conduct experimental statistics based on actual conditions). Because the IMU attitude module may accumulate errors over time, resulting in inaccurate attitude measurements, when the satellite confidence level is high, the heading and position of the BeiDou 3 positioning module are used to correct the IMU attitude module's geomagnetic direction information. This reduces the error of the IMU attitude module over long periods of operation and ensures its reliability in subsequent calculations.

[0147] When the horizontal component precision factor or the vertical component precision factor is greater than the preset precision factor threshold, the confidence level of the GPS module is considered to be unsatisfactory, and the IMU posture module is used to solve the current position. Because the ship is affected by factors such as water flow and wind direction during navigation, it may cause the ship to shake, which in turn makes the direction of navigation unstable, making the acceleration error of the IMU posture module larger. When the ship travels to areas with weak satellite signals such as under a bridge, the IMU posture module + position fitting method is used to calculate the ship's precise GNSS information. The details are as follows:

[0148] When the vessel passes under a bridge, assuming it does not change direction, i.e., it maintains a straight line under the bridge, the starting point where the satellite signal weakens (i.e., point A in Figure 5) and the end point where the satellite signal strength recovers (i.e., point B in Figure 5) are recorded. Points A and B are connected to obtain the ideal trajectory of the vessel under the bridge. Simultaneously, the IMU pose module is used to calculate the current position, obtaining the IMU-solved trajectory. Finally, the ideal trajectory is fitted with the IMU-solved trajectory information, and the final fitted trajectory is combined to obtain the precise GNSS position information under the bridge.

[0149] As shown in Figure 5, with the normal vector of the ideal trajectory AB as the projection direction, the offset amplitude of the IMU solution trajectory is divided by 3, which is the fused fitting trajectory:

[0150] here, It means that the normal vector of the ideal trajectory AB is used as the projection direction, and the IMU solves the offset projection of the trajectory at a certain point. is the offset projection of the fused trajectory.

[0151] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. Reference can be made to the common and similar parts between the various embodiments. For the devices disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple, and the relevant parts can be referred to the method description.

[0152] The above description of the disclosed embodiments is intended to enable one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be readily apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention is not limited to the embodiments shown herein but is intended to conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A shoreline sewage outlet automatic inspection and identification system, characterized in that: include: Carrier, camera module, side-scan sonar and intelligent inspection all-in-one machine; The camera module includes four cameras with different viewing angles, which are respectively mounted in the front, rear, left and right directions of the carrier; wherein the front camera is used to record the river environment; the left and right cameras are used to record the environment on both sides of the shoreline; and the rear camera is used to record the posture of the operator; The side scan sonar is mounted on both sides of the carrier to scan the underwater scenes on both sides of the coastline; The intelligent inspection integrated machine is equipped with a bridge drain outlet recognition model, an above-water sewage outlet detection model, an underwater sewage outlet detection model and a personnel posture detection model; The bridge drain outlet recognition model is used to analyze the river environment captured by the front camera and identify the bridge drain outlet; the water sewage outlet detection model is used to identify the water sewage outlet based on the image information captured by the left and right cameras; the underwater sewage outlet detection model is used to identify the underwater sewage outlet based on the underwater scene information captured by the side-scan sonar; the personnel posture detection model is used to determine whether the operating personnel are in a safe working state based on the image information captured by the rear camera.

2. The automatic shoreline sewage outlet inspection and identification system according to claim 1 is characterized in that: The left camera and the right camera use a 60-78 degree automatic zoom module or a 36-42 degree automatic zoom module. When the width of the river bank is less than 60 meters, the 60-78 degree automatic zoom module is used for recording. When the width of the river bank is greater than or equal to 60 meters, the 36-42 degree automatic zoom module is used for recording. A camera adjustment area is also provided on the working interface of the intelligent inspection all-in-one machine for zooming and adjusting the cameras on the left and right sides.

3. The automatic shoreline sewage outlet inspection and identification system according to claim 1 is characterized in that: The intelligent inspection integrated machine also integrates a GPS module and an IMU posture module; the GPS module is used to collect current position, speed and direction information; the IMU posture module is used to collect current acceleration, angular velocity and geomagnetic direction information; The intelligent inspection integrated machine determines whether the confidence of the GPS module meets the requirements according to a preset precision factor threshold. When it is less than or equal to the preset precision factor threshold, it is considered that the confidence of the GPS module meets the requirements, and the GPS module is used to solve the current position, and the information collected by the GPS module is used to correct the IMU posture module; When it is greater than a preset precision factor threshold, it is considered that the confidence of the GPS module does not meet the requirements, and the IMU posture module is used to solve the current position.

4. A method for automatically checking and identifying shoreline sewage outlets, characterized in that: The following steps are involved: The front camera based on the carrier records the river environment; The left and right cameras of the carrier record the sewage outlets on both sides of the shoreline; The rear camera of the carrier records the posture of the operator; The side-scan sonars on the left and right sides of the carrier scan the underwater scenes on both sides of the coastline; Based on the pre-trained bridge spillway recognition model, the river environment captured by the front camera is analyzed to identify the bridge spillway; The image information collected by the cameras on the left and right sides is analyzed based on the pre-trained water sewage outlet detection model to identify the water sewage outlet; Based on the underwater scene information collected by the side-scan sonar using the pre-trained underwater sewage outlet detection model, the underwater sewage outlet is identified; Based on the pre-built personnel posture detection model, the image information collected by the rear camera is analyzed to determine whether the operator is in a safe working state.

5. The method for automatically checking and identifying shoreline sewage outlets according to claim 4 is characterized in that: The training process of the water sewage outlet detection model includes the following steps: Randomly select video data of rural areas, industrial areas and built-up areas for multiple days, extract a key frame per second from the video data of multiple days to form an initial image set R; Filter out the image set R with sewage outlets from the image set R A and the image set R without sewage outlet B ; R A The image set is annotated with a rectangular frame to obtain the annotated data set R Am ; For R B The data set is randomly sampled at a ratio of 0.1 to obtain R Bm ; Combination R Am and R Bm , get the initial training data set R it 0 ; Based on the initial training data set R it 0 Train the constructed model to obtain the initial model M it 0 ; Based on the initial training data set R it 0 and the initial model M it 0 Iterative optimization of the regional FeedbackLoop is performed to obtain the final water sewage outlet detection model.

6. The method for automatically checking and identifying shoreline sewage outlets according to claim 5 is characterized in that: Based on the initial training data set R it 0 and the initial model M it 0 Performing iterative optimization includes the following steps: Select multiple days of video frame data of rural areas, industrial areas and built-up areas of the new operation and perform the i-th training data iteration; The model M of the i-1th iteration it i-1 Predict the video frame data for these days and get the dataset R with sewage outlets A i And the data set R without sewage outlet B i ; From the dataset R A i Filter out the dataset R with completely correct predictions and confidence score>=0.6 A_less i and datasets with prediction errors or prediction confidence score < 0.6 R A_more i ; Dataset R A_less i Random sampling is performed at a ratio of 0.3 and compared with the data set R A_more i Combined together, based on the model prediction box, fine-tuned through the annotation tool and added to this iteration dataset R it_delta i Middle; R A_less Indicates that the data set is not valuable, R A_more Indicates that the data set has high value, and i indicates the i-th iteration; From the dataset R B i Filter out the dataset with sewage outlets that the model missed R B_more i , R B_more i The dataset is annotated based on the annotation tool and added to this iteration dataset R it_delta i middle; The dataset R it_delta i and the training data R of the i-1th iteration it i-1 Combine to get the training data R of the i-th iteration it i ; Based on R it i Dataset pair M it i-1 Train and get the model M after the i-th iteration it i .

7. The method for automatically checking and identifying shoreline sewage outlets according to claim 4 is characterized in that: The steps of analyzing the working status of the operator by the personnel posture detection model include: Test all parts of the operator and mark the key points; According to the distance relationship between the key points, it is judged whether the operator is squatting, sitting, standing, walking or lying down; When no key points of the human body are detected and no standing or walking postures are detected before, or only a small number of key points are detected, it is determined that the operator on the ship has accidentally fallen into the water; When the human body posture is judged to be squatting, sitting, standing and walking, it is judged that the sailors on the ship are in a normal working state; When it is determined that the operator is in a lying position or has fallen into the water, a one-button alarm is triggered and the alarm information is sent to the terminals of other staff on site. The alarm information carries the name, phone number, location of the operator in the abnormal working state and the time when the alarm was first triggered.

8. The method for automatically checking and identifying shoreline sewage outlets according to claim 4 is characterized in that: There are two situations when testing the sewage outlet on a ship: Case 1: The image frame captured by the camera contains sewage outlets, and the model detects all sewage outlets in the image frame; Case 2: The image frame captured by the camera contains sewage outlets, but the model does not detect all sewage outlets; For situation 1, a life cycle sequence with an identity ID is maintained for each sewage outlet. Based on optical flow tracking, the sewage outlet with the same identity ID detected in each frame is added to the corresponding sequence. When the sewage outlet is out of the camera's field of view, the sequence life cycle is stopped, and the frame with the sewage outlet in the middle is selected as the best shot of the sewage outlet for upload; For situation two, assume that the sewage outlet is correctly detected in the image frames corresponding to times t0 and t2, but the sewage outlet at time t1 in the middle is not correctly detected, t0<t1<t2; at this time, the possible location information of the sewage outlet at time t1 is calculated based on the current navigation speed, optical flow direction and the position of the sewage outlet detected in the previous frame, and the calculated predicted location information is added to the life cycle of the sewage outlet; when the sewage outlet or the calculated predicted location information exceeds the camera field of view, the life cycle of the sewage outlet is stopped, and the frame with the sewage outlet in the middle is selected in the actual detected sequence as the best shot for uploading.

9. The method for automatically checking and identifying shoreline sewage outlets according to claim 8 is characterized in that: The updating process of the life cycle sequence includes: The current image frame captured by the camera j The first time a new sewage outlet was detected i j When , create a new i j The life cycle sequence i , and d i j And GNSS position information is stored in sequence l i ; For the next consecutive image frame j+1 , based on optical flow tracking to predict the sewage outlet d i j The predicted position d i j+1 _track, view the model in the frame at the same time j+1 Detection results of sewage outlets on the frame; If the predicted position d i j+1 If _track is outside the camera's field of view, the detection life cycle of the sewage outlet is considered to be completed, and the life cycle sequence l i , and select l i The frame with the sewage outlet in the center of the image is taken as the best shooting frame and the result is uploaded; If the model is in the image frame j+1 The sewage outlet d k j+1 _pred, and the sewage outlet d k j+1 _pre and d i j+1 _track intersects, it is considered that in the image frame frame j+1 Detected sewage outlet d k j+1 _pred and in the image frame frame j The detected d i j is the same sewage outlet, the predicted location d k j+1 _pred as d i j+1 The sewage outlet is added to l i In the life sequence; If the model is in the image frame j+1 No sewage outlet and d were detected i j+1 _track intersects, the image frame is considered j+1 The life cycle sequence on i The identity of the sewage outlet is not recognized by the model, and d i j+1 _track as d i j+1 The sewage outlet is added to l i in the life sequence.

10. The method for automatically checking and identifying shoreline sewage outlets according to claim 4 is characterized in that: Also includes: The navigation position is calculated based on the GPS module and IMU posture module, including: Evaluate the confidence accuracy of the GPS module using the horizontal component precision factor and the vertical component precision factor; When the horizontal component precision factor and the vertical component precision factor are both less than or equal to the preset threshold, it is considered that the confidence of the GPS module meets the requirement, the GPS module is used to solve the current position, and the IMU posture module is corrected using the information collected by the GPS module; When the horizontal component precision factor or the vertical component precision factor is greater than the preset precision factor threshold, it is considered that the confidence of the GPS module does not meet the requirement, and the IMU posture module is used to solve the current position; When the carrier drives under the bridge, it is assumed that the carrier keeps driving in a straight line under the bridge to obtain an ideal trajectory; the IMU posture module is used to solve the current position to obtain IMU solution trajectory information; the ideal trajectory is fitted with the IMU solution trajectory information to obtain the actual GNSS position information.

Images