An automatic detection and identification system and method for shoreline sewage outlets

Abstract

This invention relates to the field of environmental monitoring technology, specifically to an automatic system and method for identifying and detecting shoreline sewage outlets. The system includes: a carrier, a four-view camera, a side-scan sonar, and an intelligent inspection integrated machine. The intelligent inspection integrated machine is equipped with a bridge-mounted drainage outlet identification model, a surface sewage outlet detection model, an underwater sewage outlet detection model, and a personnel posture detection model. The bridge-mounted drainage outlet identification model analyzes the river environment captured by the front camera to identify the bridge-mounted drainage outlet. The surface sewage outlet detection model identifies surface sewage outlets based on image information captured by cameras on both sides. The underwater sewage outlet detection model identifies underwater sewage outlets based on underwater scene information captured by the side-scan sonar. The personnel posture detection model determines whether the personnel are in a safe working state based on image information captured by the rear camera. This invention can improve the accuracy and efficiency of the sewage outlet inspection process.

Description

Technical Field

[0001] This invention relates to the field of environmental monitoring technology, and more specifically to an automatic system and method for identifying and detecting sewage outlets along the shoreline. Background Technology

[0002] Sewage outlets along river and lake shorelines play a significant role in urban wastewater discharge. However, due to management issues or illegal discharges, these outlets can cause pollution and damage to the environment. Therefore, developing a system and method for automatically detecting and identifying sewage outlets flowing into rivers is of paramount importance.

[0003] Traditional methods for investigating sewage outlets are typically summarized as a three-tiered investigation model: "drone patrols + on-site foot inspections + quality control." The first tier primarily utilizes satellite remote sensing and drones to identify suspected sewage outlets and suspicious areas through remote sensing imagery, facilitating targeted investigations in subsequent work. The second tier involves manual, on-foot inspections, requiring personnel to conduct a comprehensive, carpet-like search along the shoreline. Emphasis is placed on verifying information regarding suspected and historical sewage outlets flowing into rivers in industrial parks, densely populated areas, and environmentally sensitive regions, identifying outlets missed by remote sensing imagery. The third tier builds upon the above, organizing experts to conduct information reconciliation, detailed investigations of key areas, and gap filling for problematic outlets. Special attention is given to difficult-to-confirm outlets, with strengthened quality control.

[0004] Although traditional investigation methods are relatively mature, they still have some significant drawbacks. First, drones, as the primary imaging equipment, have insufficient battery life, resulting in a short effective working time for the first-level investigation and thus increasing the overall cost of the first-level investigation. Second, due to the difficulty in visually identifying the location of sewage outlets and the angle issues when capturing remote sensing images, many sewage outlets cannot be accurately identified, which increases the burden of subsequent manual investigation and raises labor costs. Therefore, traditional investigation methods are both inefficient and costly.

[0005] Therefore, how to improve the accuracy and efficiency of the primary screening process is a problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0006] In view of this, the present invention provides an automatic investigation and identification system and method for shoreline sewage outlets, which can improve the accuracy and efficiency of the sewage outlet investigation process.

[0007] To achieve the above objectives, the present invention adopts the following technical solution:

[0008] In a first aspect, the present invention provides an automatic inspection and identification system for shoreline sewage outlets, comprising: a carrier, a camera module, a side-scan sonar, and an intelligent inspection integrated machine;

[0009] The camera module includes four cameras with different perspectives, mounted on the front, back, left, and right sides of the carrier respectively; the front camera is used to record the river environment; the left and right side cameras are used to record the environment on both sides of the shoreline; and the rear camera is used to record the posture of the workers.

[0010] The side-scan sonar is mounted on both sides of the carrier and is used to scan the underwater scene on both sides of the shoreline.

[0011] The intelligent inspection all-in-one machine is equipped with a bridge drainage outlet recognition model, a water surface sewage outlet detection model, an underwater sewage outlet detection model, and a personnel posture detection model.

[0012] The bridge spillway identification model is used to analyze the river environment captured by the front camera and identify the bridge spillway; the surface sewage outlet detection model is used to identify surface sewage outlets based on image information captured by cameras on the left and right sides; the underwater sewage outlet detection model is used to identify underwater sewage outlets based on underwater scene information captured by the side-scan sonar; and the personnel posture detection model is used to determine whether the workers are in a safe working state based on image information captured by the rear camera.

[0013] Furthermore, the left and right cameras employ either a 60-78 degree auto zoom module or a 36-42 degree auto zoom module. When the riverbank width is less than 60 meters, the 60-78 degree auto zoom module is used for recording; when the riverbank width is greater than or equal to 60 meters, the 36-42 degree auto zoom module is used for recording. The intelligent inspection all-in-one machine also has a camera adjustment area on its working interface for scaling the left and right cameras.

[0014] Furthermore, the intelligent inspection all-in-one machine also integrates a GPS module and an IMU pose module; the GPS module is used to collect current position, speed, and direction information; the IMU pose module is used to collect current acceleration, angular velocity, and geomagnetic direction information.

[0015] The intelligent inspection all-in-one machine determines whether the confidence level of the GPS module meets the requirements based on a preset accuracy factor threshold. When the confidence level of the GPS module is less than or equal to the preset accuracy factor threshold, it is considered that the confidence level of the GPS module meets the requirements. The GPS module is used to calculate the current position, and the information collected by the GPS module is used to correct the IMU pose module.

[0016] When the accuracy factor is greater than the preset threshold, the confidence level of the GPS module is considered to be insufficient, and the IMU pose module is used to calculate the current position.

[0017] Secondly, the present invention provides an automatic method for identifying and detecting shoreline sewage outlets, comprising the following steps:

[0018] A carrier-based front-facing camera records the river environment.

[0019] The left and right cameras on the carrier record video of the environment on both sides of the shoreline.

[0020] The rear camera of the carrier records the posture of the workers.

[0021] The underwater scene on both sides of the shoreline is scanned by side-scan sonar on the left and right sides of the carrier.

[0022] Based on a pre-trained bridge spillway recognition model, the river environment captured by the camera in front is analyzed to identify the bridge spillway.

[0023] Based on the pre-trained water discharge outlet detection model, the image information collected by the cameras on the left and right sides is analyzed to identify the water discharge outlet.

[0024] Based on the pre-trained underwater sewage outlet detection model, underwater sewage outlets are identified using underwater scene information collected by side-scan sonar.

[0025] Based on a pre-built personnel posture detection model, the image information collected by the rear camera is analyzed to determine whether the operator is in a safe working state.

[0026] Furthermore, the training process of the waterborne sewage outlet detection model includes the following steps:

[0027] Randomly select video data from rural areas, industrial areas and built-up areas for multiple days, and extract one keyframe per second from the video data from multiple days to form an initial image set R;

[0028] Select the image set R containing sewage outlets from the image set R. A Image set R without sewage outlet B ;

[0029] For R A The image set is labeled with rectangular boxes to obtain the labeled dataset R. Am For R B The dataset was randomly sampled at a ratio of 0.1 to obtain R. Bm ;

[0030] Combination R Am and R Bm The initial training dataset R is obtained. it 0 ;

[0031] Based on the initial training dataset R it0 The constructed model is trained to obtain the initial model M. it 0 ;

[0032] Based on the initial training dataset R it 0 and the initial model M it 0 The regional FeedbackLoop is iteratively optimized to obtain the final detection model for waterborne sewage outlets.

[0033] Furthermore, the initial training dataset R... it 0 and the initial model M it 0 Iterative optimization includes the following steps:

[0034] Select video frame data from rural areas, industrial areas and built-up areas of the new task over multiple days, and perform the i-th training data iteration;

[0035] The model M in the (i-1)th iteration it i-1 Predictions were made from the video frame data over several days to obtain a dataset R containing sewage outlets. A i And the dataset R without sewage outlets B i ;

[0036] From dataset R A i Select datasets R that contain all correct predictions and a confidence score >= 0.6. A_less i The dataset R with prediction errors or prediction confidence scores < 0.6 A_more i Dataset R A_less i Random sampling was performed at a ratio of 0.3, and the dataset R was compared with the dataset R. A_more i Combined together, and based on the model's predicted bounding boxes, after fine-tuning using annotation tools, they are added to the current iteration's dataset R. it_delta i Middle; R A_less This indicates that the dataset is not of high value, R A_more This indicates that the dataset is of high value, and i represents the i-th iteration;

[0037] From dataset R B i Filter out datasets with sewage outlets in R that were missed by the model. B_more i R B_morei The dataset was annotated using a labeling tool and then added to the current iteration dataset R. it_delta i middle;

[0038] Data set R it_delta i With the training data R of the (i-1)th iteration it i-1 Combine the data to obtain the training data R for the i-th iteration. it i ;

[0039] Based on R it i Dataset pair M it i-1 Training is performed to obtain the model M after the i-th iteration. it i .

[0040] Furthermore, the steps for the personnel posture detection model to analyze the working status of the workers include:

[0041] Inspect all parts of the workers' bodies and mark the locations of key points;

[0042] Based on the distance relationship between key points, determine whether the worker is in a squatting, sitting, standing, walking, or lying position;

[0043] If no key points of the human body are detected and no standing or walking postures have been detected beforehand, or only a few key points are detected, it is determined that the workers on the ship have accidentally fallen into the water.

[0044] When a person's posture is determined to be squatting, sitting, standing, or walking, it is determined that the sailors on the ship are in a normal working state.

[0045] When it is determined that the worker is lying down or has fallen into the water, a one-button alarm is triggered, and the alarm information is sent to the terminals of other staff on site. The alarm information includes the name, phone number, location of the worker in the abnormal working state, and the time when the alarm was first triggered.

[0046] Furthermore, during the process of inspecting the sewage outlet from the ship, two situations arise:

[0047] Scenario 1: The image frame captured by the camera contains sewage outlets, and the model detects all sewage outlets in the image frame;

[0048] Scenario 2: The image frames captured by the camera contain sewage outlets, but the model does not detect all of them;

[0049] For scenario one, maintain a lifecycle sequence with an identity ID for each sewage outlet. Based on optical flow tracing, add sewage outlets with the same identity ID detected in each frame to the corresponding sequence. Stop the lifecycle sequence when the sewage outlet leaves the camera's field of view, and select the frame in the middle of the sewage outlet as the best shot for the sewage outlet to upload.

[0050] For scenario two, assuming the sewage outlet is correctly detected in the image frames corresponding to times t0 and t2, but not in the middle at time t1, where t0 < t1 < t2; in this case, the possible location of the sewage outlet at time t1 is calculated based on the current navigation speed, optical flow direction, and the location of the sewage outlet detected in the previous frame, and the calculated predicted location information is added to the lifecycle of the sewage outlet; when the sewage outlet or the calculated predicted location information exceeds the camera's field of view, the lifecycle of the sewage outlet is stopped, and the frame with the sewage outlet in the middle is selected from the actually detected sequence as the best shot for uploading.

[0051] Furthermore, the update process of the lifecycle sequence includes:

[0052] The current image frame captured by the camera. j The first new sewage outlet d was detected in the middle. i j When, create d i j Life cycle sequence l i and d i j and GNSS location information stored in sequence l i ;

[0053] For the next consecutive image frame j+1 Based on optical flow tracing, predict the discharge outlet d i j Predicted position d i j+1 _track, and view the model in the frame. j+1 The results of the sewage outlet detection on the frame;

[0054] If the predicted position d i j+1 If the track is outside the camera's field of view, the detection lifecycle of the discharge outlet is considered complete, and the lifecycle sequence ends. i , and select l i The frame with the sewage outlet most centered in the image is used as the best shooting frame for uploading the results.

[0055] If the model is in the image frame j+1 The sewage outlet d was detected in the middle. kj+1 _pred, and the sewage outlet d k j+1 _pre and d i j+1 If the _tracks intersect, it is considered that they are in the same image frame. j+1 The detected sewage outlet d k j+1 _pred and in the image frame j The detected d i j It is the same sewage outlet, predicting the location d. k j+1 _pred as d i j+1 Add sewage outlet to l i In the life sequence;

[0056] If the model is in the image frame j+1 No sewage outlets were detected in the middle and d i j+1 If the _tracks intersect, then the image frame is considered to be... j+1 Lifecycle sequence l i The sewage outlet was not identified by the model, and d i j+1 _track as d i j+1 Add sewage outlet to l i In the life sequence.

[0057] Furthermore, it also includes: calculating navigation position based on the GPS module and IMU pose module, specifically including:

[0058] The confidence accuracy of the GPS module is evaluated using the horizontal component accuracy factor and the vertical component accuracy factor.

[0059] When both the horizontal component accuracy factor and the vertical component accuracy factor are less than or equal to the preset threshold, the confidence level of the GPS module is considered to meet the requirements. The GPS module is then used to calculate the current position, and the information collected by the GPS module is used to correct the IMU pose module.

[0060] When the accuracy factor of the horizontal component or the accuracy factor of the vertical component is greater than the preset accuracy factor threshold, the confidence level of the GPS module is considered to be insufficient, and the IMU pose module is used to calculate the current position.

[0061] When the vehicle travels under the bridge, it is assumed that the vehicle travels in a straight line under the bridge to obtain an ideal trajectory; the IMU pose module is used to calculate the current position to obtain the IMU calculated trajectory information; the ideal trajectory is fitted with the IMU calculated trajectory information to obtain the actual GNSS position information.

[0062] As can be seen from the above technical solution, compared with the prior art, the present invention has the following beneficial effects:

[0063] 1. This invention utilizes cameras mounted in four directions (front, back, left, and right) on a carrier to form a four-view camera system. Based on a pre-built model, it can simultaneously identify the drainage outlets on the bridge, the sewage outlets on the left and right shorelines, and the postures of personnel on board, improving the detection efficiency and accuracy of sewage outlets while ensuring that personnel on board are in a safe working posture. Simultaneously, this invention also incorporates side-scan sonar on both sides of the vessel to effectively identify underwater concealed sewage outlets, increasing the detection probability of both above-water and underwater sewage outlets.

[0064] 2. This invention uses two cameras with different field of view installed on the left and right sides of the boat to adapt to different river widths and to zoom in and out of the image information. It can perform highly accurate intelligent identification of sewage outlets in river channels that are as narrow as 5 meters and as wide as 100 meters, and has a wide range of applications.

[0065] 3. This invention integrates a GPS module and an IMU pose module, which can accurately fit and calculate the GNSS position information of ships in scenarios with weak or no satellite positioning, such as under bridges.

[0066] 4. The present invention proposes a sewage outlet deduplication and uploading method based on a lifecycle sequence with identity ID and a predicted location approximate detection location, which can accurately identify sewage outlets and select the best captured data for uploading.

[0067] 5. The regional Feedback Loop training data rapid iteration enrichment method proposed in this invention can specifically solve the problem that models in complex environments are difficult to quickly reach the product level due to the inconvenience of large-scale data expansion at one time. Through rapid and purposeful data expansion, the model can expand the scope of new scenarios while working on the production line. Attached Figure Description

[0068] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0069] Figure 1This is a schematic diagram of the automatic shoreline sewage outlet detection and identification system provided by the present invention.

[0070] Figure 2 A flowchart of the automatic detection and identification method for shoreline sewage outlets provided by the present invention;

[0071] Figure 3 The flowchart for worker posture detection and alarm provided by this invention;

[0072] Figure 4 This is a schematic diagram of the model detecting the sewage outlet for situation two, provided by the present invention.

[0073] Figure 5 This is a schematic diagram of the route calculation when driving under a bridge, provided by the present invention. Detailed Implementation

[0074] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0075] like Figure 1 As shown in the figure, an embodiment of the present invention discloses an automatic inspection and identification system for shoreline sewage outlets, including: a carrier, a camera module, a side-scan sonar, and an intelligent inspection integrated machine;

[0076] The camera module includes four cameras with different perspectives, mounted in the front, back, left, and right directions of the carrier. The front camera is used to record the river environment; the left and right side cameras are used to record the environment on both sides of the shoreline; and the rear camera is used to record the posture of the workers.

[0077] Side-scan sonars are mounted on both sides of the carrier to scan the underwater scene on both sides of the shoreline; one side-scan sonar can cover both the left and right sides.

[0078] The intelligent inspection all-in-one machine is equipped with a bridge drainage outlet recognition model, a water surface sewage outlet detection model, an underwater sewage outlet detection model, and a personnel posture detection model.

[0079] The bridge spillway identification model is used to analyze the river environment captured by the front camera and identify the bridge spillway; the surface sewage outlet detection model is used to identify surface sewage outlets based on image information captured by cameras on the left and right sides; the underwater sewage outlet detection model is used to identify underwater sewage outlets based on underwater scene information captured by side-scan sonar; and the personnel posture detection model is used to determine whether the workers are in a safe working state based on image information captured by the rear camera.

[0080] This invention utilizes a kayak as one of the carrier forms. An integrated intelligent waterborne inspection device is mounted at the bow. Cameras on both sides of the device record structures on both sides of the riverbank and identify sewage outlets using a waterborne sewage outlet detection model. A front camera records the river channel ahead and identifies bridge drainage outlets using a bridge drainage outlet identification model. A rear camera records the behavior of personnel on board and uses a personnel posture detection model for human posture recognition and early warning. The center of the hull is the crew's work area, while the stern houses the fixed propeller area, which controls the boat's direction and provides power. Underwater side-scan sonar is mounted on the sides of the boat to perform side-scanning and identification of underwater scenes, including the identification of underwater pipes. The intelligent inspection device reads data, performs image recognition, and uploads the identified data in real time, enabling real-time identification of underwater pipes and the real-time construction and uploading of results information.

[0081] In one embodiment, the left and right cameras employ either a 60-78 degree auto-zoom module or a 36-42 degree auto-zoom module. When the riverbank width is less than 60 meters, the 60-78 degree auto-zoom module is used for recording. At this distance, due to the relatively close proximity, this FOV range provides a wider field of view, allowing for better recording of various information and details in the scene. When the riverbank width is 60 meters or greater, the 36-42 degree auto-zoom module is used. At this distance, due to the relatively greater distance, a more focused approach to shooting the riverbank area is needed. This camera allows for better targeting of the riverbank, control of image distortion, and better monitoring of the details of the embankment scene.

[0082] This invention employs different cameras for river channels of varying widths, thereby improving detection accuracy and making the captured images more targeted.

[0083] Since the actual width of a river is continuous and varied, in order to make this invention more applicable to rivers of various widths, the working interface of the intelligent inspection all-in-one machine is also equipped with a camera adjustment area for scaling the left and right view cameras. By smoothly and purposefully scaling the left and right cameras on the working interface of the intelligent inspection all-in-one machine, the images captured by the left and right cameras can be adjusted to focus on the embankment area, ensuring the adaptability of the images.

[0084] The intelligent inspection all-in-one machine also integrates a GPS module and an IMU pose module; the GPS module is used to collect current position, speed and direction information; the IMU pose module is used to collect current acceleration, angular velocity and geomagnetic direction information;

[0085] The intelligent inspection all-in-one machine judges whether the confidence level of the GPS module meets the requirements based on the preset accuracy factor threshold. When it is less than or equal to the preset accuracy factor threshold, the confidence level of the GPS module is considered to meet the requirements. The GPS module is used to calculate the current position, and the information collected by the GPS module is used to correct the IMU pose module.

[0086] When the accuracy factor exceeds the preset threshold, the confidence level of the GPS module is considered insufficient, and the IMU pose module is used to calculate the current position.

[0087] like Figure 2 As shown, the present invention also provides an automatic method for identifying and detecting shoreline sewage outlets, comprising the following steps:

[0088] A carrier-based front-facing camera records the river environment.

[0089] The left and right cameras on the carrier record video of the environment on both sides of the shoreline.

[0090] The rear camera of the carrier records the posture of the workers.

[0091] The underwater scene on both sides of the shoreline is scanned by side-scan sonar on the left and right sides of the carrier.

[0092] Based on a pre-trained bridge spillway recognition model, the river environment captured by the camera in front is analyzed to identify the bridge spillway.

[0093] Based on the pre-trained water discharge outlet detection model, the image information collected by the cameras on the left and right sides is analyzed to identify the water discharge outlet.

[0094] Based on the pre-trained underwater sewage outlet detection model, underwater sewage outlets are identified using underwater scene information collected by side-scan sonar.

[0095] Based on a pre-built personnel posture detection model, the image information collected by the rear camera is analyzed to determine whether the operator is in a safe working state.

[0096] The construction and training process of the three detection models mentioned above will be further explained below.

[0097] 1. Detection model for sewage outlets on water.

[0098] The training process for the waterborne sewage outlet detection model includes the following steps:

[0099] 1) Randomly select video data from rural areas, industrial areas and built-up areas for multiple days (e.g., 5 days of video data for each area, totaling 15 days). Extract one keyframe per second from the video data from multiple days to form an initial image set R. The initial image dataset R contains more than 200,000 images.

[0100] 2) Select the image set R containing sewage outlets from the image set R. A Image set R without sewage outlet B .

[0101] 3) For R A The image set is labeled with rectangular boxes to obtain the labeled dataset R. Am For R B The dataset was randomly sampled at a ratio of 0.1 to obtain R. Bm .

[0102] 4) Combination R Am and R Bm The initial training dataset R is obtained. it 0 .

[0103] 5) Based on the initial training dataset R it 0 The constructed model is trained to obtain the initial model M. it 0 .

[0104] 6) Based on the initial training dataset R it 0 and the initial model M it 0 The regional FeedbackLoop is iteratively optimized to obtain the final detection model for waterborne sewage outlets.

[0105] Among them, based on the initial training dataset R it 0 and the initial model M it 0 Iterative optimization includes the following steps:

[0106] a. Select video frame data from rural areas, industrial areas, and built-up areas of the new operation for multiple days (e.g., 2 days of video frame data for each new operation area, totaling 6 days) and perform the i-th training data iteration.

[0107] b. The model M in the (i-1)th iteration it i-1 Predictions were made from the video frame data over several days to obtain a dataset R containing sewage outlets. A i And the dataset R without sewage outlets B i .

[0108] c. From dataset R A i Select datasets R that contain all correct predictions and a confidence score >= 0.6.A_less i The dataset R with prediction errors or prediction confidence scores < 0.6 A_more i Dataset R A_less i Random sampling was performed at a ratio of 0.3, and the dataset R was compared with the dataset R. A_more i Combined together, and based on the model's predicted bounding boxes, after fine-tuning using annotation tools, they are added to the current iteration's dataset R. it_delta i Middle; R A_less This indicates that the dataset is not of high value, R A_more The dataset is considered valuable, and i represents the i-th iteration.

[0109] d. From dataset R B i Filter out datasets with sewage outlets in R that were missed by the model. B_more i R B_more i The dataset was annotated using a labeling tool and then added to the current iteration dataset R. it_delta i middle.

[0110] e. Transfer the dataset R it_delta i With the training data R of the (i-1)th iteration it i-1 Combine the data to obtain the training data R for the i-th iteration. it i .

[0111] f. Based on R it i Dataset pair M it i-1 Training is performed to obtain the model M after the i-th iteration. it i .

[0112] To achieve real-time and high-accuracy detection of sewage outlets on both sides of the riverbank, this invention selects YoloV5large as the sewage outlet detection model, ensuring both detection accuracy and real-time performance. TensorRT, a high-performance neural network inference engine developed by NVIDIA, can provide maximum throughput and efficiency. This invention uses TensorRT to deploy the trained YoloV5 model, while using half-precision floating-point numbers (FP16) to reduce the computational load of the model while maintaining accuracy. Under GPU edge computing power, a prediction frame rate of 20fps can be achieved.

[0113] 2. Underwater sewage outlet detection model.

[0114] This invention selected underwater sewage outlets from different regions, at different times, and with varying thicknesses and materials to construct a dataset, ultimately resulting in a dataset of over ten thousand images. The different regions are chosen because varying water quality, riverbank environments, and riverbed environments can affect the side-scan sonar results; the different times are due to the influence of different operating methods and water changes within the same region on the side-scan sonar results; and the different thicknesses and materials of the underwater sewage outlets are used to make the dataset more comprehensive in terms of the types of targets detected. This method of data collection enriches and completes the background environment and targets of the data.

[0115] This invention uses the YOLOv5 model, version YOLOv5l, as the underwater sewage outlet detection model. The input is an image acquired by a side-scan sonar. The YOLOv5 model has a faster detection speed, which can meet the real-time requirements in actual work. The YOLOv5 model has a small size, low deployment cost, and is easy to train.

[0116] 3. Personnel posture detection model

[0117] This invention uses the rear camera of the intelligent waterborne inspection machine to monitor the working status of the crew in real time. At the same time, it analyzes the sailors' behavior through a behavior status analysis algorithm to determine whether the operation is safe and whether an early warning is needed.

[0118] Specifically, OpenPose is used to recognize human movements and postures. OpenPose detects various parts of the human body and marks their positions, thus forming the human posture. By analyzing the positional relationships of these key points, the current human posture can be determined.

[0119] Specifically, the steps of the personnel posture detection model to analyze the working status of workers include:

[0120] 1) Inspect all parts of the workers and mark the key points.

[0121] 2) Determine the worker's posture (squatting, sitting, standing, walking, or lying down) based on the distance relationships between key points. Specifically:

[0122] a. When the knee key point and the foot key point are approximately on the same vertical line, and the horizontal position difference between the knee key point and the hip key point is relatively small, it can be determined that the staff member is in a sitting position.

[0123] b. When the key points of the head, knees, and feet are approximately on the same horizontal plane perpendicular to the hull, it can be determined that the worker is in a standing posture.

[0124] c. When the data points of the foot and knee are detected to be on the same straight line, and the knee data point is higher than the foot data point, it can be determined that the worker is walking. Simultaneously, the movement of the arms and legs can be detected to determine if the worker is walking. If both leg gait movements and natural arm swings are detected simultaneously, the determination of whether the worker is walking can be more accurate.

[0125] d. When the data point coordinates of the head and feet are detected to be on the same straight line, and the data point coordinates of the buttocks are between the data point coordinates of the head and feet, it can be determined that the staff member is in a lying position.

[0126] The steps for issuing early warnings based on staff posture are as follows: Figure 3 As shown, specifically:

[0127] If no key points of the human body are detected and the aforementioned standing and walking postures were not detected beforehand (i.e., the preceding posture is sitting or squatting), or only a few key points are detected and the duration is greater than or equal to 5 seconds, then a voice inquiry is made to ask if it is normal. If an abnormal response or no response is received, it is determined that the workers on the ship have accidentally fallen into the water.

[0128] When a person's posture is determined to be squatting, sitting, standing, or walking, it is judged that the sailors on the ship are in a normal working state.

[0129] When it is determined that the worker is crouching for a period of 90 seconds or more, a voice prompt is sent to ask if it is normal. If an abnormal response or no response is received, a one-click alarm is triggered; otherwise, the posture detection is resumed.

[0130] When it is determined that the worker is lying down or has fallen into the water, a one-button alarm is triggered, and the alarm information is sent to the terminals of other staff on site. The alarm information includes the name, phone number, location of the worker in the abnormal working state, and the time when the alarm was first triggered.

[0131] In one specific embodiment, it is necessary to deduplicate and upload the sewage outlet detection results. During the ship's navigation, when detecting sewage outlets based on the waterborne sewage outlet detection model, there are two situations:

[0132] Scenario 1: The image frame captured by the camera contains sewage outlets, and the model detects all sewage outlets in the image frame;

[0133] Scenario 2: The image frames captured by the camera contain sewage outlets, but the model does not detect all of them;

[0134] For scenario one, maintain a lifecycle sequence with an identity ID for each sewage outlet. Based on optical flow tracing, add sewage outlets with the same identity ID detected in each frame to the corresponding sequence. Stop the lifecycle sequence when the sewage outlet leaves the camera's field of view, and select the frame in the middle of the sewage outlet as the best shot for the sewage outlet to upload.

[0135] Regarding scenario two, such as Figure 4 As shown, assuming the sewage outlet is correctly detected in the image frames corresponding to times t0 and t2, but not in the middle time t1, t0 < t1 < t2; at this time, the possible location information of the sewage outlet at time t1 is calculated based on the current navigation speed, optical flow direction, and the location of the sewage outlet detected in the previous frame, and the calculated predicted location information is added to the lifecycle of the sewage outlet; when the sewage outlet or the calculated predicted location information exceeds the camera's field of view, the lifecycle of the sewage outlet is stopped, and the frame with the sewage outlet in the middle is selected from the actually detected sequence as the best shot for uploading.

[0136] Specifically, the lifecycle sequence update process includes:

[0137] The current image frame captured by the camera. j The first new sewage outlet d was detected in the middle. i j When, create d i j Life cycle sequence l i and d i j and GNSS location information stored in sequence l i ;

[0138] For the next consecutive image frame j+1 Based on optical flow tracing, predict the discharge outlet d i j Predicted position d i j+1 _track, and view the model in the frame. j+1 The results of the sewage outlet detection on the frame;

[0139] If the predicted position d i j+1 If the track is outside the camera's field of view, the detection lifecycle of the discharge outlet is considered complete, and the lifecycle sequence ends. i , and select l i The frame with the sewage outlet most centered in the image is used as the best shooting frame for uploading the results.

[0140] If the model is in the image frame j+1The sewage outlet d was detected in the middle. k j+1 _pred, and the sewage outlet d k j+1 _pre and d i j+1 If the _tracks intersect, it is considered that they are in the same image frame. j+1 The detected sewage outlet d k j+1 _pred and in the image frame j The detected d i j It is the same sewage outlet, predicting the location d. k j+1 _pred as d i j+1 Add sewage outlet to l i In the life sequence;

[0141] If the model is in the image frame j+1 No sewage outlets were detected in the middle and d i j+1 If the _tracks intersect, then the image frame is considered to be... j+1 Lifecycle sequence l i The sewage outlet was not identified by the model, and d i j+1 _track as d i j+1 Add sewage outlet to l i In the life sequence.

[0142] In another embodiment, the present invention uses a GPS module (BeiDou-3 navigation module) and an IMU pose module to jointly constitute an inertial navigation module. The BeiDou-3 navigation module can obtain information such as the device's position, velocity, and orientation, while the IMU pose module can obtain pose information such as the object's acceleration, angular velocity, and geomagnetic direction. When satellite positioning signals are poor, the IMU pose module can perform range estimation and GNSS position calculation based on the currently obtained velocity, acceleration, and geomagnetic direction information.

[0143] This invention calculates navigation position based on the joint operation of a GPS module and an IMU pose module, specifically including:

[0144] When a ship sails under a bridge, the satellite signal may be interfered with, resulting in inaccurate position information. The confidence accuracy of the GPS module is evaluated using the horizontal component accuracy factor HDOP and the vertical component accuracy factor VDOP.

[0145] When both the horizontal component accuracy factor (HDOP) and the vertical component accuracy factor (VDOP) are less than or equal to preset thresholds, the confidence level of the GPS module is considered to meet the requirements, and the GPS module is used to calculate the current position. The optimal thresholds for HDOP and VDOP are 5 (the optimal thresholds may vary for different GPS modules; please conduct experiments and statistics based on actual conditions). Because the IMU pose module accumulates errors over long-term use, leading to inaccurate pose measurements, when the satellite confidence level is high, the heading and position of the BeiDou-3 positioning module are used to correct the geomagnetic direction information of the IMU pose module. This reduces the error of the IMU pose module during long-term operation and ensures the reliability of the IMU pose module in subsequent calculations.

[0146] When the accuracy factor of the horizontal component or the accuracy factor of the vertical component exceeds the preset accuracy factor threshold, the confidence level of the GPS module is considered insufficient, and the IMU pose module is used to calculate the current position. Because the ship is affected by factors such as water flow and wind direction during its journey, it may experience swaying and other phenomena, leading to instability in its direction of travel and increasing the acceleration error of the IMU pose module. When the ship travels to areas with weak satellite signals, such as under a bridge, a combination of IMU pose module and position fitting is used to calculate the ship's accurate GNSS information. The details are as follows:

[0147] When the vehicle travels under the bridge, assuming it does not change direction and maintains a straight line throughout its journey, the starting point where the satellite signal weakens is recorded (i.e.,...). Figure 5 Point A in the diagram records the endpoint of satellite signal strength recovery (i.e., point A). Figure 5 Point B in the diagram is used, and points A and B are connected to obtain the fitted ideal trajectory information of the ship under the bridge. At the same time, the IMU pose module is used to calculate the current position to obtain the IMU calculated trajectory information; finally, the ideal trajectory and the IMU calculated trajectory information are fitted together, and the final fitted ship trajectory is combined to obtain the accurate GNSS position information under the bridge.

[0148] like Figure 5 As shown, with the normal vector of the ideal trajectory AB as the projection direction, the offset magnitude of the trajectory calculated by the IMU divided by 3 is the fused fitted trajectory:

[0149]

[0150] here, This indicates that, with the normal vector of the ideal trajectory AB as the projection direction, the IMU calculates the offset projection of the trajectory at a certain point. This is the offset projection of the fused trajectory.

[0151] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to the method section.

[0152] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. An automatic detection and identification system for shoreline sewage outlets, characterized in that, include: Carrier, camera module, side-scan sonar and intelligent inspection integrated machine; The camera module includes four cameras with different perspectives, mounted on the front, back, left, and right sides of the carrier respectively; the front camera is used to record the river environment; the left and right side cameras are used to record the environment on both sides of the shoreline; and the rear camera is used to record the posture of the workers. The side-scan sonar is mounted on both sides of the carrier and is used to scan the underwater scene on both sides of the shoreline. The intelligent inspection all-in-one machine is equipped with a bridge drainage outlet recognition model, a water surface sewage outlet detection model, an underwater sewage outlet detection model, and a personnel posture detection model. The bridge spillway identification model is used to analyze the river environment captured by the front camera and identify the bridge spillway; the surface sewage outlet detection model is used to identify surface sewage outlets based on image information captured by cameras on the left and right sides; the underwater sewage outlet detection model is used to identify underwater sewage outlets based on underwater scene information captured by the side-scan sonar; and the personnel posture detection model is used to determine whether the workers are in a safe working state based on image information captured by the rear camera. During navigation, when detecting sewage outlets based on the waterborne sewage outlet detection model, two situations arise: Scenario 1: The image frame captured by the camera contains sewage outlets, and the model detects all sewage outlets in the image frame; Scenario 2: The image frames captured by the camera contain sewage outlets, but the model does not detect all of them; For scenario one, maintain a lifecycle sequence with an identity ID for each sewage outlet. Based on optical flow tracing, add sewage outlets with the same identity ID detected in each frame to the corresponding sequence. Stop the lifecycle sequence when the sewage outlet leaves the camera's field of view, and select the frame in the middle of the sewage outlet as the best shot for the sewage outlet to upload. For scenario two, assuming the sewage outlet is correctly detected in the image frames corresponding to times t0 and t2, but not in the middle at time t1, where t0 < t1 < t2; in this case, the possible location of the sewage outlet at time t1 is calculated based on the current navigation speed, optical flow direction, and the location of the sewage outlet detected in the previous frame, and the calculated predicted location information is added to the lifecycle of the sewage outlet; when the sewage outlet or the calculated predicted location information exceeds the camera's field of view, the lifecycle of the sewage outlet is stopped, and the frame with the sewage outlet in the middle is selected from the actually detected sequence as the best shot for uploading.

2. The automatic shoreline sewage outlet detection and identification system according to claim 1, characterized in that, The left and right cameras use either a 60-78 degree auto zoom module or a 36-42 degree auto zoom module. When the riverbank width is less than 60 meters, the 60-78 degree auto zoom module is used for recording, and when the riverbank width is greater than or equal to 60 meters, the 36-42 degree auto zoom module is used for recording. The working interface of the intelligent inspection all-in-one machine also has a camera adjustment area for zooming and adjusting the left and right cameras.

3. The automatic detection and identification system for shoreline sewage outlets according to claim 1, characterized in that, The intelligent inspection all-in-one machine also integrates a GPS module and an IMU pose module; the GPS module is used to collect current position, speed and direction information; the IMU pose module is used to collect current acceleration, angular velocity and geomagnetic direction information; The intelligent inspection all-in-one machine determines whether the confidence level of the GPS module meets the requirements based on a preset accuracy factor threshold. When the confidence level of the GPS module is less than or equal to the preset accuracy factor threshold, it is considered that the confidence level of the GPS module meets the requirements. The GPS module is used to calculate the current position, and the information collected by the GPS module is used to correct the IMU pose module. When the accuracy factor is greater than the preset threshold, the confidence level of the GPS module is considered to be insufficient, and the IMU pose module is used to calculate the current position.

4. A method for automatically identifying and detecting sewage outlets along a shoreline, characterized in that, Includes the following steps: A carrier-based front-facing camera records the river environment. The left and right cameras on the carrier record video of the sewage outlets on both sides of the shoreline. The rear camera of the carrier records the posture of the workers. The underwater scene on both sides of the shoreline is scanned by side-scan sonar on the left and right sides of the carrier. Based on a pre-trained bridge spillway recognition model, the river environment captured by the camera in front is analyzed to identify the bridge spillway. Based on the pre-trained water discharge outlet detection model, the image information collected by the cameras on the left and right sides is analyzed to identify the water discharge outlet. Based on the pre-trained underwater sewage outlet detection model, underwater sewage outlets are identified using underwater scene information collected by side-scan sonar. Based on a pre-built personnel posture detection model, the image information collected by the rear camera is analyzed to determine whether the operator is in a safe working state. During navigation, when detecting sewage outlets based on the waterborne sewage outlet detection model, two situations arise: Scenario 1: The image frame captured by the camera contains sewage outlets, and the model detects all sewage outlets in the image frame; Scenario 2: The image frames captured by the camera contain sewage outlets, but the model does not detect all of them; For scenario one, maintain a lifecycle sequence with an identity ID for each sewage outlet. Based on optical flow tracing, add sewage outlets with the same identity ID detected in each frame to the corresponding sequence. Stop the lifecycle sequence when the sewage outlet leaves the camera's field of view, and select the frame in the middle of the sewage outlet as the best shot for the sewage outlet to upload. For scenario two, assuming the sewage outlet is correctly detected in the image frames corresponding to times t0 and t2, but not in the middle at time t1, where t0 < t1 < t2; in this case, the possible location of the sewage outlet at time t1 is calculated based on the current navigation speed, optical flow direction, and the location of the sewage outlet detected in the previous frame, and the calculated predicted location information is added to the lifecycle of the sewage outlet; when the sewage outlet or the calculated predicted location information exceeds the camera's field of view, the lifecycle of the sewage outlet is stopped, and the frame with the sewage outlet in the middle is selected from the actually detected sequence as the best shot for uploading.

5. The automatic detection and identification method for shoreline sewage outlets according to claim 4, characterized in that, The training process of the waterborne sewage outlet detection model includes the following steps: Randomly select video data from rural areas, industrial areas and built-up areas for multiple days, and extract one keyframe per second from the video data from multiple days to form an initial image set R; R is the image set filtered from the image set R A R is the image set filtered from the image set R B ; For R A The image set is labeled with rectangular boxes to obtain the labeled dataset R. Am For R B The dataset was randomly sampled at a ratio of 0.1 to obtain R. Bm ; Combination R Am and R Bm The initial training dataset R is obtained. it 0 ; Based on the initial training dataset R it 0 The constructed model is trained to obtain the initial model M. it 0 ; Based on the initial training dataset R it 0 and the initial model M it 0 The regional FeedbackLoop is iteratively optimized to obtain the final detection model for waterborne sewage outlets.

6. The automatic detection and identification method for shoreline sewage outlets according to claim 5, characterized in that, The initial training dataset R it 0 and the initial model M it 0 Iterative optimization includes the following steps: Select video frame data from rural areas, industrial areas and built-up areas of the new task over multiple days, and perform the i-th training data iteration; The model M in the (i-1)th iteration it i-1 Predictions were made from the video frame data over several days to obtain a dataset R containing sewage outlets. A i And the dataset R without sewage outlets B i ; From dataset R A i Select datasets R that contain all correct predictions and a confidence score >= 0.

6. A_less i The dataset R with prediction errors or prediction confidence scores < 0.6 A_more i Dataset R A_less i Random sampling was performed at a ratio of 0.3, and the dataset R was compared with the dataset R. A_more i Combined together, and based on the model's predicted bounding boxes, after fine-tuning using annotation tools, they are added to the current iteration's dataset R. it_delta i Middle; R A_less This indicates that the dataset is not of high value, R A_more This indicates that the dataset is of high value, and i represents the i-th iteration; From dataset R B i Filter out datasets with sewage outlets in R that were missed by the model. B_more i R B_more i The dataset was annotated using a labeling tool and then added to the current iteration dataset R. it_delta i middle; Data set R it_delta i With the training data R of the (i-1)th iteration it i-1 Combine the data to obtain the training data R for the i-th iteration. it i ; Based on R it i Dataset pair M it i-1 Training is performed to obtain the model M after the i-th iteration. it i .

7. The automatic detection and identification method for shoreline sewage outlets according to claim 4, characterized in that, The steps of the personnel posture detection model in analyzing the working status of workers include: Inspect all parts of the workers' bodies and mark the locations of key points; Based on the distance relationship between key points, determine whether the worker is in a squatting, sitting, standing, walking, or lying position; If no key points of the human body are detected and no standing or walking postures have been detected beforehand, or only a few key points are detected, it is determined that the workers on the ship have accidentally fallen into the water. When a person's posture is determined to be squatting, sitting, standing, or walking, it is determined that the sailors on the ship are in a normal working state. When it is determined that the worker is lying down or has fallen into the water, a one-button alarm is triggered, and the alarm information is sent to the terminals of other staff on site. The alarm information includes the name, phone number, location of the worker in the abnormal working state, and the time when the alarm was first triggered.

8. The automatic detection and identification method for shoreline sewage outlets according to claim 4, characterized in that, The update process of the lifecycle sequence includes: The current image frame captured by the camera. j The first new sewage outlet d was detected in the middle. i j When, create d i j Life cycle sequence l i and d i j and GNSS location information stored in sequence l i ; For the next consecutive image frame j+1 Based on optical flow tracing, predict the discharge outlet d i j Predicted position d i j+1 _track, and view the model in the frame. j+1 The results of the sewage outlet detection on the frame; If the predicted position d i j+1 If the track is outside the camera's field of view, the detection lifecycle of the discharge outlet is considered complete, and the lifecycle sequence ends. i , and select l i The frame with the sewage outlet most centered in the image is used as the best shooting frame for uploading the results. If the model is in the image frame j+1 The sewage outlet d was detected in the middle. k j+1 _pred, and the sewage outlet d k j+1 _pre and d i j+1 If the _tracks intersect, it is considered that they are in the same image frame. j+1 The detected sewage outlet d k j+1 _pred and in the image frame j The detected d i j It is the same sewage outlet, predicting the location d. k j+1 _pred as d i j+1 Add sewage outlet to l i In the life sequence; If the model is in the image frame j+1 No sewage outlets were detected in the middle and d i j+1 If the _tracks intersect, then the image frame is considered to be... j+1 Lifecycle sequence l i The sewage outlet was not identified by the model, and d i j+1 _track as d i j+1 Add sewage outlet to l i In the life sequence.

9. The automatic detection and identification method for shoreline sewage outlets according to claim 4, characterized in that, Also includes: The navigation position is calculated based on the GPS module and the IMU pose module, specifically including: The confidence accuracy of the GPS module is evaluated using the horizontal component accuracy factor and the vertical component accuracy factor. When both the horizontal component accuracy factor and the vertical component accuracy factor are less than or equal to the preset threshold, the confidence level of the GPS module is considered to meet the requirements. The GPS module is then used to calculate the current position, and the information collected by the GPS module is used to correct the IMU pose module. When the accuracy factor of the horizontal component or the accuracy factor of the vertical component is greater than the preset accuracy factor threshold, the confidence level of the GPS module is considered to be insufficient, and the IMU pose module is used to calculate the current position. When the vehicle travels under the bridge, it is assumed that the vehicle travels in a straight line under the bridge to obtain an ideal trajectory; the IMU pose module is used to calculate the current position to obtain the IMU calculated trajectory information; the ideal trajectory is fitted with the IMU calculated trajectory information to obtain the actual GNSS position information.

Images