Water area target detection method and water area target detection system

Abstract

The invention relates to the technical field of target detection, in particular to a water area target detection method and a water area target detection system. The technical problems that in the prior art, vibration and displacement are generated due to waves and wind power, the data alignment precision is reduced, and the positioning precision is affected are solved or improved. According to the water area target detection method provided by the invention, the initial point cloud data detected by the laser radar and the initial image shot by the camera are dynamically compensated by adopting the real-time attitude information so as to generate the compensated point cloud data and image, and fusion is performed according to the compensated point cloud data and image so as to generate the fusion data; and determining whether the target object exists in the target area based on the fusion data. The real-time attitude information is adopted to dynamically compensate the laser radar and the camera, so that the influence of vibration, swing and the like on the camera and the laser radar in the water area environment on the data precision is eliminated, the continuous precision of space-time mapping is ensured, and the global positioning precision of the target is further improved.

Description

Technical Field

[0001] This application relates to the field of target detection technology, specifically to a method and system for detecting targets in water areas. Background Technology

[0002] Safety issues frequently arise in key waterways such as rivers, lakes, parks, scenic areas, and reservoirs, with numerous drowning incidents resulting from illegal swimming and accidental falls into the water. To safeguard water safety, intelligent life-saving systems have emerged, enabling automatic detection and rapid rescue of individuals falling into the water.

[0003] Existing target detection technologies can be categorized into two types based on data type: image target detection and point cloud target detection. These technologies identify all targets of interest in an image or point cloud and determine their category and precise location.

[0004] However, whether target detection is performed based on images or point cloud data, corresponding detectors, such as cameras and lidar, are required. However, since the carrier (such as a ship) is in a water environment, it is easily affected by waves and wind. Therefore, the cameras and lidar installed on the carrier will vibrate and shift due to waves and wind. This leads to a decrease in the data alignment accuracy between multiple cameras, multiple lidars, and between cameras and lidars, thus affecting the global positioning accuracy of the target. Summary of the Invention

[0005] In view of this, this application provides a water target detection method and a water target detection system, which solves or improves the technical problem in the prior art that the data alignment accuracy between multiple cameras, multiple lidars, and between cameras and lidars is reduced due to vibration and displacement caused by waves and wind, thereby affecting the global positioning accuracy of the target.

[0006] According to the first aspect of this application, this application provides a method for detecting water targets, comprising: The inertial data of the carrier detected by the inertial sensors mounted on the carrier is processed to generate the attitude data of the carrier. The inertial sensors include a gyroscope, an accelerometer, and a magnetometer. The inertial data of the carrier includes angular velocity, acceleration, and magnetic field strength. The attitude data of the carrier includes the three-axis translational degrees of freedom, the three-axis rotational degrees of freedom, and the heading angle of the carrier. Based on the attitude data of the carrier, attitude compensation is performed on multiple initial images of the target area captured by multiple cameras to generate multiple images of the target area; The initial point cloud data of the target area detected by the lidar is compensated for according to the attitude data of the carrier to generate the point cloud data of the target area. The attitude data of the carrier includes the three-axis translational degrees of freedom, the three-axis rotational degrees of freedom and the heading angle of the carrier. The presence of a target object in the target area is determined based on multiple images and the point cloud data.

[0007] In one embodiment of this application, the step of processing the inertial data of the carrier detected by the inertial sensor mounted on the carrier to generate the attitude data of the carrier includes: Based on the angular velocity, acceleration, and magnetic field strength of the carrier, at least one of the following weights is adjusted: a first weight for the angular velocity, a second weight for the acceleration, and a third weight for the magnetic field strength. The angular velocity and its first weight, the acceleration and its second weight, and the magnetic field strength and its third weight are input into the extended Kalman filter for calculation to generate the attitude data of the carrier.

[0008] In one embodiment of this application, adjusting at least one of the first weight of the angular velocity, the second weight of the acceleration, and the third weight of the magnetic field strength based on the angular velocity, acceleration, and magnetic field strength of the carrier includes: The modulus deviation and acceleration change rate are calculated based on the acceleration of the carrier; and when the modulus deviation is greater than a first preset deviation, or the acceleration change rate is greater than a first preset change rate, or the angular velocity is greater than a first preset angular velocity, the second weight of the acceleration is reduced, and the reduced second weight is greater than 0.05 and less than 0.2.

[0009] In one embodiment of this application, adjusting at least one of the first weight of the angular velocity, the second weight of the acceleration, and the third weight of the magnetic field strength based on the angular velocity, acceleration, and magnetic field strength of the carrier further includes: The magnetic field strength deviation is calculated based on the magnetic field strength; the magnetic field direction and heading angle are determined based on the magnetic field strength; the direction deviation is calculated based on the magnetic field direction and the geomagnetic angle; and the heading deviation is calculated based on the heading angle and the target heading angle. If the magnetic field strength deviation is greater than a first preset magnetic field deviation, or the direction deviation is greater than a preset direction deviation, or the heading deviation is greater than a first preset heading deviation, the third weight of the magnetic field strength is reduced; or When the magnetic field strength deviation is greater than the second preset magnetic field deviation or the heading deviation is greater than the second preset heading deviation, the third weight of the magnetic field strength is reduced to 0, wherein the first preset magnetic field deviation is less than the second preset magnetic field deviation and the first preset heading deviation is less than the second preset heading deviation.

[0010] In one embodiment of this application, adjusting at least one of the first weight of the angular velocity, the second weight of the acceleration, and the third weight of the magnetic field strength based on the angular velocity, acceleration, and magnetic field strength of the carrier further includes: When the angular velocity is less than the second preset angular velocity within a preset time period, and the magnitude deviation is less than the second preset deviation, and the acceleration change rate is less than the second preset change rate, the sum of the first weight and the third weight is adjusted to be greater than or equal to 0.8; wherein, the second preset angular velocity is less than the first preset angular velocity, the second preset deviation is less than the first preset deviation, and the second preset change rate is less than the first preset change rate.

[0011] In one embodiment of this application, the step of performing pose compensation on multiple initial images of a target region captured by multiple cameras based on the pose data of the carrier to generate multiple images of the target region includes: The attitude change of the carrier is calculated based on the attitude data of the carrier in two adjacent frames. Transform the initial image of the current frame to the reference plane to obtain pixel drift; Based on the pose change and the pixel drift, determine whether the initial image needs compensation; When the initial image needs compensation, the initial image is compensated based on the camera intrinsic parameters, the extrinsic parameters of the camera and the inertial sensor, the parameters of the reference plane, and the attitude change of the carrier.

[0012] In one embodiment of this application, determining whether the initial image needs compensation based on the pose change and the pixel drift includes: When the maximum value of the change in the three-axis rotational degrees of freedom during the attitude change is greater than or equal to 0.1°, it is determined that the initial image needs compensation; or When the pixel drift is greater than or equal to 0.5, it is determined that the initial image needs to be compensated.

[0013] In one embodiment of this application, the step of performing attitude compensation on the initial point cloud data of the target area monitored by the lidar based on the attitude data of the carrier to generate point cloud data of the target area includes: The angular velocity and acceleration of the carrier are calculated based on the three-axis rotational degrees of freedom and three-axis translational degrees of freedom of the carrier during the scanning cycle, and the angle change and displacement change of the carrier during the scanning cycle are calculated respectively. When the angle change value is greater than the preset angle change value or the displacement change value is greater than the preset displacement change value, the initial point cloud data detected by the lidar within the scanning cycle is determined to be the target compensation point cloud data. Based on the extrinsic parameter matrix between the inertial sensor and the lidar, calculate the coordinate transformation matrix from the lidar coordinate system to the standard attitude coordinate system; Based on the target compensation point cloud data in the lidar coordinate system and the coordinate transformation matrix, the pose difference is calculated, which includes rotation difference and translation difference. The target compensation point cloud data is compensated based on the pose difference.

[0014] In one embodiment of this application, when there is a time difference between the inertial sensor and the lidar; After calculating the coordinate transformation matrix from the lidar coordinate system to the standard attitude coordinate system, the detection method further includes: The time deviation between the inertial sensor and the lidar was determined by the external parameter-time delay joint calibration method. The coordinate transformation matrix is ​​corrected based on the time deviation.

[0015] As a second aspect of this application, this application also provides a water target detection system, comprising: Multiple cameras are used to capture initial images of the target area around the carrier; Multiple lidar sensors are used to detect point cloud data of the target area around the carrier. An inertial sensor, wherein the inertial sensor is used to detect the inertial data of the carrier; The detection controller includes a processor and a memory, the memory storing a computer program, and when the processor executes the computer program, it performs the above-described method for detecting water targets.

[0016] This application provides a method for detecting targets in water areas. It utilizes real-time attitude information of a carrier detected by an inertial sensor to dynamically compensate for initial point cloud data detected by a lidar and initial images captured by a camera, generating compensated point cloud data and images. These compensated point cloud data and images are then fused to generate fused data, which is used to determine the presence of a target in the target area. By employing real-time attitude information for dynamic compensation of the lidar and camera, the method effectively eliminates the impact of vibration and swaying of the camera and lidar in the water environment on data accuracy. This ensures the continuous accuracy of the spatiotemporal mapping between multiple cameras, multiple lidars, and between cameras and lidars, further improving the global positioning accuracy of the target. Attached Figure Description

[0017] The above and other objects, features, and advantages of this application will become more apparent from the more detailed description of the embodiments of this application in conjunction with the accompanying drawings. The drawings are provided to further illustrate the embodiments of this application and form part of the specification. They are used together with the embodiments of this application to explain this application and do not constitute a limitation thereof. In the drawings, the same reference numerals generally represent the same components or steps.

[0018] Figure 1 The diagram shown is a flowchart of a water target detection method provided in an embodiment of this application.

[0019] Figure 2 The diagram shown is a schematic representation of the installation positions of the camera and lidar provided in one embodiment of this application.

[0020] Figure 3 The diagram shown is a schematic representation of the installation positions of the camera and lidar according to another embodiment of this application.

[0021] Figure 4 The diagram shown is a schematic representation of the installation positions of the camera and lidar according to another embodiment of this application.

[0022] Figure 5 The diagram shown is a flowchart of a water target detection method provided in an embodiment of this application.

[0023] Figure 6 The diagram shown is a flowchart of a water target detection method provided in another embodiment of this application.

[0024] Figure 7 The diagram shown is a schematic diagram of the working principle of an electronic device provided in an embodiment of this application. Detailed Implementation

[0025] In the description of this application, "multiple" means at least two, such as two, three, etc., unless otherwise explicitly specified. All directional indications (such as up, down, left, right, front, back, top, bottom, etc.) in the embodiments of this application are only used to explain the relative positional relationships and movement of the components in a specific posture (as shown in the figures). If the specific posture changes, the directional indication will also change accordingly. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or device that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products, or devices.

[0026] Furthermore, the reference to "embodiment" herein means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0027] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0028] Exemplary methods As a first aspect of this application, this application provides a method for detecting targets in water areas. Figure 1 The diagram shown is a flowchart illustrating a water target detection method according to an embodiment of this application. Figure 1 As shown, the water target detection method provided in this application includes the following steps: S10: Solve the inertial data of the carrier detected by the inertial sensor installed on the carrier to generate the carrier's attitude data.

[0029] Specifically, inertial sensors are installed on the carrier (such as a ship). These inertial sensors include gyroscopes, accelerometers, and magnetometers. The gyroscopes are used to detect the angular velocity of the carrier, the accelerometers are used to detect the acceleration of the carrier, and the magnetometers are used to detect the magnetic field strength of the carrier. Therefore, the inertial data of the carrier includes angular velocity, acceleration, and magnetic field strength.

[0030] The attitude data of the carrier can be obtained by solving the inertial data. The attitude data includes three translational degrees of freedom, namely linear motion along mutually perpendicular coordinate axes (X, Y, Z); three rotational degrees of freedom, namely rotational motion around three mutually perpendicular coordinate axes (X, Y, Z); and heading angle.

[0031] S20: Based on the carrier's attitude data, perform attitude compensation on multiple initial images of the target area captured by multiple cameras to generate multiple images of the target area; Dynamic attitude compensation is performed on multiple initial images captured by the camera based on the attitude data of the carrier, effectively eliminating the impact of camera vibration and swaying on data accuracy in the aquatic environment.

[0032] Specifically, in combination Figures 2-4As shown, multiple cameras 100 and lidar 200 are mounted on the carrier. The multiple cameras 100 and lidar 200 form a detection device assembly. The number of cameras 100, the number of lidar 200, and the positional relationship between the cameras 100 and lidar 200 in this detection device assembly can be defined as follows: (1) There are 3 cameras 100 and 1 lidar 200, and the 3 cameras 100 are arranged around the lidar 200, such as Figure 2 As shown, the field of view that can be captured by any two adjacent cameras 100 partially overlaps.

[0033] Specifically, the angle between any two of the three cameras 100 is 120°, meaning that the three cameras 100 are evenly arranged around the lidar 200.

[0034] Specifically, the field of view of the LiDAR 200 is 360°.

[0035] (2) There are two cameras 100 and one lidar 200, with the two cameras 100 positioned on either side of the lidar 200, such as... Figure 3 As shown.

[0036] Similarly, the fields of view captured by the two cameras 100 partially overlap. The field of view of the lidar 200 is 120°.

[0037] It should be noted that a set of detection devices can be set on the carrier, and this set of detection devices can be set at the head or the tail of the carrier.

[0038] Two sets of detection devices can also be installed on the carrier, one set at the head of the carrier and the other set at the tail of the carrier, such as... Figure 4 As shown. When the two sets of detection devices are respectively set at the head and tail of the carrier, the set of detection devices at the tail of the carrier includes one lidar 200 and three cameras 100 evenly arranged around the lidar 200. The set of detection devices at the head of the carrier includes one lidar 200 and two cameras 100 arranged on both sides of the lidar 200, as shown. Figure 4 As shown.

[0039] Specifically, such as Figure 4 As shown, when both the head and tail of the carrier are equipped with detection device combinations, the field of view that any two cameras in the two detection device combinations can capture partially overlaps. In this way, a panoramic image of the carrier's surroundings can be obtained from the initial image captured by each camera.

[0040] When the cameras capture images of the water area surrounding the carrier and the lidar detects the water area around the carrier, the initial images collected are the initial images captured by all the cameras set up on the carrier, for example... Figure 4 As shown, when the carrier head has two cameras and the carrier tail has three cameras, the initial images acquired are the initial images of the target area captured by all the cameras. Specifically, the target area refers to the water area surrounding the carrier, including the water area to the left of the carrier, the water area to the right of the carrier, etc. The acquired point cloud data is the point cloud data of the target area detected by the two lidar sensors.

[0041] S30: Based on the carrier's attitude data, perform attitude compensation on the initial point cloud data of the target area detected by the lidar to generate point cloud data of the target area; Dynamic attitude compensation is performed on the initial point cloud data detected by the lidar based on the attitude data of the carrier, effectively eliminating the impact of lidar vibration and swaying on data accuracy in the aquatic environment.

[0042] S40: Determine whether a target object exists in the target area based on multiple images and point cloud data.

[0043] Specifically, S40: Determining whether a target object exists in the target area based on multiple images and point cloud data may include the following steps: S401: Fuse multiple images to generate a panoramic image of the target area; S402: Acquire point cloud data detected by LiDAR, and map the point cloud data to the panoramic image according to the data mapping relationship between LiDAR and camera to obtain fused data; S403: Determine whether a target object exists in the target area and the location of the target object based on the fused data.

[0044] By leveraging the data mapping relationship between LiDAR and cameras, pixel-level precise mapping of multi-sensor data is achieved, significantly reducing fusion errors and enabling accurate target positioning in the global coordinate system, thereby improving positioning accuracy.

[0045] This application provides a method for detecting targets in water areas. It utilizes real-time attitude information of a carrier detected by an inertial sensor to dynamically compensate for initial point cloud data detected by a lidar and initial images captured by a camera, generating compensated point cloud data and images. These compensated point cloud data and images are then fused to generate fused data, which is used to determine the presence of a target in the target area. By employing real-time attitude information for dynamic compensation of the lidar and camera, the method effectively eliminates the impact of vibration and swaying of the camera and lidar in the water environment on data accuracy. This ensures the continuous accuracy of the spatiotemporal mapping between multiple cameras, multiple lidars, and between cameras and lidars, further improving the global positioning accuracy of the target.

[0046] In one embodiment of this application, an adaptive Kalman filter algorithm is used to solve the inertial data in order to achieve high-precision pose estimation. Specifically, S10 (solving the inertial data of the carrier detected by the inertial sensor mounted on the carrier to generate the carrier's attitude data) includes the following steps: S101: Adjust at least one of the following weights based on the angular velocity, acceleration, and magnetic field strength of the carrier: the first weight of angular velocity, the second weight of acceleration, and the third weight of magnetic field strength. Gyroscopes provide angular velocity information and have high short-term accuracy but are prone to drift; accelerometers can provide gravity direction reference when the vehicle is in static or uniform motion, but are susceptible to dynamic acceleration interference; magnetometers provide magnetic north direction reference, but are susceptible to electromagnetic interference. Therefore, the weights of angular velocity, acceleration and magnetic field strength are dynamically adjusted according to the motion state of the vehicle.

[0047] Specifically, the adjustment methods for the weights of the carrier's angular velocity, acceleration, and magnetic field strength can be as follows: (1) When the carrier is in violent motion, the weight of acceleration is significantly reduced: The modulus deviation and acceleration change rate are calculated based on the acceleration of the carrier; and when the modulus deviation is greater than the first preset deviation, or the acceleration change rate is greater than the first preset change rate, or the angular velocity is greater than the first preset angular velocity, the second weight of the acceleration is reduced, and the reduced second weight is greater than 0.05 and less than 0.2.

[0048] Specifically, let the second weight of the accelerometer be... This is achieved by scaling the noise covariance. . (Modulus deviation) Rate of change of acceleration And using angular velocity |ω| as the dynamic intensity index, calculate the dynamic adjustment value of the second weight w_a: Therefore, when the modulus deviates from |Δg|>0.2g or the rate of change of acceleration... When the angular velocity |ω|>50° / s, w_a decreases significantly, dropping to 0.05–0.2, and continuously adapts to changes with the index.

[0049] (2) When the magnetic field interference is large, reduce the weight of the magnetic field strength: The magnetic field strength deviation is calculated based on the magnetic field strength. The magnetic field direction and heading angle are determined based on the magnetic field strength. The direction deviation is calculated based on the magnetic field direction and the geomagnetic angle. The heading deviation is calculated based on the heading angle and the target heading angle. If the magnetic field strength deviation is greater than a first preset magnetic field deviation, or the direction deviation is greater than a preset direction deviation, or the heading deviation is greater than a first preset heading deviation, the third weight of the magnetic field strength is reduced. When the magnetic field strength deviation is greater than the second preset magnetic field deviation or the heading deviation is greater than the second preset heading deviation, the third weight of the magnetic field strength is reduced to 0, wherein the first preset magnetic field deviation is less than the second preset magnetic field deviation and the first preset heading deviation is less than the second preset heading deviation.

[0050] Specifically, let the weight of the magnetic field strength be... ,and Magnetic intensity deviation The directional deviation δ_θ and heading deviation r_ψ are indicators, and the weights of the magnetic field strength are dynamically adjusted. Therefore, when the magnetic intensity deviation |δ_B|>0.15 or the direction deviation δ_θ>10° or the heading deviation r_ψ>8°, the weight of the magnetic field is significantly reduced, that is, the third weight is significantly reduced. The first preset magnetic field deviation is 0.15, the preset direction deviation is 10°, and the first preset heading deviation is 8°.

[0051] When the magnetic field strength deviation increases further, for example, |δ_B|>0.4 or the heading deviation increases further, for example, r_ψ>15°, the weight w_m of the magnetic field strength can be reduced to 0, that is, the magnetometer update can be temporarily turned off. The second magnetic field deviation is 0.4, and the first preset heading deviation is 15°.

[0052] (3) When the carrier is stationary or moving at a constant speed, the weights of the lifting acceleration and the magnetic field strength are: When the angular velocity is less than the second preset angular velocity within the preset time period, the magnitude deviation is less than the second preset deviation, and the rate of change of acceleration is less than the second preset rate of change, the sum of the first weight and the third weight is adjusted to be greater than or equal to 0.8; wherein, the second preset angular velocity is less than the first preset angular velocity, the second preset deviation is less than the first preset deviation, and the second preset rate of change is less than the first preset rate of change.

[0053] Specifically, for example, the preset duration can be 1 second. Within the window T_s=1 second, if the angular velocity |ω|<0.5° / s, the magnitude deviation |a_m|-g<0.02g, and the mean of the rate of change of acceleration <0.05 m·s^-3, the carrier is determined to be stationary or in uniform linear motion; at this time, the weight of angular velocity w_a and the weight of secondary magnetic field strength w_m are increased to ≥0.8, and zero angular velocity / zero specific force update is enabled to correct gyroscope drift over a long period of time.

[0054] S102: Input the angular velocity and its first weight, the acceleration and its second weight, and the magnetic field strength and its third weight into the extended Kalman filter for calculation to generate the carrier's attitude data.

[0055] After dynamic weight adjustment, a "two-level adaptive" approach is used to integrate the weights into the EKF: ① The front-end quality evaluator calculates q_a and q_m based on the above indicators and obtains w_a and w_m; when q < 0.05, a threshold rejection is performed on the channel.

[0056] ② Based on innovation covariance matching, let the statistic for measuring innovation ν and covariance S be... ,according to Scaling measurement noise Ultimately used .

[0057] In one embodiment of this application, as Figure 5 As shown, the specific compensation method for dynamically compensating the initial images of the camera using pose data, namely S20 (performing pose compensation on multiple initial images of the target area captured by multiple cameras based on the pose data of the carrier, generating multiple images of the target area), specifically includes the following steps: S201: Calculate the attitude change of the carrier based on the attitude data of the carrier in two adjacent frames; S202: Transform the initial image of the current frame to the reference plane to obtain pixel drift; S203: Determine whether the initial image needs compensation based on pose changes and pixel drift; Specifically, S203 (determining whether the initial image needs compensation based on pose changes and pixel drift) includes the following steps: S2031: When the maximum value of the change in the three-axis rotational degrees of freedom during attitude change is greater than or equal to 0.1°, the initial image is determined to require compensation; or S2032: When the pixel drift is greater than or equal to 0.5, it is determined that the initial image needs to be compensated.

[0058] For example: when the pose change estimated by the IMU satisfies the following conditions within the frame exposure time Δt_exp or the synchronization error Δt_sync between the camera and other sensors. Compensation is triggered when the pixel drift caused by the reference plane is ≥0.5 pixels.

[0059] S204: When the initial image needs compensation, the initial image is compensated based on the camera intrinsic parameters, the extrinsic parameters of the camera and inertial sensor, the parameters of the reference plane, and the attitude change of the carrier.

[0060] Specifically, the geometric parameters that need to be compensated in the initial image are: The geometric parameters to be compensated are as follows: For global shutter cameras, apply the homography matrix H (reference plane distance d, normal n) derived from the pose increment (ΔR, Δt) or its pure rotational degeneracy H = KΔR K^{-1} for the entire frame; when the viewpoint change is small and only fast correction is needed, use the affine parameters (A, b). For rolling shutter cameras, use the row-dependent H(y) for each line y.

[0061] Specifically, when compensation is required, the necessary data includes camera intrinsic parameters K, camera-IMU extrinsic parameters T_CI, attitude change ΔR and displacement Δt, and reference plane parameters (n,d). (n,d) can be determined through one-time calibration (e.g., water surface height) or estimated online by depth / laser; a rolling shutter also requires a time-shift model to calculate H(y). After compensation, interpolation remapping and boundary clipping are performed, along with linear exposure compensation to ensure photometric continuity in the stitched area.

[0062] This application's real-time compensation system employs a prediction-correction processing mode. It utilizes the high-frequency sampling characteristics of inertial sensors (typically 100-1000Hz) to predict the carrier's attitude at high frequency. Then, it performs interpolation calculations based on the sampling times of the lidar and cameras to obtain precise attitude information at the data acquisition time of each camera. The compensation algorithm also considers the time delay differences between cameras, establishing a time synchronization compensation model to ensure the consistency of data from all sensors in the time dimension. This compensation mechanism enables the system to maintain stable detection performance in complex dynamic environments, significantly improving the accuracy and reliability of water target detection.

[0063] In one embodiment of this application, as Figure 6 As shown, the specific compensation method for compensating the initial point cloud data detected by the lidar using the carrier's attitude data, namely S30 (performing attitude compensation on the initial point cloud data of the target area monitored by the lidar based on the carrier's attitude data to generate point cloud data of the target area), specifically includes the following steps: S301: Calculate the angular velocity and acceleration of the carrier based on the three-axis rotational and three-axis translational degrees of freedom of the carrier during the scanning cycle, and calculate the angular change and displacement change of the carrier during the scanning cycle. S302: When the angle change value is greater than the preset angle change value or the displacement change value is greater than the preset displacement change value, the initial point cloud data detected by the lidar within the scanning cycle is determined as the target compensation point cloud data. Specifically, the method for determining the target point cloud data for compensation, that is, the process for determining the object that needs compensation, is as follows: Compensation is performed on initial point cloud data acquired within a single scan cycle that exhibits significant attitude / displacement changes in the carrier during the scan. This includes full-frame point clouds from mechanically rotating 3D LiDAR, column-by-column point clouds from solid-state scanning LiDAR, and any zigzag scan data carrying a timestamp t_i for each point. If an angle change |Δθ| ≥ 0.1° or a displacement change |Δs| ≥ 1 cm occurs within a single exposure cycle scan period Δt_scan, compensation is deemed necessary; otherwise, no compensation is required.

[0064] S303: Calculate the coordinate transformation matrix from the lidar coordinate system to the standard attitude coordinate system based on the extrinsic parameter matrix between the inertial sensor and the lidar. S304: Calculate the pose difference based on the radar coordinates and coordinate transformation matrix of the target compensation point cloud data in the lidar coordinate system. The pose difference includes rotation difference and translation difference. Specifically, the pose difference can be calculated as follows: for each initial point cloud data point with radar coordinates p_i^L (calculated from range and azimuth), the pose difference is calculated based on its timestamp t_i. It includes rotation ΔR_i and translation Δt_i. The unified coordinates after compensation are The compensation amount is the "reverse motion between t_i and t_0", which is uniquely determined by ΔT_i and does not require manual setting of the amplitude.

[0065] S305: Compensate the target point cloud data based on the pose difference.

[0066] Optionally, when there is a time difference between the inertial sensor and the lidar; In S303: Calculate the coordinate transformation matrix from the lidar coordinate system to the standard attitude coordinate system. Afterwards, the detection method also includes the following steps: S3031: By using the external parameter-time delay joint calibration method, it was determined that there is a time deviation between the inertial sensor and the lidar; S3032: Correct the coordinate transformation matrix based on time deviation.

[0067] Specifically, when there is a time deviation τ between the inertial sensor and the lidar, τ is first obtained through extrinsic parameter-time delay joint calibration, and then the time deviation is calculated using spline or linear interpolation. .

[0068] S304 is executed again based on the corrected coordinate transformation matrix, that is, the pose difference is calculated based on the radar coordinates of the target compensation point cloud data in the lidar coordinate system and the corrected coordinate transformation matrix. The pose difference includes rotation difference and translation difference.

[0069] Exemplary device As a second aspect of this application, this application also provides a water target detection system, comprising: Multiple cameras are used to capture initial images of the target area around the carrier; Multiple lidar sensors are used to detect point cloud data of the target area around the carrier. An inertial sensor, wherein the inertial sensor is used to detect the inertial data of the carrier; The detection controller includes a processor and a memory, the memory storing a computer program, and when the processor executes the computer program, it performs the above-described method for detecting water targets.

[0070] Specifically, such as Figures 2-4 As shown, the number of cameras and their mounting positions on the carrier, as well as the number of lidars and their mounting positions on the carrier, are as described above and will not be repeated here.

[0071] This application provides a water target detection system that uses real-time attitude information of a carrier detected by an inertial sensor to dynamically compensate for initial point cloud data detected by a lidar and initial images captured by a camera, generating compensated point cloud data and images. The compensated point cloud data and images are then fused to generate fused data, which is used to determine whether a target exists in the target area. By using real-time attitude information for dynamic compensation of the lidar and camera, the system effectively eliminates the impact of vibration and swaying of the camera and lidar in the water environment on data accuracy, ensuring the continuous accuracy of the spatiotemporal mapping between multiple cameras, multiple lidars, and between cameras and lidars, further improving the global positioning accuracy of the target.

[0072] Exemplary electronic devices Below, for reference Figure 7 This describes an electronic device according to embodiments of the present application.

[0073] Figure 7The figure shows a structural block diagram of an electronic device according to an embodiment of the present application.

[0074] like Figure 7 As shown, the electronic device 10 includes one or more processors 110 and memory 120.

[0075] The processor 110 may be a central processing unit (CPU) or other form of processing unit with data processing capabilities and / or instruction execution capabilities, and may control other components in the electronic device 10 to perform desired functions.

[0076] The memory 120 may include one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may include, for example, random access memory (RAM) and / or cache memory. The non-volatile memory may include, for example, read-only memory (ROM), hard disk, flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage medium, and the processor 110 may execute the program instructions to implement the water target detection methods of the various embodiments of this application described above, and / or other desired functions.

[0077] In one example, the electronic device 10 may also include an input device 130 and an output device 140, which are interconnected via a bus system and / or other forms of connection mechanism (not shown).

[0078] When the electronic device is a standalone device, the input device 130 can be a communication network connector for receiving the acquired input signals from the first device and the second device.

[0079] In addition, the input device 130 may also include, for example, a keyboard, a mouse, etc.

[0080] The output device 140 can output various information to the outside, including determined distance information, direction information, etc. The output device 140 may include, for example, a display, a speaker, a printer, and a communication network and its connected remote output devices, etc.

[0081] Of course, for the sake of simplicity, Figure 7 Only some of the components of the electronic device 10 relevant to this application are shown in this illustration; components such as buses, input / output interfaces, etc., are omitted. In addition, the electronic device 10 may include any other suitable components depending on the specific application.

[0082] This application provides a computer-readable storage medium storing a computer program for executing the above-described water target detection method and device. In addition, embodiments of this application may also be computer program products, including computer program information. When run by a processor, the computer program information causes the processor to execute the steps of the water target detection method according to various embodiments of this application as described in this specification.

[0083] Computer program products can be written in any combination of one or more programming languages ​​to perform the operations of the embodiments of this application. The programming languages ​​include object-oriented programming languages ​​such as Java and C++, as well as conventional procedural programming languages ​​such as C or similar languages. The program code can be executed entirely on the user's computing device, partially on the user's computing device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0084] Furthermore, embodiments of this application may also be computer-readable storage media storing computer program information thereon, which, when run by a processor, causes the processor to execute the steps in the water target detection method according to various embodiments of this application.

[0085] Computer-readable storage media may take the form of any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may, for example, include, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatuses, or devices, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: electrical connections having one or more wires, portable disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0086] The basic principles of this application have been described above with reference to specific embodiments. However, it should be noted that the advantages, benefits, and effects mentioned in this application are merely examples and not limitations, and should not be considered as essential features of each embodiment of this application. Furthermore, the specific details disclosed above are for illustrative and facilitative purposes only, and are not limitations. These details do not limit the application to the necessity of employing the aforementioned specific details for implementation.

[0087] The block diagrams of devices, apparatuses, devices, and systems involved in this application are merely illustrative examples and are not intended to require or imply that they must be connected, arranged, or configured in the manner shown in the block diagrams. As those skilled in the art will recognize, these devices, apparatuses, devices, and systems can be connected, arranged, and configured in any manner. Words such as “comprising,” “including,” “having,” etc., are open-ended terms meaning “including but not limited to,” and are used interchangeably with them. The terms “or” and “and” as used herein refer to the terms “and / or,” and are used interchangeably with them unless the context clearly indicates otherwise. The term “such as” as used herein refers to the phrase “such as but not limited to,” and is used interchangeably with it.

[0088] It should also be noted that in the apparatus, equipment, and methods of this application, the components or steps can be disassembled and / or recombined. These disassemblies and / or recombinations should be considered as equivalent solutions of this application.

Claims

1. A method for detecting targets in water areas, characterized in that, include: The inertial data of the carrier detected by the inertial sensors mounted on the carrier is processed to generate the attitude data of the carrier. The inertial sensors include a gyroscope, an accelerometer, and a magnetometer. The inertial data of the carrier includes angular velocity, acceleration, and magnetic field strength. The attitude data of the carrier includes the three-axis translational degrees of freedom, the three-axis rotational degrees of freedom, and the heading angle of the carrier. Based on the attitude data of the carrier, attitude compensation is performed on multiple initial images of the target area captured by multiple cameras to generate multiple images of the target area; Based on the attitude data of the carrier, attitude compensation is performed on the initial point cloud data of the target area detected by the lidar to generate the point cloud data of the target area; The presence of a target object in the target area is determined based on multiple images and the point cloud data.

2. The detection method according to claim 1, characterized in that, The step of processing the inertial data of the carrier detected by the inertial sensors mounted on the carrier to generate the attitude data of the carrier includes: Based on the angular velocity, acceleration, and magnetic field strength of the carrier, at least one of the following weights is adjusted: a first weight for the angular velocity, a second weight for the acceleration, and a third weight for the magnetic field strength. The angular velocity and its first weight, the acceleration and its second weight, and the magnetic field strength and its third weight are input into the extended Kalman filter for calculation to generate the attitude data of the carrier.

3. The detection method according to claim 2, characterized in that, The step of adjusting at least one of the first weight of the angular velocity, the second weight of the acceleration, and the third weight of the magnetic field strength based on the angular velocity, acceleration, and magnetic field strength of the carrier includes: The modulus deviation and acceleration change rate are calculated based on the acceleration of the carrier; and when the modulus deviation is greater than a first preset deviation, or the acceleration change rate is greater than a first preset change rate, or the angular velocity is greater than a first preset angular velocity, the second weight of the acceleration is reduced, and the reduced second weight is greater than 0.05 and less than 0.

2.

4. The detection method according to claim 3, characterized in that, The step of adjusting at least one of the first weight of the angular velocity, the second weight of the acceleration, and the third weight of the magnetic field strength based on the angular velocity, acceleration, and magnetic field strength of the carrier further includes: The magnetic field strength deviation is calculated based on the magnetic field strength; the magnetic field direction and heading angle are determined based on the magnetic field strength; the direction deviation is calculated based on the magnetic field direction and the geomagnetic angle; and the heading deviation is calculated based on the heading angle and the target heading angle. If the magnetic field strength deviation is greater than a first preset magnetic field deviation, or the direction deviation is greater than a preset direction deviation, or the heading deviation is greater than a first preset heading deviation, the third weight of the magnetic field strength is reduced; or When the magnetic field strength deviation is greater than the second preset magnetic field deviation or the heading deviation is greater than the second preset heading deviation, the third weight of the magnetic field strength is reduced to 0, wherein the first preset magnetic field deviation is less than the second preset magnetic field deviation and the first preset heading deviation is less than the second preset heading deviation.

5. The detection method according to claim 3, characterized in that, The step of adjusting at least one of the first weight of the angular velocity, the second weight of the acceleration, and the third weight of the magnetic field strength based on the angular velocity, acceleration, and magnetic field strength of the carrier further includes: When the angular velocity is less than the second preset angular velocity within a preset time period, and the magnitude deviation is less than the second preset deviation, and the acceleration change rate is less than the second preset change rate, the sum of the first weight and the third weight is adjusted to be greater than or equal to 0.8; wherein, the second preset angular velocity is less than the first preset angular velocity, the second preset deviation is less than the first preset deviation, and the second preset change rate is less than the first preset change rate.

6. The detection method according to claim 1, characterized in that, The step of performing pose compensation on multiple initial images of the target region captured by multiple cameras based on the pose data of the carrier to generate multiple images of the target region includes: The attitude change of the carrier is calculated based on the attitude data of the carrier in two adjacent frames. Transform the initial image of the current frame to the reference plane to obtain pixel drift; Based on the pose change and the pixel drift, determine whether the initial image needs compensation; When the initial image needs compensation, the initial image is compensated based on the camera's intrinsic parameters, the extrinsic parameters of the camera and the inertial sensor, the parameters of the reference plane, and the attitude change of the carrier.

7. The detection method according to claim 6, characterized in that, The step of determining whether the initial image needs compensation based on the pose change and the pixel drift includes: When the maximum value of the change in the three-axis rotational degrees of freedom during the attitude change is greater than or equal to 0.1°, it is determined that the initial image needs compensation; or When the pixel drift is greater than or equal to 0.5, it is determined that the initial image needs to be compensated.

8. The detection method according to claim 1, characterized in that, The step of performing attitude compensation on the initial point cloud data of the target area monitored by the lidar based on the attitude data of the carrier to generate point cloud data of the target area includes: The angular velocity and acceleration of the carrier are calculated based on the three-axis rotational degrees of freedom and three-axis translational degrees of freedom of the carrier during the scanning cycle, and the angle change and displacement change of the carrier during the scanning cycle are calculated respectively. When the angle change value is greater than the preset angle change value or the displacement change value is greater than the preset displacement change value, the initial point cloud data detected by the lidar within the scanning cycle is determined to be the target compensation point cloud data. Based on the extrinsic parameter matrix between the inertial sensor and the lidar, calculate the coordinate transformation matrix from the lidar coordinate system to the standard attitude coordinate system; Based on the target compensation point cloud data in the lidar coordinate system and the coordinate transformation matrix, the pose difference is calculated, which includes rotation difference and translation difference. The target compensation point cloud data is compensated based on the pose difference.

9. The detection method according to claim 8, characterized in that, When there is a time difference between the inertial sensor and the lidar; After calculating the coordinate transformation matrix from the lidar coordinate system to the standard attitude coordinate system, the detection method further includes: The time deviation between the inertial sensor and the lidar was determined by the external parameter-time delay joint calibration method. The coordinate transformation matrix is ​​corrected based on the time deviation.

10. A water target detection system, characterized in that, include: Multiple cameras are used to capture initial images of the target area around the carrier; Multiple lidar sensors are used to detect point cloud data of the target area around the carrier. An inertial sensor, wherein the inertial sensor is used to detect the inertial data of the carrier; A detection controller, comprising a processor and a memory, wherein the memory stores a computer program, and when the processor executes the computer program, it performs the water target detection method according to any one of claims 1-9.

Images