A multi-source joint sensing method and system for unmanned surface vessels

Abstract

This invention relates to the field of multi-source information joint sensing technology, and in particular to a method and system for multi-source information joint sensing of unmanned surface vessels (USVs). The invention includes acquiring target detection data of the USV's surface environment; performing coordinate transformation and point cloud coordinate correction on the target detection data to construct grid situation maps; segmenting targets and extracting contour features from the grid situation maps to obtain target contours; distinguishing between shore-based and surface targets based on the complexity and size of the target contours; performing many-to-one target matching and establishing target attributes using lidar and navigation radar; performing frame-by-frame target matching and updating the target state list; and performing many-to-many matching between the target and the target state list established by Kalman filtering, updating the target's attribute information. This invention can achieve complementary advantages between lidar and navigation radar, reducing target misses and loss, lowering false alarms, and improving target tracking accuracy and stability.

Description

Technical Field

[0001] This invention relates to the field of multi-source information joint perception technology, and in particular to a multi-source information joint perception method and system for unmanned surface vessels. Background Technology

[0002] Unmanned surface vessels (USVs) possess advantages such as strong maneuverability, convenient deployment, wide coverage, and low cost, enabling them to adapt to more dangerous working environments and demonstrating broad application prospects. The quality of environmental perception capability is a crucial indicator of an USV's autonomy. This is primarily achieved through sensors mounted on the unmanned intelligent equipment, acquiring information about the vessel's own state and environmental parameters, surrounding dynamics, and the relative distances, angles, and motion states of multi-scale targets. However, due to the diverse sources, multiple layers, complex forms, high timeliness, massive data volume, and uneven spatiotemporal scales of marine environmental information, achieving comprehensive and accurate environmental perception for USVs is one of the greatest challenges in realizing fully autonomous navigation.

[0003] Navigation radar can provide relative azimuth and distance information of surrounding obstacles over a large area, and has many functions such as sea clutter suppression, rain and snow suppression, and co-channel interference suppression. It is a standard navigation sensor for target tracking and collision avoidance. However, it can only reflect obstacle information in two-dimensional space, the target in the image is relatively coarse, and there is a certain blind zone at close range. In addition, the radar update frequency is slow, which makes it easy to miss or lose targets. Therefore, a standalone marine radar cannot accurately reflect the current environmental information. LiDAR, which has been successfully applied to autonomous vehicles, is suitable for short-range measurement. It has high accuracy and fast scanning frequency and can provide accurate azimuth and distance information of obstacles in the near-shore range. However, it has disadvantages such as being unable to detect obstructed targets, detecting only one side of the target, data loss due to the swaying of the target and the unmanned vessel, and inaccurate target segmentation. No single sensor can achieve complete and thorough perception of the environment. For joint perception by multiple sensors, it is necessary to solve how to extract useful information from the large amount of environmental data collected that has both overlap and complementarity, so as to correctly and reliably perceive the environment and realize the complementary advantages of different sensors. Therefore, there is an urgent need to provide a method and system for joint perception of multi-source information of unmanned surface vessels. Summary of the Invention

[0004] In view of the shortcomings of the prior art described above, the purpose of this invention is to provide a multi-source joint perception method and system for unmanned surface vessels, which can realize the complementary advantages of lidar and navigation radar, reduce target missed detection and loss, reduce false alarms, and improve target tracking accuracy and stability.

[0005] To achieve the above and other related objectives, this invention provides a method for joint perception of multi-source information of unmanned surface vessels, comprising:

[0006] S1. Use navigation radar and lidar to acquire target detection data of the surface environment of the unmanned surface vessel;

[0007] S2. Perform coordinate transformation and point cloud coordinate correction on the target detection data, and construct grid situation maps respectively;

[0008] S3. Perform target segmentation and contour feature extraction on the grid situation map to obtain the target contour;

[0009] S4. Differentiate between shore-based targets and surface targets by utilizing the complexity and size of the target outline;

[0010] S5. Every first preset time interval, i.e., the first preset number of scans by the lidar, the lidar and navigation radar complete the many-to-one target matching and target attribute establishment by using the minimum distance between the center point of the target contour and the target contour.

[0011] S6. Every second preset time interval, i.e., the navigation radar scans for the second preset number of cycles, the target is matched frame by frame and the target status list is updated based on Kalman filtering, so as to achieve accurate target tracking and calculation of motion elements.

[0012] S7. Every third preset time interval, use the features of the target in the current frame to complete a many-to-many match between the target and the target state list established by Kalman filtering, and update the target's attribute information.

[0013] In one embodiment of the present invention, the first preset time is 200ms and the first preset week number is two weeks; the second preset time is 1200ms and the second preset week number is one week; the third preset time is 200ms.

[0014] In one embodiment of the present invention, step S2, which involves performing coordinate transformation and point cloud coordinate correction on the target detection data and constructing a grid situation map, includes:

[0015] Step S2.1: The coordinate transformation is to convert the coordinates (d, θ) in the spherical coordinate system obtained by the real-time scanning of the lidar into coordinates (d, θ). vertical ,θ horizon Transform to Cartesian coordinates, where these represent the distance to the target point, vertical angle, and horizontal offset angle, respectively. The transformation formula is as follows:

[0016]

[0017] The point cloud coordinate correction addresses the discrepancy between the data obtained by the 3D lidar and the actual data due to changes in the unmanned surface vessel's heading, pitch, and roll. The correction utilizes the onboard integrated navigation system to acquire the unmanned surface vessel's motion attitude (α, β, θ) in real time, representing the roll angle, pitch angle, and heading angle, respectively. The correction formula is as follows:

[0018]

[0019] Where R is the attitude transformation matrix and T is the translation matrix;

[0020]

[0021] Step S2.2: The coordinate transformation involves converting the coordinates (d, γ) obtained from the real-time scanning of the navigation radar in the polar coordinate system to the Cartesian coordinate system, where d represents the distance to the target point and γ represents the horizontal angle. The transformation formula is as follows:

[0022]

[0023] Navigation radar image correction is performed based on the current heading angle of the radar image. The unmanned surface vessel's heading information θ is obtained in real-time using the integrated navigation system installed on the vessel, ensuring that the top of the radar image remains due north. The formula is:

[0024]

[0025] Step S2.3: The construction of the grid situation map involves dividing the map into a series of grid units in rows and columns according to the range of the navigation radar. The unmanned surface vessel (USV) remains at the center of the grid map. The grid resolution r is adjustable. The target points detected by the lidar and navigation radar are projected onto the grid map using the following conversion formula:

[0026]

[0027] Where (l, l) are the x and y coordinates of the center point of the raster image, and the length and width of the raster map are equal; r is the resolution of the raster map; (x, y) are the coordinates of the target in the Cartesian coordinate system; (a, b) are the x and y coordinates of the target point in the raster map. The gray value of the point at this coordinate in the raster image is set to 1, that is, 1 in the raster image indicates that there is a target at this location, and 0 indicates that there is no target at this location.

[0028] In one embodiment of the present invention, step S3, which involves target segmentation and contour feature extraction of the grid situation map to obtain the target contour, includes:

[0029] Step S3.1: Perform connected component segmentation on the target by calling the cv::findContours function in the OpenCV image processing library, extract the contour of each target in the image, and store all continuous contour points on the target boundary into the vector contours.

[0030] Step S3.2: Calculate the minimum bounding rectangle of the target contour using cv::minAreaRect from the OpenCV image processing library. Determine the coordinates of the target's center point, the target's length and width, and the angle and distance of the target relative to the unmanned surface vessel. The target's contour features are described as follows:

[0031] TarFeature={(x,y),(w,l),(dis,angle),contour}.

[0032] In one embodiment of the present invention, step S4, which distinguishes between shore-based targets and surface targets based on the complexity and size of the target outline, includes:

[0033] Step S4.1: Use the cv::approxPolyDP function in the OpenCV image processing library to perform polygon approximation on the target's contour point set. The approximation accuracy is epsilon, and the output is the number of polylines and the coordinates of the polylines approximated by the target contour point set. The more polylines there are, the more complex the target edge is, and vice versa.

[0034] Step S4.2: Calculate the area of ​​the target contour using the cv::contourArea function in the OpenCV image processing library or the length and width of the target contour features;

[0035] Step S4.3: Set the target area threshold and contour complexity threshold to divide the target into shore-based targets and water targets. The shore-based targets are identified as static targets and there is no need to calculate the target's motion attributes.

[0036] In one embodiment of the present invention, step S5, which involves performing many-to-one target matching and establishing target attributes between the lidar and navigation radar every first preset time interval (i.e., the lidar scans a first preset number of cycles), utilizes the minimum distance between the center point of the target contour and the target contour. This includes:

[0037] Step S5.1: Call the cv::pointPolygonTest function in the OpenCV image processing library to calculate the shortest distance between the center point of the lidar target and the outline of the navigation radar target, and use it as the matching standard to match the targets detected by the lidar and the navigation radar. Multiple lidar targets can correspond to one navigation radar target. The matching is updated once every two rotations of the lidar.

[0038] Step S5.2: Establish target attributes based on the matching results, which can be described as:

[0039] TarAttribute={ID,TarFeature,MatchID,Confidence,MotionEle,TarSource}

[0040] Wherein, ID is the target's number; TarFeature is the target contour feature calculated in step S3; MatchID is the number of the target in the Kalman filter's target state list, currently empty; Confidence is the target's confidence level, currently set to 0.3 for targets detected by navigation radar, 0.4 for targets detected by lidar, and 0.7 for targets successfully matched by lidar and navigation radar; MotionEle is the target's motion element, currently empty; TarSource is the target's source.

[0041] In one embodiment of the present invention, step S6, which involves performing frame-by-frame target matching and updating the target status list based on Kalman filtering every second preset time interval (i.e., the navigation radar scans a second preset number of cycles) to achieve accurate target tracking and calculation of motion elements, includes:

[0042] Step S6.1, the target state information in Kalman filtering is described as follows:

[0043] KAL_MEAN={x,y,angle,width,length,v x ,v y},

[0044] Among them, (v x ,v y Let (a, b) represent the target's speed on the x and y axes, where x and y are the coordinates of the target's center point. Since coordinates (a, b) are in a raster map coordinate system, a transformation is needed to adapt to the speed's coordinate system. The formula is as follows:

[0045]

[0046] In the Kalman filtering process, target states are set. When the number of unmatched times (age) is greater than MaxAge, the target is in a deleted state, meaning it is removed from the target state list. When the target is a new target (meaning it has no matching value in the target state list), it is in an uncertain state. When the number of consecutive matches (n_init) is greater than Init, the target is in a deterministic state. Cascaded matching is used in the target matching process, meaning targets with matching values ​​in the target state list are matched first. Hungarian matching is used as the matching method. The matching criteria are based on the distance between target center points, the target's length, width, and angle, as shown in the following formula:

[0047] dis=α*euclideanDistance+β*min(Diffwidth,Difflength)+λ*AngleDiff,

[0048] Where euclideanDistance is the Euclidean distance between target points; min(Diffwidth,Difflength) is the minimum value of the length and width difference of the target; AngleDiff is the angle difference of the target; α=0.8, β=0.1, λ=0.1 are the weight values ​​respectively;

[0049] Step S6.2: Based on the target state information (v) x ,v y ), (x,y) and the real-time speed, heading, and position of the unmanned surface vessel are used to calculate the speed, heading, distance, angle, latitude and longitude of the corresponding target;

[0050] Distance calculation formula:

[0051] Formula for calculating speed:

[0052] Heading calculation formula:

[0053] Angle calculation formula: angle=atan2((xx o )*r,(yy o )*r);

[0054] in, Let x be the velocity of the unmanned surface vessel on the x and y axes, (x... o ,y o ) represents the coordinates of the unmanned surface vessel; r represents the resolution of the raster map, and the constructed target state list is represented as:

[0055] TarState i ={id,KAL_MEAN,l,angle,v,head,lat,lon}.

[0056] In one embodiment of the present invention, in step S7, every third preset time interval, the many-to-many matching of the target with the target state list established by Kalman filtering is performed using the features of the target in the current frame, and the target attribute information is updated, including:

[0057] The matching of the current frame target with the target state list will be performed in a many-to-many manner to update the attributes of each target. The matching criterion is the minimum distance between the target center point and the target contour, which is calculated by calling the cv::pointPolygonTest function in the OpenCV image processing library. After a successful match, the MatchID, MotionEle, and Confidence attributes of the current frame target constructed in step S5 will be updated, and the confidence of the successfully matched target will increase.

[0058] This invention also provides a multi-source joint perception system for unmanned surface vessels, comprising:

[0059] The target detection data module is used to acquire target detection data of the surface environment of the unmanned surface vessel using navigation radar and lidar.

[0060] The grid situation map construction module is used to perform coordinate transformation and point cloud coordinate correction on the target detection data, and to construct grid situation maps respectively.

[0061] The extraction module is used to perform target segmentation and contour feature extraction on the grid situation map to obtain the target contour;

[0062] The differentiation module is used to distinguish between shore-based targets and surface targets based on the complexity and size of the target outline;

[0063] The target matching and target attribute establishment module is used to complete the many-to-one target matching and target attribute establishment of the lidar and navigation radar every first preset time interval, that is, the first preset number of times the lidar scans;

[0064] The calculation module is used to perform frame-by-frame matching of the target and update the target status list based on Kalman filtering every second preset time interval, i.e., the second preset number of navigation radar scans, so as to achieve accurate target tracking and calculation of motion elements.

[0065] The target attribute information module is used to perform a many-to-many match between the target and the target state list established by Kalman filtering every third preset time interval, using the features of the target in the current frame, and update the target attribute information.

[0066] The present invention also provides an electronic device, including a processor and a memory, wherein the memory stores program instructions, and the processor executes the program instructions to implement the above-described method for joint perception of multi-source information of unmanned surface vessels.

[0067] As described above, the multi-source joint perception method and system for unmanned surface vessels of the present invention has the following beneficial effects:

[0068] (1) The multi-source joint perception method for unmanned surface vessels of the present invention can achieve complementary advantages of lidar and navigation radar, reduce the missed detection and loss of targets, reduce false alarms of targets, and improve the tracking accuracy and stability of targets.

[0069] (2) The multi-source joint perception method of the unmanned surface vessel of the present invention can reduce the false detection rate of unmanned surface vessels for small targets at close range, and can improve the accuracy and stability of dynamic target motion elements and the integrity of target tracking chain.

[0070] (3) The multi-source joint perception method of unmanned surface vessels of the present invention can reduce the interference of wake waves and spray on the collision avoidance planning of unmanned surface vessels and adapt to high sea state and complex situations. Attached Figure Description

[0071] Figure 1 A flowchart illustrating a multi-source joint perception method for unmanned surface vessels provided in this application embodiment.

[0072] Figure 2 A schematic diagram of the three-dimensional lidar coordinate system used in a multi-source joint perception method for unmanned surface vessels provided in this application embodiment.

[0073] Figure 3 This is a schematic diagram of the navigation radar coordinate system transformation used in a multi-source joint perception method for unmanned surface vessels provided in an embodiment of this application.

[0074] Figure 4 This application provides a fusion situational awareness map of navigation radar and lidar for an unmanned surface vessel multi-information joint perception method.

[0075] Figure 5 This is a target motion element display diagram calculated by a multi-source joint perception method for unmanned surface vessels provided in an embodiment of this application. Detailed Implementation

[0076] The following specific examples illustrate the implementation of the present invention. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and various details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that, unless otherwise specified, the following embodiments and features described therein can be combined with each other.

[0077] It should be noted that the illustrations provided in the following embodiments are only schematic representations of the basic concept of the present invention. Therefore, the illustrations only show the components related to the present invention and are not drawn according to the actual number, shape and size of the components in the actual implementation. In the actual implementation, the form, quantity and proportion of each component can be arbitrarily changed, and the layout of the components may also be more complex.

[0078] Terms such as "first" or "second" may be used to describe various components, but these components are not limited by the terms described above. The terms described above are used to distinguish one component from another; for example, without departing from the scope of the concept according to this disclosure, a first component may be referred to as a second component, and similarly, a second component may be referred to as a first component.

[0079] Furthermore, "connected / linked" indicates that one component is directly electrically connected to another component or indirectly electrically connected through another component. Unless otherwise explicitly stated in the sentence, the singular form may include the plural form. Additionally, the terms "comprising / including" or "containing / including" as used in this specification indicate the presence or addition of one or more components, steps, operations, and elements. Specific structural or functional descriptions of examples of embodiments of the concepts disclosed in this specification are merely illustrative to describe examples of embodiments of the concepts, and examples of embodiments of the concepts can be implemented in various forms, but these descriptions are not limited to the examples of embodiments described in this specification.

[0080] Based on the concept, various modifications and changes can be applied to examples of embodiments, such that examples of embodiments will be illustrated in the accompanying drawings and described in the specification. However, examples of embodiments based on the concept are not limited to specific embodiments, but include all changes, equivalents, or substitutions included within the spirit and scope of this disclosure.

[0081] It should be understood that when describing an element as "connected" or "linked" to another element, the element may be directly connected or linked to the other element, or it may be connected or linked to the other element via a third element. Conversely, it should be understood that when an element is described as "directly connected to" or "directly linked to" another element, no other element is placed between them. Other expressions describing relationships between components (i.e., "between" and "directly between" or "adjacent to" and "directly adjacent to") need to be interpreted in the same way.

[0082] The terminology used in this specification is for the purpose of describing specific examples of implementations only and is not intended to limit this disclosure. The singular form may include the plural form unless there is an explicit contrary meaning in the context. It should be understood in this specification that the terms "comprising" or "having" indicate the presence of the features, quantities, steps, operations, components, parts, or combinations thereof described in the specification, but do not preclude the possibility of the presence or addition of one or more other features, quantities, steps, operations, components, parts, or combinations thereof.

[0083] Unless otherwise defined, all terms used herein (including technical or scientific terms) shall have the same meaning as commonly understood by one of ordinary skill in the art. If a term is not clearly defined in a common dictionary in this specification, it shall be interpreted as having the same meaning as in the context of the relevant art, and not as an ideal or overly formal meaning.

[0084] Descriptions of known components and processing techniques may be omitted to avoid unnecessarily obscuring the embodiments of this disclosure.

[0085] Throughout this specification, the same reference numerals refer to the same elements. Therefore, even if a reference numeral is not mentioned or described with reference to one drawing, it may be mentioned or described with reference to another drawing. Furthermore, even if a reference numeral is not shown in one drawing, it may be mentioned or described with reference to another drawing.

[0086] Additionally, the logic level of a signal may be different from or opposite to the logic level described. For example, a signal described as having a logic "high" level may optionally have a logic "low" level, and a signal described as having a logic "low" level may optionally have a logic "high" level.

[0087] The embodiments of this disclosure will now be described in detail with reference to the accompanying drawings. However, those skilled in the art will understand that many technical details have been provided in the embodiments of this disclosure to facilitate a better understanding of the disclosure. However, the technical solutions claimed in this disclosure can be implemented even without these technical details and various variations and modifications based on the following embodiments.

[0088] Please see Figure 1 , Figure 1 This document provides a flowchart of a multi-source joint perception method for unmanned surface vessels, as illustrated in an embodiment of this application. The present invention provides a multi-source joint perception method for unmanned surface vessels, which can achieve complementary advantages of lidar and navigation radar, reduce target missed detections and loss, decrease false alarms, and improve target tracking accuracy and stability. The method includes:

[0089] Step S1: Use navigation radar and lidar to acquire target detection data of the unmanned surface vessel's water environment;

[0090] Step S2: Perform coordinate transformation and point cloud coordinate correction on the obtained target detection data, and construct grid situation maps respectively;

[0091] Step S3: Perform target segmentation and contour feature extraction on the constructed grid situation map to obtain the target contour;

[0092] Step S4: Differentiate between shore-based targets and surface targets by the complexity and size of the target outline. Shore-based targets are static targets, and there is no need to calculate the target's motion elements.

[0093] Step S5: Every 200ms, that is, after the lidar scans twice, the lidar and navigation radar complete the many-to-one target matching and target attribute establishment by using the minimum distance between the center point of the target contour and the target contour.

[0094] Step S6: Every 1200ms, that is, the navigation radar scans once, the frame-by-frame matching of the target and the update of the target status list are completed based on Kalman filtering, so as to achieve accurate target tracking and calculation of motion elements;

[0095] Step S7: Every 200ms, use the features of the target in the current frame to complete a many-to-many match between the target and the target state list established by Kalman filtering, and update the target's attribute information.

[0096] Specifically, step S2, which involves performing coordinate transformation and point cloud coordinate correction on the obtained target detection data and constructing a grid situation map, includes:

[0097] Step S2.1: The coordinate transformation is to convert the spherical coordinates (d, θ) of the three-dimensional lidar... vertical ,θ horizon To convert to the Cartesian coordinate system, the conversion formula is as follows:

[0098]

[0099] The point cloud coordinate correction utilizes the integrated navigation system installed on the vessel to obtain the unmanned surface vessel's rolling angle, pitch angle, and heading angle (α, β, θ) in real time. The correction formula is as follows:

[0100]

[0101] Where R is the attitude transformation matrix and T is the translation matrix;

[0102]

[0103] Step S2.2: The coordinate transformation involves converting the coordinates (d, γ) obtained from the real-time scanning of the navigation radar in the polar coordinate system to the Cartesian coordinate system. The transformation formula is as follows:

[0104]

[0105] Navigation radar image correction utilizes the onboard integrated navigation system to acquire the unmanned surface vessel's heading information θ in real time, ensuring that the top of the radar image remains due north. The formula is:

[0106]

[0107] Step S2.3: The construction of the grid situation map involves dividing the map into a series of grid cells in rows and columns based on the range of the navigation radar. The unmanned surface vessel (USV) remains at the center of the grid map, and the target points detected by the lidar and navigation radar are projected onto the grid map. The conversion formula is as follows:

[0108]

[0109] Where (l, l) are the x and y coordinates of the center point of the raster image, and the length and width of the raster map are equal; r is the resolution of the raster map; (x, y) are the coordinates of the target in the Cartesian coordinate system; (a, b) are the x and y coordinates of the target point in the raster map. The gray value of the point at this coordinate in the raster image is set to 1, that is, 1 in the raster image indicates that there is a target at this location, and 0 indicates that there is no target at this location.

[0110] Specifically, step S3, which involves target segmentation and contour feature extraction of the grid situation map to obtain the target contour, includes:

[0111] Step S3.1: The edge information of the target contour in the image contains most of the information of the connected components corresponding to the target. This invention performs connected component segmentation on the target by calling the cv::findContours function in the OpenCV image processing library, extracts the contour of each target in the image, and stores all continuous contour points on the target boundary into the vector contours.

[0112] Step S3.2: Calculate the minimum bounding rectangle of the target contour using cv::minAreaRect from the OpenCV image processing library, thereby determining the coordinates of the target's center point, the target's length and width, and the angle and distance of the target relative to the unmanned surface vessel. The contour features of the target can be described as follows:

[0113] TarFeature={(x,y),(w,l),(dis,angle),contour},

[0114] Specifically, step S4, which distinguishes between shore-based targets and surface targets based on the complexity and size of the target outline, includes:

[0115] Step S4.1: There are many targets in the radar image. It is necessary to classify the targets to extract the targets of interest. On the sea surface, large targets are mostly large islands, coastlines and bridges, while small targets are mostly ships and buoys. The cv::approxPolyDP function in the OpenCV image processing library is used to perform polygon approximation on the target's contour point set. The approximation accuracy is epsilon. The output is the number of polylines and the coordinates of the polylines approximated by the target contour point set. The more polylines, the more complex the target edge, and vice versa.

[0116] Step S4.2: Calculate the area of ​​the target contour using the cv::contourArea function in the OpenCV image processing library or the length and width of the target contour features.

[0117] Step S4.3: Based on the understanding of the detection principle and image construction of navigation radar, it is found that the target shapes of obstacles such as ships and buoys on the water surface are mostly elliptical in the navigation radar image, while shore-based targets are mostly large in area and have complex outline edges. Based on this understanding and the working environment of the unmanned surface vessel, the ratio of the navigation radar image to the actual size of the object, target area threshold and outline complexity threshold are set to classify the target into shore-based targets and water targets. Shore-based targets are identified as static targets, and there is no need to calculate the target's motion attributes.

[0118] Specifically, step S5, which utilizes the minimum distance between the center point of the target contour and the target contour to complete the many-to-one target matching and target attribute establishment for both the lidar and navigation radar, includes:

[0119] Step S5.1: Due to the detection principle of lidar, it has disadvantages such as short detection range, inability to detect obstructed targets, detection data only for one side of the target, data loss due to target and UAV swaying, and inaccurate target segmentation. Navigation radar, due to its slow clutter suppression at close range and slow scanning frequency, has disadvantages such as discontinuity, loss, or interference from sea clutter for small targets at close range. In order to increase the accuracy and stability of target detection, the cv::pointPolygonTest function in the OpenCV image processing library is called to calculate the shortest distance between the center point of the lidar target and the outline of the navigation radar target, and this distance is used as the matching standard to match the targets detected by lidar and navigation radar. Multiple lidar targets can correspond to one navigation radar target, and the matching is updated every two rotations of the lidar.

[0120] Step S5.2: Establish target attributes based on the matching results, which can be described as:

[0121] TarAttribute={ID,TarFeature,MatchID,Confidence,MotionEle,TarSource},

[0122] In the formula, ID is the target number; TarFeature is the target contour feature calculated in step S3; MatchID is the number of the target in the target state list in the Kalman filter, which is currently empty; Confidence is the target confidence level, currently set to 0.3 for targets detected by navigation radar, 0.4 for targets detected by lidar, and 0.7 for targets successfully matched by lidar and navigation radar; MotionEle is the target motion element, which is currently empty; TarSource is the target source.

[0123] Specifically, step S6, which involves frame-by-frame target matching and target state list updates based on Kalman filtering to achieve accurate target tracking and motion element calculation, includes:

[0124] Step S6.1: Due to the short interval between radar image acquisitions, the change in the position of each target is relatively small within a short time. Therefore, based on this characteristic, one-to-one matching of targets in continuous images can be performed. Based on Kalman filtering, the motion elements of moving targets can be estimated recursively from noisy measurement signals. Furthermore, to address the occasional loss of targets in radar images, Kalman filtering can predict the target's position information within a short time, establishing a complete target tracking chain. However, due to the motion of the unmanned surface vessel and the drift and instability of the radar image causing the target's center point to shift, this invention performs matching tracking every 1200ms to increase the stability of target motion element prediction. The target state information in the Kalman filter is described as follows:

[0125] KAL_MEAN={x,y,angle,width,length,v x ,v y},

[0126] Among them, (v x ,v y (a, b) represents the target's speed on the x and y axes, where x and y are the coordinates of the target's center point, respectively. Since the coordinates (a, b) in step S3 are in the raster map coordinate system, a conversion is needed to adapt to the speed coordinate system. The formula is as follows:

[0127]

[0128] In the Kalman filtering process, target states are set. When the number of unmatched times (age) of a target is greater than MaxAge, the target is in a deleted state, meaning it is removed from the target state list. When the target is a new target (meaning it has no matching value in the target state list), the target is in an uncertain state. When the number of consecutive matches (n_init) is greater than Init, the target is in a determined state. Cascaded matching is used in the target matching process, prioritizing matches for targets with matching values ​​in the target state list. The matching method uses the Hungarian matching algorithm to complete as many matches as possible. The matching criteria are based on the distance between the target center points, the target's length, width, and angle, as shown in the following formula:

[0129] dis=α*euclideanDistance+β*min(Diffwidth,Difflength)+λ*AngleDiff,

[0130] Where euclideanDistance is the Euclidean distance between target points; min(Diffwidth,Difflength) is the minimum value of the length and width difference of the target; AngleDiff is the angle difference of the target; α=0.8, β=0.1, λ=0.1 are the weight values ​​respectively;

[0131] Step S6.2: Based on the target state information (v x ,v y ), (x,y) and the real-time speed, heading, and position of the unmanned surface vessel are used to calculate the speed, heading, distance, angle, and latitude and longitude of the corresponding target; the distance calculation formula is: Formula for calculating speed:

[0132] Heading calculation formula: Angle calculation formula: angle=atan2((xx o )*r,(yy o )*r); where Let x be the velocity of the unmanned surface vessel on the x and y axes, (x... o ,y o ) represents the coordinates of the unmanned surface vessel; r represents the resolution of the raster map. The constructed target state list is represented as:

[0133] TarState i ={id,KAL_MEAN,l,angle,v,head,lat,lon}

[0134] Specifically, step S7, which involves using the features of the target in the current frame to perform many-to-many matching between the target and the target state list established by Kalman filtering, and updating the target's attribute information, specifically includes:

[0135] Step S7.1: To meet the real-time requirements of unmanned surface vessel (USV) perception, this invention updates target attributes every 200ms. Since radar images may divide a single target into multiple targets, this invention uses a many-to-many matching method between the current frame target and the target state list to update the attributes of each target. The matching criterion is the minimum distance between the target center point and the target outline, calculated by calling the cv::pointPolygonTest function in the OpenCV image processing library. After a successful match, the MatchID, MotionEle, and Confidence attributes of the current frame target constructed in step S5 will be updated, and the confidence of the successfully matched target will increase.

[0136] Please see Figures 2-5 , Figure 2 A schematic diagram of the three-dimensional lidar coordinate system used in a multi-source joint perception method for unmanned surface vessels provided in this application embodiment. Figure 3 This is a schematic diagram of the navigation radar coordinate system transformation used in a multi-source joint perception method for unmanned surface vessels provided in an embodiment of this application. Figure 4 This application provides a fusion situational awareness map of navigation radar and lidar for an unmanned surface vessel multi-information joint perception method. Figure 5 This application provides a target motion element display diagram for a multi-source joint perception method for unmanned surface vessels, as illustrated in an embodiment of the present application. In one embodiment, the present invention provides a multi-source joint perception method for unmanned surface vessels, specifically comprising:

[0137] Step S1. Use navigation radar and lidar to acquire target detection data of the unmanned surface vessel's water environment.

[0138] In this example, an RS80 model lidar and a GYRadar model navigation radar are used. The radars are horizontally mounted on the top of the unmanned surface vessel in an unobstructed position. The lidar rotates and scans at a constant speed of 10Hz, and the navigation radar rotates and scans at a speed of 48rad / min. The scan data is parsed according to the radar's network communication protocol.

[0139] Step S2. Perform coordinate transformation and correction on the obtained data, and construct grid situation maps respectively.

[0140] Step S2.1: Coordinate transformation, transform the coordinates obtained in step S1 from the radar coordinate system to the Cartesian coordinate system (x, y, z).

[0141] Step S2.2: Correction. The data obtained in step S2.1 is corrected using the real-time motion attitude (α,β,θ) of the unmanned surface vessel (USV) output by the integrated navigation equipment installed on the USV. During the correction process, due to the large amount of radar data, a matrix library is used for parallel computation.

[0142] Step S2.3: Situational grid map construction. First, initialize matrix Mat according to the navigation radar's range. The matrix's length and width are the navigation radar's detection range divided by the grid map's resolution, with the unmanned surface vessel (USV) at the center of the grid map. Then, project the target points detected by the calibrated lidar and navigation radar onto the grid map, filling matrix Mat. The lidar grid map is constructed from two frames of lidar data. A schematic diagram of the lidar spherical coordinate system is shown below. Figure 2 As shown, the coordinate system of the navigation radar and the coordinate system of the grid map are as follows: Figure 3 As shown.

[0143] Step S3. Perform target segmentation and contour feature extraction on the constructed situation map.

[0144] Step S3.1: Process the binary raster map by calling the cv::findContours function in the OpenCV image processing library, complete the connected component segmentation of the target, extract the contour of each target in the image, and store all continuous contour points on the target boundary into the vector contours.

[0145] Step S3.2: Calculate the minimum bounding rectangle of the target contour contours using cv::minAreaRect in the OpenCV image processing library, and then determine the coordinates of the target's center point, the target's length and width, and the angle and distance of the target relative to the unmanned surface vessel.

[0146] Step S4. Use the complexity and size of the target outline to distinguish between shore-based targets and surface targets.

[0147] Step S4.1: Use the cv::approxPolyDP function in the OpenCV image processing library to perform polygon approximation on the contour point set contours of the target. The approximation accuracy is epsilon, and the output is the number of polylines Conc approximated by the target contour point set.

[0148] Step S4.2: Calculate the area TarArea of ​​the target contour using the cv::contourArea function in the OpenCV image processing library or the length and width of the target contour features.

[0149] Step S4.3: Set the target area threshold AreaTh and the contour complexity threshold CTh, and designate targets whose target area is greater than the threshold and whose target contour complexity is greater than the threshold as shore-based targets.

[0150] Step S5. Every 200ms, use the minimum distance between the center point of the target contour and the target contour to complete the many-to-one target matching and target attribute establishment of the lidar and navigation radar.

[0151] Step S5.1: Call the cv::pointPolygonTest function in the OpenCV image processing library to calculate the shortest distance between the center point of the LiDAR target and the outline of the navigation radar target, and use it as the matching standard to match the targets detected by the LiDAR and the navigation radar. Multiple LiDAR targets can correspond to one navigation radar target. The matching is updated once every two rotations of the LiDAR.

[0152] Step S5.2: Establish target attributes based on the matching results, which can be described as:

[0153] TarAttribute={ID,TarFeature,MatchID,Confidence,MotionEle,TarSource}

[0154] Step S6. Every 1200ms, the target is matched frame by frame and the target state list is updated based on Kalman filtering, so as to achieve accurate target tracking and calculation of motion elements;

[0155] Step S6.1: The target state information in the Kalman filter is described as follows:

[0156] KAL_MEAN={x,y,angle,width,length,v x ,v y},

[0157] In the Kalman filtering process, target states are set. When the number of unmatched times (age) of a target is greater than MaxAge, the target is in a deleted state, meaning it is removed from the target state list. When the target is a new target (meaning it has no matching value in the target state list), the target is in an uncertain state. When the number of consecutive matches (n_init) is greater than Init, the target is in a determined state. Cascaded matching is used in the target matching process, prioritizing matches for targets with matching values ​​in the target state list. The matching method uses the Hungarian matching algorithm to complete as many matches as possible. The matching criteria are based on the distance between the target center points, the target's length, width, and angle, as shown in the following formula:

[0158] dis=α*euclideanDistance+β*min(Diffwidth,Difflength)+λ*AngleDiff,

[0159] Step S6.2: Based on the target state information (v x ,v y The real-time speed, heading, and position of (x, y) and the unmanned surface vessel are used to calculate the speed, heading, distance, angle, and latitude and longitude of the corresponding target; the constructed target state list is represented as follows:

[0160] TarState i ={id,KAL_MEAN,l,angle,v,head,lat,lon},

[0161] Step S7. Every 200ms, use the features of the target in the current frame to complete a many-to-many match between the target and the target state list established by Kalman filtering, and update the target's attribute information.

[0162] Step S7.1: The matching of the current frame target with the target state list adopts a many-to-many approach to complete the attribute update of each target; the matching criterion is the minimum distance between the target center point and the target contour, which is calculated by calling the cv::pointPolygonTest function in the OpenCV image processing library; after a successful match, the MatchID, MotionEle, and Confidence in the current frame target attributes constructed in step S5 will be updated, and the confidence of the successfully matched target will increase by 0.3.

[0163] like Figure 4 , Figure 5 As described above, the red targets in the figure are shore-based targets, the blue targets are navigation radar targets, and the purple targets are fusion targets of navigation radar and lidar. For moving targets, the target batch number, heading, and speed are marked, and the red arrows indicate the direction of movement of the target. For static targets, the target batch number is marked. The center point in the figure is the position of the unmanned surface vessel, and the red arrows indicate the heading of the unmanned surface vessel.

[0164] Similar to the principle of the multi-source joint perception method for unmanned surface vessels of the present invention, the present invention also provides a multi-source joint perception system for unmanned surface vessels, comprising:

[0165] The target detection data module is used to acquire target detection data of the surface environment of the unmanned surface vessel using navigation radar and lidar.

[0166] The grid situation map construction module is used to perform coordinate transformation and point cloud coordinate correction on the target detection data, and to construct grid situation maps respectively.

[0167] The extraction module is used to perform target segmentation and contour feature extraction on the grid situation map to obtain the target contour;

[0168] The differentiation module is used to distinguish between shore-based targets and surface targets based on the complexity and size of the target outline;

[0169] The target matching and target attribute establishment module is used to complete the many-to-one target matching and target attribute establishment of the lidar and navigation radar every first preset time interval, that is, the first preset number of times the lidar scans;

[0170] The calculation module is used to perform frame-by-frame matching of the target and update the target status list based on Kalman filtering every second preset time interval, i.e., the second preset number of navigation radar scans, so as to achieve accurate target tracking and calculation of motion elements.

[0171] The target attribute information module is used to perform a many-to-many match between the target and the target state list established by Kalman filtering every third preset time interval, using the features of the target in the current frame, and update the target attribute information.

[0172] This invention also proposes an electronic device comprising a processor and a memory. The memory stores program instructions, and the processor executes these program instructions to implement the aforementioned method for joint perception of multi-dimensional information from unmanned surface vessels. The processor can be a general-purpose processor, including a Central Processing Unit (CPU), a Network Processor (NP), etc.; it can also be a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. The memory may include Random Access Memory (RAM) or Non-Volatile Memory, such as at least one disk storage device. The memory can also be an internal memory of the Random Access Memory (RAM) type. The processor and memory can be integrated into one or more independent circuits or hardware, such as an Application Specific Integrated Circuit (ASIC). It should be noted that when the computer program in the aforementioned memory is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, electronic device, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of the present invention.

[0173] This invention also proposes a computer-readable storage medium storing computer instructions for instructing the computer to execute the aforementioned multi-source joint sensing method for unmanned surface vessels. The computer-readable storage medium can be an electronic medium, magnetic medium, optical medium, electromagnetic medium, infrared medium, or semiconductor system or propagation medium. The computer-readable storage medium may also include semiconductor or solid-state memory, magnetic tape, removable computer disk, random access memory (RAM), read-only memory (ROM), hard disk, and optical disc. Optical discs may include optical disc-read-only memory (CD-ROM), optical disc-read / write (CD-RW), and DVD.

[0174] In summary, the multi-source joint perception method for unmanned surface vessels of the present invention can achieve complementary advantages of lidar and navigation radar, reduce target missed detection and loss, reduce false alarms, and improve target tracking accuracy and stability.

[0175] The above embodiments are merely illustrative of the principles and effects of the present invention and are not intended to limit the invention. Any person skilled in the art can modify or alter the above embodiments without departing from the spirit and scope of the present invention. Therefore, all equivalent modifications or alterations made by those skilled in the art without departing from the spirit and technical concept disclosed in the present invention should still be covered by the claims of the present invention.

Claims

1. A method for multi-source information joint perception of an unmanned surface vehicle, characterized in that, The method comprises the following steps: S1, acquiring target detection data of the water surface environment of the unmanned surface vehicle by using a navigation radar and a laser radar; S2, performing coordinate conversion and point cloud coordinate correction on the target detection data, and constructing a grid situation map respectively; S3, performing target segmentation and contour feature extraction on the grid situation map to obtain a target contour; S4, distinguishing a shore-based target and a water target according to the complexity and area size of the target contour; S5, every first preset time, i.e. the laser radar scans a first preset number of circles, a multi-to-one target matching of the laser radar and the navigation radar and establishment of target attributes are completed by using the minimum distance between the center point of the target contour and the target contour; S6, every second preset time, i.e. the navigation radar scans a second preset number of circles, frame-by-frame matching of the target and updating of a target state list are completed based on Kalman filtering, and accurate tracking of the target and calculation of motion elements are realized; S7, every third preset time, multi-to-multi matching of the target and the target state list established by Kalman filtering is completed by using the features of the current frame target, and the attribute information of the target is updated; The step S5 in the method comprises the following steps: S5.1, a function cv: pointPolygonTest in an Opencv image processing library is called to calculate the nearest distance between the center point of the laser radar target and the target contour of the navigation radar, and the nearest distance is used as a matching standard to match the targets detected by the laser radar and the navigation radar, a plurality of laser radar targets can correspond to one navigation radar target, and the laser radar updates the matching once every two rotations; S5.2, target attributes are established based on the matching result, which can be expressed as: Wherein, ID is the number of the target; TarFeature is the target contour feature calculated in the step S3; MatchID is the number of the target corresponding to the target state list in the Kalman filtering, which is currently empty; Confidence is the confidence of the target, the confidence of the target detected by the navigation radar is currently set to 0.3, the confidence of the target detected by the laser radar is set to 0.4, and the confidence of the target matched successfully by the laser radar and the navigation radar is modified to 0.7; MotionEle is the motion element of the target, which is currently empty; TarSource is the source of the target.

2. The unmanned surface vehicle multi-source information joint perception method according to claim 1, characterized in that: The first preset time is 200 ms, the first preset number of circles is two, the second preset time is 1200 ms, and the second preset number of circles is one; The third preset time is 200 ms.

3. The unmanned surface vehicle multi-source information joint perception method according to claim 1, characterized in that, The step S2 in the method comprises the following steps: Step S2.1, the coordinate conversion is to convert the coordinates in the spherical coordinate system obtained by the real-time scanning of the laser radar Convert to the Cartesian coordinate system, which are the distance, vertical angle and horizontal offset angle of the target point respectively, and the conversion formula is as follows: ， The point cloud coordinate correction is to correct the deviation between the data obtained by the three-dimensional laser radar and the real data with the change of the unmanned ship's heading, pitch and roll The roll angle, pitch angle and heading angle of the unmanned ship, and the correction formula is as follows: ， Wherein, R is a posture transformation matrix, and T is a translation matrix; ， Step S2.2, the coordinate conversion is to convert the polar coordinate system under the coordinate acquired by the navigation radar real-time scanning converted into the Cartesian coordinate system, which are the distance and horizontal angle of the target point respectively, and the conversion formula is as follows: ， The navigation radar image is corrected according to the heading angle at the current time, the heading information θ of the unmanned vehicle is acquired in real time by using the combined navigation installed on the vehicle, the top of the radar image is kept as the north direction, and the formula is as follows: ， Step S2.3, the grid situation map is constructed by dividing the map into a series of grid cells according to the range of the navigation radar, the unmanned ship is always the center of the grid map, the resolution r of the grid is adjustable, the target points detected by the laser radar and the navigation radar are projected into the grid map, and the conversion formula is as follows: ， wherein, is the horizontal and vertical coordinates of the center point of the grid image, the length and width of the grid map are equal; is the resolution of the grid map; is the coordinates of the target in the Cartesian coordinate system; is the horizontal and vertical coordinates of the target point in the grid map, and the gray value of the coordinate point in the grid image is set to 1, that is, 1 in the grid image indicates that there is a target at this position, and 0 indicates that there is no target at this position.

4. The unmanned surface vehicle multi-source information joint perception method according to claim 3, characterized in that, The target segmentation and contour feature extraction of the grid situation map in step S3 include: Step S3.1, the target is segmented by calling the cv: findContours function in the Opencv image processing library, and the contour of each target in the image is extracted, and all continuous contour points on the target boundary are stored in the vector contours; Step S3.2, the minimum circumscribed rectangle of the target contour is calculated by the cv: minAreaRect in the Opencv image processing library, the center point coordinates of the target, the length and width of the target, the angle and distance of the target relative to the unmanned ship are determined, and the contour feature of the target is expressed as: 。 5. The unmanned surface vehicle multi-source information joint perception method according to claim 4, characterized in that, The step S4 of distinguishing the shore-based target and the water target by using the complexity and area size of the target contour includes: Step S4.1, the polygon approximation of the target contour point set is performed by the cv: approxPolyDP function in the Opencv image processing library, the approximation precision value is epsilon, and the output is the number of polyline and the polyline coordinates of the target contour point set approximation, the more the number of polyline, the more complex the target edge, and vice versa, the simpler the target edge; Step S4.2, the area of the target contour is calculated by the cv: contourArea function in the Opencv image processing library or the length and width in the target contour feature; Step S4.3, the target area threshold and the contour complexity threshold are set, the target is divided into the shore-based target and the water target, the shore-based target is confirmed as a static target, and the motion attribute of the target does not need to be calculated.

6. The unmanned surface vehicle multi-source information joint perception method according to claim 1, characterized in that, The step S6 includes that every second preset time, that is, the navigation radar scans the second preset number of times, the frame-by-frame matching of the target and the update of the target state list are completed based on the Kalman filter, and the accurate tracking of the target and the calculation of the motion elements include: Step S6.1, the target state information in the Kalman filter is described as: ， wherein, is the target speed on the x, y axis, x, y are the coordinates of the center point of the target, since the coordinates (a, b) are the grid map coordinates, in order to adapt to the coordinate system of the speed, conversion is needed, the formula is as follows: ， In the Kalman filter process, the state of the target is set, when the number of target matching times age is greater than MaxAge, the target is in the deletion state, that is, the target state list is deleted; when the target is a new target, that is, there is no matching value in the target state list, the target is in the uncertain state; when the number of continuous matching times n_init is greater than Init, the target is in the determined state, and the target matching process adopts cascade matching, that is, the targets with matching values in the target state list are matched preferentially; the matching method adopts the Hungarian matching; the matching standard is composed of the distance between the target center points, the length and width of the target, and the angle, and the formula is as follows: ， wherein, is the Euclidean distance between the target points; is the minimum value of the target length-width difference; is the angle difference of the target; are weight values, respectively; Step S6.2, according to the target state information in the target state information , and the real-time speed, heading, position of the unmanned ship, the speed, heading, distance, angle, latitude and longitude of the corresponding target are calculated. Distance calculation formula: ; Speed calculation formula: ; Course calculation formula: ; Angle calculation formula: ; wherein, is the velocity of the USV in the x, y axes, is the coordinate point of the USV; r is the resolution of the grid map, and the constructed target state list is represented as: 。 7. The unmanned surface vehicle multi-source information joint perception method according to claim 6, characterized in that, The step S7 includes that every third preset time, the multi-to-multi matching of the target and the target state list established by the Kalman filter is completed by using the characteristics of the current frame target, and the attribute information of the target is updated, including: The matching of the current frame target with the target state list adopts multiple-to-multiple, to complete attribute updating of each target; the matching standard adopts the minimum distance between the target center point and the target contour, which is calculated by calling the cv: pointPolygonTest function in the Opencv image processing library; after successful matching, the MatchID, MotionEle and Confidence in the target attribute of the current frame constructed in step S5 are updated, and the confidence of the successfully matched target is increased.

8. An unmanned surface vehicle multi-element information joint perception system, characterized in that, The method comprises the following steps: a target detection data module, configured to acquire target detection data of a water surface environment of the unmanned surface vehicle by using a navigation radar and a laser radar; a grid situation map construction module, configured to perform coordinate conversion and point cloud coordinate correction on the target detection data, and construct a grid situation map; an extraction module, configured to perform target segmentation and contour feature extraction on the grid situation map to obtain a target contour; a distinguishing module, configured to distinguish a shore-based target and a water-based target according to the complexity and area size of the target contour; a target matching and target attribute establishment module, configured to perform multiple-to-one target matching and target attribute establishment of the laser radar and the navigation radar by using the minimum distance between the center point of the target contour and the target contour every first preset time, i.e., the laser radar scans a first preset number of times; a calculation module, configured to perform frame-by-frame matching of the target and update a target state list based on Kalman filtering every second preset time, i.e., the navigation radar scans a second preset number of times, to realize accurate tracking of the target and calculation of motion elements; a target attribute information module, configured to perform multiple-to-multiple matching of the target and the target state list established by Kalman filtering by using the features of the current frame target every third preset time, and update the attribute information of the target; the target matching and target attribute establishment module is further configured to call the cv: pointPolygonTest function in the Opencv image processing library to calculate the nearest distance between the center point of the laser radar target and the target contour of the navigation radar, and use the nearest distance as a matching standard to match the targets detected by the laser radar and the navigation radar; one navigation radar target can correspond to multiple laser radar targets, and the laser radar updates the matching every two rotations; the target matching and target attribute establishment module is further configured to establish target attributes based on the matching result, which can be expressed as: wherein, ID is the number of the target; TarFeature is the target contour feature calculated in step S3; MatchID is the number of the target corresponding to the target state list in Kalman filtering, which is currently empty; Confidence is the confidence of the target, the confidence of the target detected by the navigation radar is currently set to 0.3, the confidence of the target detected by the laser radar is set to 0.4, and the confidence of the target matched successfully by the laser radar and the navigation radar is modified to 0.7; MotionEle is the motion element of the target, which is currently empty; TarSource is the source of the target.

9. An electronic device comprising a processor and a memory, the memory storing program instructions, wherein: The processor runs program instructions to realize the unmanned surface vehicle multi-element information joint perception method according to any one of claims 1 to 7.

Images