Millimeter wave radar-based water surface target moving and static segmentation method and device

Abstract

The application provides a millimeter wave radar-based water surface target dynamic and static segmentation method and device, which comprises the following steps: performing dynamic and static semantic labeling on millimeter wave radar point clouds in a data set; using two frames of point clouds with an interval of a frames to build a speed estimation cross self-attention network, and training the network; using the speed estimation cross self-attention network to estimate the speed of each data packet in the data set, and performing millimeter wave radar point cloud radial speed compensation; using the compensated millimeter wave radar point clouds with an interval of a frames to build a dynamic and static segmentation cross self-attention network, and training the network; and using the trained model to obtain the dynamic and static semantics of the millimeter wave radar point clouds in the current water surface environment when the unmanned ship is performing a task. The application has the beneficial effect that the millimeter wave radar point clouds are subjected to dynamic and static semantic segmentation under the condition that no other additional sensors are used, the dynamic and static semantic perception of the water surface target is completed, and a low-cost device for water surface target dynamic and static perception is realized.

Description

Technical Field

[0001] This invention relates to the field of environmental perception technology in the field of unmanned vessels, and in particular to a method and device for separating static and dynamic water surface targets based on millimeter-wave radar. Background Technology

[0002] With the popularization and application of unmanned surface vessel (USV) technology, target motion / static semantic segmentation is a key perception technology in functions such as collision detection and autonomous obstacle avoidance in unmanned surface driving. In the past few years, target motion / static semantic segmentation has mainly been achieved based on lidar and camera sensors, achieving good performance. However, these optical sensors are easily affected by the environment; their perception performance deteriorates significantly in rain, snow, and fog, and lidar is expensive. Millimeter-wave radar, on the other hand, is inexpensive and an all-weather, all-time sensor. Previous research mainly used Doppler velocity data from millimeter-wave radar to detect targets, combined with USV odometry information, for target motion / static semantic recognition. However, due to clutter in water surface point clouds and detection errors, this type of point cloud motion / static segmentation method based on Doppler velocity is basically unusable in water surface environments. In the past two years, some scholars have proposed millimeter-wave radar point cloud motion / static segmentation methods based on scene flow, but their performance in water surface environments remains low. Therefore, achieving a low-cost, environmentally robust target motion / static segmentation system in water surface environments remains a significant challenge. Summary of the Invention

[0003] The technical problem to be solved by the present invention is to provide a method and device for dynamic and static segmentation of water surface targets based on millimeter-wave radar. The aim is to perform dynamic and static semantic segmentation of millimeter-wave radar point clouds without other additional sensors, complete the dynamic and static semantic perception of water surface targets, and realize a low-cost device for dynamic and static perception of water surface targets.

[0004] To solve the above-mentioned technical problems, the technical solution adopted by the present invention is: a method for separating static and dynamic water surface targets based on millimeter-wave radar, comprising:

[0005] The millimeter-wave radar on the unmanned vessel continuously collects millimeter-wave radar point clouds of the surrounding environment and the speed data of the unmanned vessel itself. All the collected data are constructed into a dataset, and the millimeter-wave radar point clouds in the dataset are labeled with dynamic and static semantics.

[0006] A rate estimation cross-self-attention network M1 is constructed using two point clouds with an interval of a frames.

[0007] Using the velocity corresponding to the millimeter-wave radar point cloud in the next frame as the ground truth, a Doppler loss function L is constructed using the dynamic and static semantic ground truth. doppler and the mean square loss function L mse Construct a loss function and train the rate estimation cross-attention network M1;

[0008] The velocity estimation cross-self-attention network M1 is used to estimate the velocity of the existing data, and the self-velocity V of each packet in the dataset is obtained. P And perform radial velocity compensation for millimeter-wave radar point clouds;

[0009] A dynamic-static segmentation cross-attention network M2 is constructed using the compensated millimeter-wave radar point cloud with an interval of a frames.

[0010] Constructing the dynamic and static semantic loss function L seg The dynamic-static segmentation cross-attention network M2 was trained.

[0011] When the unmanned vessel is performing a mission in real time, it collects millimeter-wave radar point clouds of the environment in real time, and uses the trained models M1 and M2 to obtain the dynamic and static semantics of the millimeter-wave radar point clouds in the water surface environment at the current moment.

[0012] Furthermore, the continuous acquisition of millimeter-wave radar point clouds of the surrounding environment and the velocity data of the unmanned vessel via its millimeter-wave radar, constructing a dataset from all the acquired data, and performing dynamic and static semantic annotation on the millimeter-wave radar point clouds in the dataset specifically includes:

[0013] The unmanned surface vessel (USV) collects millimeter-wave radar point cloud data, positioning data, IMU data, and image data using sensors, and timestamps these data. The positioning data includes the USV's location C = (x... r ,y r ), x r ,y r The northeast coordinates of the unmanned vessel are in the northeast coordinate system, with east as the x-axis and north as the y-axis. The IMU data includes the unmanned vessel's orientation angle data d, where 0 degrees is north and positive is clockwise.

[0014] Data from different sensors is synchronized using corresponding timestamps. The synchronized data is then packaged separately to form sequence data S = {(P} i C i ,d i ,I i |i>0,i∈N *}, where P i This is the millimeter-wave radar point cloud corresponding to each data packet. The coordinate system of this millimeter-wave radar point cloud is established with the unmanned vessel's own position as the center, with the y-axis pointing forward, the x-axis pointing to the right, and the z-axis pointing upward. i An image corresponding to each packet of data;

[0015] By using the position C and orientation d of the unmanned vessel at each moment, spatial compensation is performed on the corresponding millimeter-wave radar point cloud to obtain the position of each frame of millimeter-wave radar point cloud in the northeast coordinate system.

[0016] By utilizing the changes in data between consecutive frames and the corresponding image data, the moving target point cloud is identified and labeled to obtain the dynamic and static semantic label S of each frame of millimeter-wave radar point cloud.

[0017] The velocity v of the unmanned surface vessel in each data packet is calculated using the position C of the preceding and following frames and the corresponding timestamps. gt This velocity direction is assumed to be the orientation of the unmanned vessel in subsequent steps, and is integrated to form sequence data S′={(P i C i ,d i ,I i ,S i V i |i>0,i∈N *}

[0018] Furthermore, the construction of the rate estimation cross-self-attention network using two point clouds with an interval of 'a' specifically includes:

[0019] Take one frame of millimeter-wave radar point cloud P from the sequence data S′ respectively i ={(x i ,y i ,z i ,v i )|i≤N i} and its corresponding millimeter-wave radar point cloud P in the previous a frame i-a As network input data, x i ,y i ,z i These represent the x-axis, y-axis, and z-axis coordinates of each point in its own coordinate system, respectively, and v i For each point, the relative radial velocity with respect to the unmanned vessel, N i This represents the number of points in the corresponding millimeter-wave radar point cloud.

[0020] Specifically targeting millimeter-wave radar point clouds P i With P i-a Perform point cloud resampling to obtain data P′ i With P′ i-a By sampling a fixed number of N points, the number of millimeter-wave radar point clouds input into the model is kept the same each time;

[0021] Using point-transformer structures to target millimeter-wave radar point clouds P′ i With P′ i-a Feature extraction is performed to obtain feature F. i With F i-a ;

[0022] Feature F i With Fi-a The points-crossformer structure is passed into the points-crossformer structure to obtain the temporal spatial features F formed by two frames of millimeter-wave radar point clouds;

[0023] The temporal features F are processed through a multilayer perceptron network to output the final predicted velocity v. p The rate estimation cross-attention network M1 was constructed.

[0024] Furthermore, the velocity corresponding to the millimeter-wave radar point cloud in the next frame is used as the ground truth, and the Doppler loss function L is constructed using the dynamic and static semantic ground truth. doppler and the mean square loss function L mse Constructing the loss function and training the rate estimation cross-self-attention network M1 specifically includes the following steps:

[0025] Using dynamic and static semantic truth values ​​from millimeter-wave radar point cloud P i Select the static target point cloud P from the middle s There are a total of N points. s One point;

[0026] The static point cloud P s radial velocity v at all points i Projecting the image onto the direction directly in front of the unmanned vessel, we obtain the projected velocity.

[0027] Projection velocity of each static point cloud Equals the rate estimation of the predicted velocity v of the cross-attention network M1 p The radial velocity components at each point are used to construct the Doppler loss function L. doppler ;

[0028] Estimating the prediction velocity v of the cross-self-attention network M1 using rate estimation p With true velocity v gt Construct the mean squared error loss function L mse , where M is the number of all data packets;

[0029] Using the Doppler loss function L doppler With mean square error loss function L mse Constructing the rate estimation loss function L during training of the cross-self-attention network M1 v ;

[0030] Using loss function L v The Adam-optimized rate estimation cross-attention network M1 is trained.

[0031] Furthermore, the rate estimation cross-attention network M1 is used to estimate the velocity of the existing data, thereby obtaining the self-velocity V corresponding to each packet of data in the dataset.P And perform radial velocity compensation for millimeter-wave radar point clouds, specifically including,

[0032] For the data in the data sequence S′, the network M1 is used to estimate its own velocity, forming the sequence data S″={(P i C i ,d i ,I i ,S i V i V i P |i>0,i∈N *};

[0033] For the millimeter-wave radar point cloud P in the sequence data S″ i Doppler velocity compensation is performed to obtain the corresponding millimeter-wave radar point cloud P. i ", forming sequence data S″′={(P i ",C i ,d i ,I i ,S i V i V i P |i>0,i∈N *}

[0034] Furthermore, the millimeter-wave radar point cloud P in the sequence data S″ i Doppler velocity compensation is performed to obtain the corresponding millimeter-wave radar point cloud P″. i Specifically, including,

[0035] Take a point cloud P, and measure the Doppler velocity v in P. i Compensated to the horizontal plane coordinate system, the velocity v″ is obtained. i ;

[0036] Calculate the projected velocity v in each direction of P using the predicted velocity of the current packet. compen ;

[0037] Calculate the compensation velocity v′ at each point i ;

[0038] Constructing a millimeter-wave radar point set P″={(x i ,y i ,z i ,v i ′)|i≤N}, where N is the number of point clouds in the current packet.

[0039] Furthermore, the construction of the dynamic-static segmentation cross-attention network M2 using the compensated millimeter-wave radar point cloud at intervals of a frames specifically includes:

[0040] Take one frame of millimeter-wave radar point cloud P from the sequence data S″ respectively i "={(x i ,y i ,z i ,v i ′)|i≤N i} and its corresponding millimeter-wave radar point cloud P″ of the previous a frame i-a The point cloud is used as network input data;

[0041] The knn sampling method in the point-transformer is modified to ball query sampling method. Target self-attention calculation is performed within a stable target receptive field, forming a radar-transformer structure.

[0042] Using radar-transformer architecture to target millimeter-wave radar point cloud data P″ i With P″ i-a Feature extraction is performed to obtain the feature F for each point. i With F i-a ;

[0043] The feature F of each point i With F i-a The data is passed into the points-crossformer structure to obtain the temporal spatial features F formed by two frames of millimeter-wave radar point clouds;

[0044] By connecting the temporal spatial feature F with the spatial feature of each point through channels, we obtain the deep and shallow layer features F;

[0045] The radar-transformer is used to fuse the deep and shallow features F to obtain the dynamic and static semantic features of each point;

[0046] The dynamic and static semantic features of each point are passed through a multilayer perceptron network to output the dynamic and static semantic information of each point, thus completing the construction of the dynamic and static segmentation cross-attention network M2.

[0047] Furthermore, the construction of the dynamic-static semantic loss function L seg Training the dynamic-static segmentation cross-attention network M2 specifically includes:

[0048] The cross-entropy loss function L is calculated point-by-point by combining the static / dynamic semantic category with the truth value. cross ;

[0049] Construct a static semantic loss function L using the radial velocity of the millimeter-wave radar point cloud predicted as static semantics. static The number of static target point clouds is M′;

[0050] Using the cross-entropy loss function L cross With static semantic loss function L static Constructing the rate estimation loss function L during training of the cross-self-attention network M1 seg ;

[0051] Using loss function L seg The Adam optimization was used to train the dynamic-static segmentation cross-attention network M2.

[0052] Furthermore, the step of acquiring millimeter-wave radar point clouds of the environment in real time while the unmanned vessel is performing its mission, and using trained models M1 and M2 to obtain the dynamic and static semantics of the millimeter-wave radar point clouds in the water surface environment at the current moment, specifically includes:

[0053] The millimeter-wave radar point cloud of the current frame and the millimeter-wave radar point cloud of the previous a frame are resampled to obtain a fixed number N millimeter-wave radar point clouds, which are then input into model M1 to obtain the current speed v of the unmanned vessel.

[0054] Radial velocity compensation is performed on the millimeter-wave radar point cloud of the current frame using the velocity v at the current moment to obtain the velocity-compensated point cloud, and the same steps are performed on the millimeter-wave radar point cloud of the previous a frames.

[0055] The compensated millimeter-wave radar point cloud and the compensated millimeter-wave radar point cloud of the previous a frame are input into model M2 to obtain the dynamic and static semantics of the millimeter-wave radar point cloud in the water surface environment at the current moment.

[0056] The present invention also provides a device for separating static and dynamic water surface targets based on millimeter-wave radar, comprising:

[0057] The data acquisition module is used to continuously acquire millimeter-wave radar point clouds of the surrounding environment and the speed data of the unmanned vessel itself through the millimeter-wave radar on the unmanned vessel, construct all the acquired data into a dataset, and perform dynamic and static semantic annotation on the millimeter-wave radar point clouds in the dataset.

[0058] The first network construction module is used to construct a rate estimation cross-self-attention network M1 using two frames of point cloud with an interval of a frames.

[0059] The first network training module uses the velocity corresponding to the millimeter-wave radar point cloud in the next frame as the ground truth, and constructs the Doppler loss function L using the dynamic and static semantic ground truth. doppler and the mean square loss function L mse Construct a loss function and train the rate estimation cross-attention network M1;

[0060] The radial velocity compensation module is used to estimate the velocity of existing data using the rate estimation cross-self-attention network M1, and obtain the self-velocity V of each packet in the dataset. P And perform radial velocity compensation for millimeter-wave radar point clouds;

[0061] The second network construction module is used to build a dynamic-static segmentation cross-attention network M2 using the compensated millimeter-wave radar point cloud with an interval of a frames.

[0062] The second network training module is used to construct the dynamic and static semantic loss function L. seg The dynamic-static segmentation cross-attention network M2 was trained.

[0063] The dynamic-static segmentation module is used to collect millimeter-wave radar point clouds of the environment in real time when the unmanned vessel is performing tasks. It uses the trained models M1 and M2 to obtain the dynamic-static semantics of the millimeter-wave radar point clouds in the water surface environment at the current moment.

[0064] The beneficial effects of this invention are: without any other additional sensors, dynamic and static semantic segmentation of millimeter-wave radar point clouds is performed to complete the dynamic and static semantic perception of water surface targets, thereby realizing a low-cost device for dynamic and static perception of water surface targets. Attached Figure Description

[0065] The specific structure of the present invention will now be described in detail with reference to the accompanying drawings.

[0066] Figure 1 This is a flowchart of the dynamic and static target segmentation method for water surface based on millimeter-wave radar according to an embodiment of the present invention;

[0067] Figure 2 This is a block diagram of a water surface target dynamic and static segmentation device based on millimeter-wave radar according to an embodiment of the present invention;

[0068] Figure 3 This is a schematic diagram of the Points-crossformer structure according to an embodiment of the present invention;

[0069] Figure 4 This is a schematic diagram of a rate estimation cross-self-attention network according to an embodiment of the present invention;

[0070] Figure 5 This is a schematic diagram of a dynamic-static segmentation cross-attention network according to an embodiment of the present invention.

[0071] Figure 6 This is a schematic block diagram of a computer device according to an embodiment of the present invention. Detailed Implementation

[0072] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0073] It should be understood that, when used in this specification and the appended claims, the terms "comprising" and "including" indicate the presence of the described features, integrals, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or collections thereof.

[0074] It should also be understood that the terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to limit the invention. As used in this specification and the appended claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms unless the context clearly indicates otherwise.

[0075] It should also be further understood that the term "and / or" as used in this specification and the appended claims refers to any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0076] like Figure 1 As shown, the first embodiment of the present invention is: a method for separating static and dynamic water surface targets based on millimeter-wave radar, comprising:

[0077] S10. Continuously collect millimeter-wave radar point clouds of the surrounding environment and the speed data of the unmanned vessel itself through the millimeter-wave radar on the unmanned vessel, construct a dataset from all the collected data, and perform dynamic and static semantic annotation on the millimeter-wave radar point clouds in the dataset.

[0078] S20. Use two frames of point clouds with an interval of a to build a rate estimation cross-self-attention network M1;

[0079] S30. Using the velocity corresponding to the millimeter-wave radar point cloud in the next frame as the ground truth, construct the Doppler loss function L using the dynamic and static semantic ground truth. doppler and the mean square loss function L mse Construct a loss function and train the rate estimation cross-attention network M1;

[0080] S40. Use the rate estimation cross-self-attention network M1 to estimate the rate of the existing data and obtain the self-rate V corresponding to each packet of data in the dataset. P And perform radial velocity compensation for millimeter-wave radar point clouds;

[0081] S50. A dynamic-static segmentation cross-attention network M2 is constructed using the compensated millimeter-wave radar point cloud with an interval of a frames.

[0082] S60. Construct the dynamic and static semantic loss function L seg The dynamic-static segmentation cross-attention network M2 was trained.

[0083] S70. When the unmanned vessel is performing a task in real time, it collects millimeter-wave radar point clouds of the environment in real time, and uses the trained model M1 and model M2 to obtain the dynamic and static semantics of the millimeter-wave radar point clouds in the water surface environment at the current moment.

[0084] In step S10, the millimeter-wave radar on the unmanned surface vessel (USV) continuously collects millimeter-wave radar point clouds of the surrounding environment and the USV's own speed data. All collected data is then constructed into a dataset, and the millimeter-wave radar point clouds in the dataset are subjected to dynamic and static semantic annotation. Specifically, this includes...

[0085] S11. Collect millimeter-wave radar point cloud, positioning data, IMU data, and image data using sensors on the unmanned surface vessel (USV), and mark them with timestamps. The positioning data includes the USV's location C = (x... r ,y r ), x r ,y r The northeast coordinates of the unmanned vessel are in the northeast coordinate system, with east as the x-axis and north as the y-axis. The IMU data includes the unmanned vessel's orientation angle data d, where 0 degrees is north and positive is clockwise.

[0086] S12. Synchronize data from different sensors using corresponding timestamps, and package the synchronized data separately to form sequence data S = {(P i C i ,d i ,I i |i>0,i∈N *}, where P i This is the millimeter-wave radar point cloud corresponding to each data packet. The coordinate system of this millimeter-wave radar point cloud is established with the unmanned vessel's own position as the center, with the y-axis pointing forward, the x-axis pointing to the right, and the z-axis pointing upward. i An image corresponding to each packet of data;

[0087] S13. Use the position C and direction d of the unmanned ship at each moment to perform spatial compensation on the corresponding millimeter-wave radar point cloud to obtain the position of each frame of millimeter-wave radar point cloud in the northeast coordinate system.

[0088] S14. Using the point_labeler tool, the moving target point cloud is identified by the changes in the data of the previous and next frames and the corresponding image data, and then labeled to obtain the dynamic and static semantic label S of each frame of millimeter-wave radar point cloud.

[0089] S15. Calculate the velocity v of the unmanned surface vessel in each data packet using the position C of the preceding and following frames and the corresponding timestamps. gt This velocity direction is assumed to be the orientation of the unmanned vessel in subsequent steps, and is integrated to form sequence data S′={(P i C i ,d i ,I i ,S i V i |i>0,i∈N *}

[0090] Specifically, step S20, which involves constructing a rate estimation cross-self-attention network using two point clouds with an interval of 'a', includes the following:

[0091] S21. Take one frame of millimeter-wave radar point cloud P from the sequence data S′ respectively. i ={(x i ,y i ,z i ,v i )|i≤N i} and its corresponding millimeter-wave radar point cloud P in the previous a frame i-a As network input data, x i ,y i ,z i These represent the x-axis, y-axis, and z-axis coordinates of each point in its own coordinate system, respectively, and v i For each point, the relative radial velocity with respect to the unmanned vessel, N i This represents the number of points in the corresponding millimeter-wave radar point cloud.

[0092] S22, respectively targeting millimeter-wave radar point clouds P i With P i-a Perform point cloud resampling to obtain data P′ i With P′ i-a By sampling a fixed number of N points, the number of millimeter-wave radar point clouds input into the model is kept the same each time;

[0093] S23. Using the point-transformer structure, respectively targeting the millimeter-wave radar point cloud P′ i With P′ i-a Feature extraction is performed to obtain feature F. i With F i-a ;

[0094] S24, Feature Fi With F i-a The data is passed into the points-crossformer structure to obtain the temporal spatial features F formed by two frames of millimeter-wave radar point clouds; for example... Figure 3 The diagram shows the points-crossformer structure; the points in the two large boxes on the left correspond to the features f of each point, which are processed by a perceptron to obtain α, β, ω, γ, and then the final result is obtained by addition and subtraction.

[0095] S25. Using the temporal feature F, pass it through a multilayer perceptron network to output the final predicted speed v. p The rate estimation cross-self-attention network M1 was constructed. The structure of the rate estimation cross-self-attention network M1 is as follows: Figure 4 As shown.

[0096] In step S30, the velocity corresponding to the millimeter-wave radar point cloud in the next frame is used as the ground truth, and the Doppler loss function L is constructed using the dynamic and static semantic ground truth. doppler and the mean square loss function L mse Constructing the loss function and training the rate estimation cross-self-attention network M1 specifically includes the following steps:

[0097] S31. Utilizing dynamic and static semantic truth values ​​from millimeter-wave radar point cloud P i Select the static target point cloud P from the middle s There are a total of N points. s One point;

[0098] S32. The static target point cloud P s radial velocity v at all points i Projecting the image onto the direction directly in front of the unmanned vessel, we obtain the projected velocity.

[0099]

[0100] S33, Projection velocity of the point cloud for each static target Equals the rate estimation of the predicted velocity v of the cross-attention network M1 p The radial velocity components at each point are used to construct the Doppler loss function L. doppler ;

[0101]

[0102] S34. Estimating the prediction speed v of the cross-self-attention network M1 using rate estimation. p With true velocity v gt Construct the mean squared error loss function L mse , where M is the number of all data packets;

[0103]

[0104] S35. Using the Doppler loss function L doppler With mean square error loss function L mse Constructing the rate estimation loss function L during training of the cross-self-attention network M1 v ;

[0105] L v =L doppler +L mse

[0106] S36. Using the loss function L v The Adam-optimized rate estimation cross-attention network M1 is trained.

[0107] In step S40, the rate estimation cross-attention network M1 is used to estimate the rate of the existing data to obtain the self-rate V corresponding to each packet of data in the dataset. P And perform radial velocity compensation for millimeter-wave radar point clouds, specifically including,

[0108] S41. Using network M1, the velocity of the data in the data sequence S′ is estimated to form the sequence data S″={(P i C i ,d i ,I i ,S i V i V i P |i>0,i∈N *};

[0109] S42. For the millimeter-wave radar point cloud P in the sequence data S″ i Doppler velocity compensation is performed to obtain the corresponding millimeter-wave radar point cloud P. i ", forming sequence data S″′={(P i ",C i ,d i ,I i ,S i V i V i P |i>0,i∈N *}

[0110] In step S42, the millimeter-wave radar point cloud P in the sequence data S″ is... i Doppler velocity compensation is performed to obtain the corresponding millimeter-wave radar point cloud P″. i Specifically, including,

[0111] S421. Take a point cloud P and record the Doppler velocity v in P. i Compensated to the horizontal plane coordinate system, the velocity v″ is obtained. i ;

[0112]

[0113] S422. Calculate the projected velocity v in the direction of each point in P using the predicted velocity of the current packet. compen ;

[0114]

[0115] S423. Calculate the compensation velocity v′ at each point. i ;

[0116] v′ i =v″ i -v compen

[0117] S424. Constructing a millimeter-wave radar point cloud P″={(x i ,y i ,z i ,v i ′)|i≤N}, where N is the number of point clouds in the current packet.

[0118] Specifically, in step S50, the construction of the dynamic-static segmentation cross-attention network M2 using the compensated millimeter-wave radar point cloud at interval a frames includes:

[0119] S51. Take one frame of millimeter-wave radar point cloud P from the sequence data S″ respectively. i "={(x i ,y i ,z i ,v i ′)|i≤N i} and its corresponding millimeter-wave radar point cloud P″ of the previous a frame i-a The point cloud is used as network input data;

[0120] S52. Modify the knn sampling in the point-transformer to ball query sampling, and perform target self-attention calculation within a stable target receptive field to form a radar-transformer structure;

[0121] Since k-nn selects the k nearest points for each point for calculation, and millimeter-wave radar point clouds are relatively sparse, these k points may come from relatively close points or relatively distant points. By using ball query, each point will only select points in a nearby region for calculation, and will not generate data from irrelevant points at a greater distance, thus avoiding instability in the results and ultimately improving recognition accuracy.

[0122] S53, using the radar-transformer structure to target millimeter-wave radar point cloud data P′ respectively i With P″ i-a Feature extraction is performed to obtain the feature F for each point. i With F i-a ;

[0123] S54. Extract the features F of each point. i With F i-a The data is passed into the points-crossformer structure to obtain the temporal spatial features F formed by two frames of millimeter-wave radar point clouds;

[0124] S55. Connect the temporal spatial feature F with the spatial feature of each point to obtain the deep and shallow layer features F;

[0125] S56. Use radar-transformer to fuse the deep and shallow features F to obtain the dynamic and static semantic features of each point.

[0126] S57. The dynamic and static semantic features of each point are processed through a multilayer perceptron network to output the dynamic and static semantic information of each point, thus completing the construction of the dynamic and static segmentation cross-self-attention network M2. The structure diagram of the dynamic and static segmentation cross-self-attention network M2 is as follows. Figure 5 As shown.

[0127] In step S60, the construction of the dynamic-static semantic loss function L seg Training the dynamic-static segmentation cross-attention network M2 specifically includes:

[0128] S61. Calculate the cross-entropy loss function L formed by the dynamic and static semantic categories and the truth value for each point. cross ;

[0129]

[0130] S62. Construct a static semantic loss function L using the radial velocity of the millimeter-wave radar point cloud predicted as static semantics. static The number of static target point clouds is M′;

[0131]

[0132] S63, Using the cross-entropy loss function L cross With static semantic loss function L static Constructing the rate estimation loss function L during training of the cross-self-attention network M1 seg ;

[0133] L seg =L cross +L static

[0134] S64. Using the loss function L seg The Adam optimization was used to train the dynamic-static segmentation cross-attention network M2.

[0135] In step S70, the step of acquiring millimeter-wave radar point clouds of the environment in real time while the unmanned vessel is performing its mission, and obtaining the dynamic and static semantics of the millimeter-wave radar point clouds in the water surface environment at the current moment using trained models M1 and M2, specifically includes:

[0136] S71. Resample the millimeter-wave radar point cloud of the current frame and the millimeter-wave radar point cloud of the previous a frame respectively to obtain a fixed number N millimeter-wave radar point clouds, and input them into model M1 to obtain the current speed v of the unmanned vessel.

[0137] S72. Use the current velocity v to perform radial velocity compensation on the millimeter-wave radar point cloud of the current frame to obtain the velocity-compensated point cloud, and perform the same steps on the millimeter-wave radar point cloud of the previous a frames.

[0138] S73. Input the compensated millimeter-wave radar point cloud and the compensated millimeter-wave radar point cloud of the previous a frame into model M2 to obtain the dynamic and static segmentation semantics of the millimeter-wave radar point cloud in the water surface environment at the current moment.

[0139] The technical solution of this application utilizes millimeter-wave radar to complete the dynamic and static semantic segmentation of water surface targets without the need for other additional sensors, thereby realizing the dynamic and static semantic perception of targets and greatly reducing the cost and complexity of functional development.

[0140] like Figure 2 As shown, the present invention also provides a surface target dynamic and static segmentation device based on millimeter-wave radar, comprising:

[0141] The data acquisition module 10 is used to continuously acquire millimeter-wave radar point clouds of the surrounding environment and the speed data of the unmanned vessel itself through the millimeter-wave radar on the unmanned vessel, construct all the acquired data into a dataset, and perform dynamic and static semantic annotation on the millimeter-wave radar point clouds in the dataset.

[0142] The first network construction module 20 is used to construct a rate estimation cross-self-attention network M1 using two frames of point clouds with an interval of a frames.

[0143] The first network training module 30 is used to take the velocity corresponding to the millimeter-wave radar point cloud in the next frame as the ground value, and construct the Doppler loss function L using the dynamic and static semantic ground values. doppler and the mean square loss function L mse Construct a loss function and train the rate estimation cross-attention network M1;

[0144] Radial velocity compensation module 40 is used to estimate the velocity of existing data using the rate estimation cross-self-attention network M1, and obtain the self-velocity V corresponding to each packet of data in the dataset. P And perform radial velocity compensation for millimeter-wave radar point clouds;

[0145] The second network construction module 50 is used to construct a dynamic-static segmentation cross self-attention network M2 using the compensated millimeter-wave radar point cloud with an interval of a frames.

[0146] The second network training module 60 is used to construct the dynamic and static semantic loss function L. seg The dynamic-static segmentation cross-attention network M2 was trained.

[0147] The static / dynamic segmentation module 70 is used to collect millimeter-wave radar point clouds of the environment in real time when the unmanned vessel is performing tasks in real time, and to obtain the static / dynamic semantics of the millimeter-wave radar point clouds in the water surface environment at the current moment using the trained model M1 and model M2.

[0148] It should be noted that those skilled in the art can clearly understand that the specific implementation process of the above-mentioned water surface target dynamic and static segmentation device based on millimeter-wave radar can be referred to the corresponding description in the foregoing method embodiments. For the sake of convenience and brevity, it will not be repeated here.

[0149] The aforementioned surface target dynamic and static separation device based on millimeter-wave radar can be implemented as a computer program, which can be used in, for example... Figure 6 It runs on the computer device shown.

[0150] Please see Figure 6 , Figure 6 This is a schematic block diagram of a computer device provided in an embodiment of this application. The computer device 500 can be a terminal or a server. The terminal can be an electronic device with communication functions, such as a smartphone, tablet, laptop, desktop computer, personal digital assistant, or wearable device. The server can be a standalone server or a server cluster composed of multiple servers.

[0151] See Figure 6The computer device 500 includes a processor 502, a memory, and a network interface 505 connected via a system bus 501. The memory may include a non-volatile storage medium 503 and internal memory 504.

[0152] The non-volatile storage medium 503 may store an operating system 5031 and a computer program 5032. The computer program 5032 includes program instructions that, when executed, cause the processor 502 to perform a method for separating static and dynamic water surface targets based on millimeter-wave radar.

[0153] The processor 502 provides computing and control capabilities to support the operation of the entire computer device 500.

[0154] The internal memory 504 provides an environment for the operation of the computer program 5032 in the non-volatile storage medium 503. When the computer program 5032 is executed by the processor 502, the processor 502 can execute a method for separating the static and dynamic targets on the water surface based on millimeter-wave radar.

[0155] This network interface 505 is used for network communication with other devices. Those skilled in the art will understand that... Figure 6 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device 500 to which the present application is applied. The specific computer device 500 may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0156] The processor 502 is used to run the computer program 5032 stored in the memory to implement the dynamic and static target segmentation method based on millimeter-wave radar as described above.

[0157] It should be understood that, in the embodiments of this application, the processor 502 may be a central processing unit (CPU), or it may be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor.

[0158] It will be understood by those skilled in the art that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program includes program instructions and can be stored in a storage medium, which is a computer-readable storage medium. The program instructions are executed by at least one processor in the computer system to implement the process steps of the embodiments of the above methods.

[0159] Therefore, the present invention also provides a storage medium. This storage medium can be a computer-readable storage medium. The storage medium stores a computer program, wherein the computer program includes program instructions. When executed by a processor, the program instructions cause the processor to perform the millimeter-wave radar-based method for separating static and dynamic water surface targets as described above.

[0160] The storage medium can be any computer-readable storage medium capable of storing program code, such as a USB flash drive, portable hard drive, read-only memory (ROM), magnetic disk, or optical disk.

[0161] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0162] In the several embodiments provided by this invention, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative. For example, the division of each unit is merely a logical functional division, and there may be other division methods in actual implementation. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed.

[0163] The steps in the method of this invention can be adjusted, merged, or reduced in order according to actual needs. The units in the device of this invention can be merged, divided, or reduced according to actual needs. Furthermore, the functional units in the various embodiments of this invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0164] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, a terminal, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention.

[0165] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in the present invention, and these modifications or substitutions should all be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A method for separating static and dynamic water surface targets based on millimeter-wave radar, characterized in that, include: The millimeter-wave radar on the unmanned vessel continuously collects millimeter-wave radar point clouds of the surrounding environment and the speed data of the unmanned vessel itself. All the collected data are constructed into a dataset, and the millimeter-wave radar point clouds in the dataset are labeled with dynamic and static semantics. A rate estimation cross-self-attention network M1 is constructed using two point clouds with an interval of a frames. Using the velocity corresponding to the millimeter-wave radar point cloud in the next frame as the ground truth, a Doppler loss function L is constructed using the dynamic and static semantic ground truth. doppler and the mean square loss function L mse Construct a loss function and train the rate estimation cross-attention network M1; The velocity estimation cross-self-attention network M1 is used to estimate the velocity of the existing data, and the self-velocity V of each packet in the dataset is obtained. P And perform radial velocity compensation for millimeter-wave radar point clouds; A dynamic-static segmentation cross-attention network M2 is constructed using the compensated millimeter-wave radar point cloud with an interval of a frames. Constructing the dynamic and static semantic loss function L seg The dynamic-static segmentation cross-attention network M2 was trained. When the unmanned vessel is performing a mission in real time, it collects millimeter-wave radar point clouds of the environment in real time, and uses the trained models M1 and M2 to obtain the dynamic and static semantics of the millimeter-wave radar point clouds in the water surface environment at the current moment.

2. The method for separating static and dynamic water surface targets based on millimeter-wave radar as described in claim 1, characterized in that: The process of continuously collecting millimeter-wave radar point clouds of the surrounding environment and the velocity data of the unmanned vessel using millimeter-wave radar on the unmanned vessel, constructing a dataset from all the collected data, and performing dynamic and static semantic annotation on the millimeter-wave radar point clouds in the dataset specifically includes: The unmanned surface vessel (USV) collects millimeter-wave radar point cloud data, positioning data, IMU data, and image data using sensors, and timestamps these data. The positioning data includes the USV's location C = (x... r ,y r ), x r ,y r The northeast coordinates of the unmanned vessel are in the northeast coordinate system, with east as the x-axis and north as the y-axis. The IMU data includes the unmanned vessel's orientation angle data d, where 0 degrees is north and positive is clockwise. Data from different sensors is synchronized using corresponding timestamps. The synchronized data is then packaged separately to form sequence data S = {(P} i C i ,d i ,I i |i>0,i∈N * }, where P i This is the millimeter-wave radar point cloud corresponding to each data packet. The coordinate system of this millimeter-wave radar point cloud is established with the unmanned vessel's own position as the center, with the y-axis pointing forward, the x-axis pointing to the right, and the z-axis pointing upward. i An image corresponding to each packet of data; By using the position C and orientation d of the unmanned vessel at each moment, spatial compensation is performed on the corresponding millimeter-wave radar point cloud to obtain the position of each frame of millimeter-wave radar point cloud in the northeast coordinate system. By utilizing the changes in data between consecutive frames and the corresponding image data, the moving target point cloud is identified and labeled to obtain the dynamic and static semantic label S of each frame of millimeter-wave radar point cloud. The velocity v of the unmanned surface vessel in each data packet is calculated using the position C of the preceding and following frames and the corresponding timestamps. gt This velocity direction is assumed to be the orientation of the unmanned vessel in subsequent steps, and is integrated to form sequence data S′={(P i C i ,d i ,I i ,S i V i |i>0,i∈N * } 3. The method for separating static and dynamic water surface targets based on millimeter-wave radar as described in claim 2, characterized in that: The construction of the rate estimation cross-self-attention network M1 using two point clouds with an interval of 'a' specifically includes: Take one frame of millimeter-wave radar point cloud P from the sequence data S′ respectively i ={(x i ,y i ,z i ,v i )|i≤N i } and its corresponding millimeter-wave radar point cloud P in the previous a frame i-a As network input data, x i ,y i ,z i These represent the x-axis, y-axis, and z-axis coordinates of each point in its own coordinate system, respectively, and v i For each point, the relative radial velocity with respect to the unmanned vessel, N i This represents the number of points in the corresponding millimeter-wave radar point cloud. Specifically targeting millimeter-wave radar point clouds P i With P i-a Perform point cloud resampling to obtain data P i ′ and P′ i-a By sampling a fixed number of N points, the number of millimeter-wave radar point clouds input into the model is kept the same each time. Using point-transformer structures to target millimeter-wave radar point clouds P i ′ and P′ i-a Feature extraction is performed to obtain feature F. i With F i-a ; Feature F i With F i-a The points-crossformer structure is passed into the points-crossformer structure to obtain the temporal spatial features F formed by two frames of millimeter-wave radar point clouds; The temporal features F are processed through a multilayer perceptron network to output the final predicted velocity v. p The rate estimation cross-attention network M1 was constructed.

4. The method for separating static and dynamic water surface targets based on millimeter-wave radar as described in claim 3, characterized in that: The velocity corresponding to the millimeter-wave radar point cloud in the next frame is used as the ground truth value, and the Doppler loss function L is constructed using the dynamic and static semantic ground truth values. doppler and the mean square loss function L mse Constructing the loss function and training the rate estimation cross-self-attention network M1 specifically includes the following steps: Using dynamic and static semantic truth values ​​from millimeter-wave radar point cloud P i Select the static target point cloud P from the middle s There are a total of N points. s One point; P static point cloud s radial velocity v at all points i Projecting the image onto the direction directly in front of the unmanned vessel, we obtain the projected velocity. Projection velocity of each static point cloud Equals the rate estimation of the predicted velocity v of the cross-attention network M1 p The radial velocity components at each point are used to construct the Doppler loss function L. doppler ; Estimating the prediction velocity v of the cross-self-attention network M1 using rate estimation p With true velocity v gt Construct the mean squared error loss function L mse ; Using the Doppler loss function L doppler With mean square error loss function L mse Constructing the rate estimation loss function L during training of the cross-self-attention network M1 v ; Using loss function L v The Adam-optimized rate estimation cross-attention network M1 is trained.

5. The method for separating static and dynamic water surface targets based on millimeter-wave radar as described in claim 4, characterized in that: The rate estimation cross-self-attention network M1 is used to estimate the rate of the existing data, thereby obtaining the self-rate V corresponding to each packet of data in the dataset. P And perform radial velocity compensation for millimeter-wave radar point clouds, specifically including, For the data in the data sequence S′, the network M1 is used to estimate its own velocity, forming the sequence data S″={(P i C i ,d i ,I i ,S i V i V i P |i>0,i∈N * }; For the millimeter-wave radar point cloud P in the sequence data S″ i Doppler velocity compensation is performed to obtain the corresponding millimeter-wave radar point cloud P. i ", forming sequence data S″′={(P i ",C i ,d i ,I i ,S i V i V i P |i>0,i∈N * } 6. The method for separating static and dynamic water surface targets based on millimeter-wave radar as described in claim 5, characterized in that: The millimeter-wave radar point cloud P in the sequence data S″ i Doppler velocity compensation is performed to obtain the corresponding millimeter-wave radar point cloud P. i Specifically, including, Take a point cloud P, and measure the Doppler velocity v in P. i Compensated to the horizontal plane coordinate system, the velocity v″ is obtained. i ; Calculate the projected velocity v in each direction of P using the predicted velocity of the current packet. compen ; Calculate the compensation velocity v′ at each point i ; Constructing a millimeter-wave radar point set P″={(x i ,y i ,z i ,v′ i )|i≤N}, where N is the number of point clouds in the current packet.

7. The method for separating static and dynamic water surface targets based on millimeter-wave radar as described in claim 5, characterized in that: The construction of the dynamic-static segmentation cross-attention network M2 using the compensated millimeter-wave radar point cloud with an interval of a frames specifically includes, Take one frame of millimeter-wave radar point cloud P from the sequence data S″ respectively i "={(x i ,y i ,z i ,v′ i )|i≤N i } and its corresponding millimeter-wave radar point cloud P″ of the previous a frame i-a The point cloud is used as network input data; The knn sampling method in the point-transformer is modified to ball query sampling method. Target self-attention calculation is performed within a stable target receptive field, forming a radar-transformer structure. Using radar-transformer architecture to target millimeter-wave radar point cloud data P i "With P" i-a Feature extraction is performed to obtain the feature F for each point. i With F i-a ; The feature F of each point i With F i-a The data is passed into the points-crossformer structure to obtain the temporal spatial features F formed by two frames of millimeter-wave radar point clouds; By connecting the temporal spatial feature F with the spatial feature of each point through channels, we obtain the deep and shallow layer features F; The radar-transformer is used to fuse the deep and shallow features F to obtain the dynamic and static semantic features of each point; The dynamic and static semantic features of each point are passed through a multilayer perceptron network to output the dynamic and static semantic information of each point, thus completing the construction of the dynamic and static segmentation cross-attention network M2.

8. The method for separating static and dynamic water surface targets based on millimeter-wave radar as described in claim 7, characterized in that: The construction of the dynamic and static semantic loss function L seg Training the dynamic-static segmentation cross-attention network M2 specifically includes: The cross-entropy loss function L is calculated point-by-point by combining the static / dynamic semantic category with the truth value. cross ; Construct a static semantic loss function L using the radial velocity of the millimeter-wave radar point cloud predicted as static semantics. static The number of static target point clouds is M′; Using the cross-entropy loss function L cross With static semantic loss function L static Constructing the rate estimation loss function L during training of the cross-self-attention network M1 seg ; Using loss function L seg The Adam optimization was used to train the dynamic-static segmentation cross-attention network M2.

9. The method for separating static and dynamic water surface targets based on millimeter-wave radar as described in claim 1, characterized in that: The process of acquiring millimeter-wave radar point clouds of the environment in real time while the unmanned vessel is performing its mission, and using trained models M1 and M2 to obtain the dynamic and static semantics of the millimeter-wave radar point clouds in the water surface environment at the current moment, specifically includes: The millimeter-wave radar point cloud of the current frame and the millimeter-wave radar point cloud of the previous a frame are resampled to obtain a fixed number N millimeter-wave radar point clouds, which are then input into model M1 to obtain the current speed v of the unmanned vessel. Radial velocity compensation is performed on the millimeter-wave radar point cloud of the current frame using the velocity v at the current moment to obtain the velocity-compensated point cloud, and the same steps are performed on the millimeter-wave radar point cloud of the previous a frames. The compensated millimeter-wave radar point cloud and the compensated millimeter-wave radar point cloud of the previous a frame are input into model M2 to obtain the dynamic and static segmentation semantics of the meter-wave radar point cloud in the current scene.

10. A device for separating static and dynamic water surface targets based on millimeter-wave radar, characterized in that, include: The data acquisition module is used to continuously acquire millimeter-wave radar point clouds of the surrounding environment and the speed data of the unmanned vessel itself through the millimeter-wave radar on the unmanned vessel, construct all the acquired data into a dataset, and perform dynamic and static semantic annotation on the millimeter-wave radar point clouds in the dataset. The first network construction module is used to construct a rate estimation cross-self-attention network M1 using two frames of point cloud with an interval of a frames. The first network training module uses the velocity corresponding to the millimeter-wave radar point cloud in the next frame as the ground truth, and constructs the Doppler loss function L using the dynamic and static semantic ground truth. doppler and the mean square loss function L mse Construct a loss function and train the rate estimation cross-attention network M1; The radial velocity compensation module is used to estimate the velocity of existing data using the rate estimation cross-self-attention network M1, and obtain the self-velocity V of each packet in the dataset. P And perform radial velocity compensation for millimeter-wave radar point clouds; The second network construction module is used to build a dynamic-static segmentation cross-attention network M2 using the compensated millimeter-wave radar point cloud with an interval of a frames. The second network training module is used to construct the dynamic and static semantic loss function L. seg The dynamic-static segmentation cross-attention network M2 was trained. The dynamic-static segmentation module is used to collect millimeter-wave radar point clouds of the environment in real time when the unmanned vessel is performing tasks. It uses the trained models M1 and M2 to obtain the dynamic-static semantics of the millimeter-wave radar point clouds in the water surface environment at the current moment.

Images