Unmanned ship automatic control method based on multi-modal feature fusion

Abstract

The invention discloses an unmanned ship automatic control method based on multi-modal feature fusion, and the method comprises the steps: obtaining a multi-view RGB image and a radar data image, carrying out the BEV feature extraction, and obtaining a visual BEV feature and a radar BEV feature; performing coding fusion on the visual BEV features and the radar BEV features; performing target detection on the fused BEV features through a target sensor to obtain effective target features; predicting a future trajectory of the effective target through a motion prediction module; the aggregation plan query generated through combination is interacted with the fused BEV features, and the final trajectory point of the unmanned ship is predicted; and based on the unmanned ship final track point, generating an unmanned ship operation instruction through PID control. According to the method, radar data is introduced as an auxiliary modal feature, BEV feature fusion is performed on RGB image data and the radar data, and multilayer perception is performed according to self ship query and BEV feature fusion to predict a final track point, so that a real-time control instruction of the unmanned ship is generated.

Description

Technical Field

[0001] The present invention relates to the technical fields of marine environment perception, multimodal feature fusion, and unmanned ship automatic control decision making, and in particular to an unmanned ship automatic control method based on multimodal feature fusion. Background Art

[0002] The unmanned ship automatic control technology aims to enable unmanned ships to have the ability to automatically drive and avoid obstacles after sensing the surrounding environment. With the development of human exploration of the ocean, unmanned ship automatic control has become a high-tech technology that assists humans in efficient and safe ocean exploration. Unmanned ship automatic control can not only free up certain labor costs, but also use unmanned ships to detect and sense small areas such as caves. Therefore, it is very important to develop efficient and intelligent unmanned ship automatic control technology.

[0003] However, the current unmanned ship automatic control technology is not mature enough, and the unmanned ship automatic control methods are too scarce. Using images to generate BEV feature maps (bird's eye view) and performing subsequent target perception and track point prediction on the BEV feature map is an effective unmanned control solution. However, due to low visibility in some marine environments such as nighttime and rainy days, the single-modal data of RGB images may not be able to generate good BEV features, which limits the promotion and application of BEV feature maps in unmanned ship perception and planning tasks. Summary of the invention

[0004] In view of the shortcomings of the prior art, the present invention proposes an unmanned ship automatic control method based on multimodal feature fusion. The method introduces radar data as an auxiliary modal feature, fuses RGB image data and radar data with BEV features, and performs multi-layer perception based on the ship's own query and fused BEV features to predict the final track point, thereby generating real-time control instructions for the unmanned ship.

[0005] The technical means adopted by the present invention are as follows:

[0006] An unmanned ship automatic control method based on multimodal feature fusion, comprising:

[0007] S1, acquiring a multi-view RGB image and a radar data image, performing perspective conversion on the multi-view RGB image and the radar data image, and then extracting BEV features respectively, thereby obtaining visual BEV features and radar BEV features;

[0008] S2, encoding and fusing the visual BEV feature and the radar BEV feature to generate a fused BEV feature;

[0009] S3, performing target detection on the fused BEV feature through a target sensor, thereby obtaining valid target features, each of which represents a valid target, and the valid targets include the current unmanned ship and other targets;

[0010] S4, processing the effective target features and the fused BEV features through a motion prediction module, thereby predicting the future trajectory of the effective target;

[0011] S5, generating a first self-query of the unmanned ship based on the target sensor, and a second self-query of the current unmanned ship based on the motion prediction module, taking the first self-query and the second self-query as inputs of the planner module, interacting the combined generated aggregated plan query with the fused BEV feature, and predicting the final trajectory point of the unmanned ship;

[0012] S6. Based on the final trajectory point of the unmanned ship, the unmanned ship operation instruction is generated through PID control.

[0013] Furthermore, a multi-view RGB image and a radar data image are obtained, and after the view angles of the multi-view RGB image and the radar data image are converted, BEV features are extracted respectively, so as to obtain visual BEV features and radar BEV features, including:

[0014] S101, using Swin-T as an RGB image feature extractor to process the input RGB image, performing multi-scale fusion and downsampling to output RGB image features through FPN; converting the acquired RGB image features into BEV space and performing BEVpooling processing to obtain visual BEV features;

[0015] S102, using VoxelNet as a radar data feature extractor to process the input radar data image, directly performing BEV conversion processing on the radar feature map output by the radar data feature extractor, and then generating radar BEV features.

[0016] Further, the visual BEV feature and the radar BEV feature are coded and fused to generate a fused BEV feature, including:

[0017] S201, performing dimension splicing on the visual BEV features and the radar BEV features;

[0018] S202: Using a convolution-based BEV encoder to compensate for the misalignment in the process of splicing visual BEV features and radar BEV features, thereby generating a fused BEV feature.

[0019] Furthermore, the target sensor includes N TLayer Transformer Encoder, at the beginning of each training sequence, all queries are detection queries. The GrandTurth corresponding to the detection query is matched through the Hungarian algorithm. The query interaction module stores and updates the successfully matched detection queries and uses them as tracking queries for the next timestamp of TrackFormer. In the next timestamp, the detection query is still matched with the GrandTruth target through the Hungarian algorithm, while the tracking query will directly match a part of the GrandTruth target according to the corresponding track ID.

[0020] Furthermore, the motion prediction module includes N A Layers, each layer contains two perceptrons: target-target and target-whole;

[0021] The target-target sensor is:

[0022] Q a =MHCA(MHSA(Q),Q A )

[0023] Among them, MHCA and MHSA represent the multi-head cross attention module and the multi-head self-attention module, Q represents the initialization learnable query, and Q a It seeks the interactive perception characteristics between the current unmanned ship and other targets;

[0024] The target-holistic perceptron is:

[0025]

[0026] in, DeformAttn(q,r,x) represents the predicted trajectory point of the previous layer. DeformAttn(q,r,x) is a deformable attention module. The reference point r and the spatial feature x are used to obtain the learnable query q. It is spatially represented by sparsely paying attention to the features around the reference point. cur Represents the fused BEV features corresponding to the current frame (t=cur).

[0027] Furthermore, the effective target features and the fused BEV features are processed by a motion prediction module to predict the future trajectory of the effective target, including:

[0028] S401, the Q generated by the two perceptrons of target-target and target-overall a and Q g After splicing, it is passed to two different multi-layer perceptrons (MLP) to generate predicted trajectory points respectively. Query context Then Q ctx Send to the next layer as input for Q initialization;

[0029] S402: The input query Q of each layer includes two parts: the query context Q generated by the previous layer ctx And query location Q pos , where Q pos Obtained by the following calculation

[0030]

[0031] Among them, PE(·) is the sinusoidal position encoding, MLP is used to encode the position points, and I s is the scene-level anchor position, I a is the target-level anchor, is the starting point,

[0032] is the predicted endpoint;

[0033] S403, after N A After the motion prediction module of the layer performs action perception prediction, it predicts the multimodal future motion trajectories of all valid targets.

[0034] Furthermore, the aggregated plan query generated by combining the first self-query and the second self-query as inputs of the planner module interacts with the fused BEV features to predict the final trajectory point of the unmanned ship, including:

[0035] The decoder cross-perceives the aggregated plan query and the fused BEV features, combines the BEV features with the sinusoidal position embedding, and then regresses the planned trajectory using the MLP layer. The prediction is recorded as Indicates the future T p The predicted path track point position information of the frame.

[0036] Furthermore, based on the final track point of the unmanned ship, the unmanned ship operation instruction is generated through PID control, including: By sending it into PID control, the automatic control instructions for the unmanned ship at the current moment can be generated.

[0037] Compared with the prior art, the present invention has the following advantages:

[0038] The present invention proposes an automatic control method for unmanned boats based on multimodal feature fusion. The method generates a more usable fused BEV feature representation for subsequent perception in a multimodal fusion manner, which is more effective and robust than previous methods. In addition, the present invention designs an efficient trajectory prediction method that can predict the future trajectories of all valid targets at one time, thereby improving the reasoning speed in practical applications and being able to generate better automatic driving solutions for emergencies more quickly. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0040] Figure 1 The present invention is a flowchart of an automatic control method for an unmanned boat based on multimodal feature fusion in an embodiment of the present invention.

[0041] Figure 2 This is a model architecture diagram of an unmanned boat automatic control method based on multimodal feature fusion in an embodiment of the present invention.

[0042] Figure 3 This is a visualization segmentation diagram of BEV features (bird's eye view) in an embodiment of the present invention. DETAILED DESCRIPTION

[0043] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0044] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0045] like Figure 1 As shown, the present invention provides an unmanned boat automatic control method based on multimodal feature fusion, which specifically includes:

[0046] S1. Obtain multi-view RGB images and radar data images, perform perspective conversion on the multi-view RGB images and radar data images, and then perform BEV feature extraction on them respectively, thereby obtaining visual BEV features and radar BEV features. This step is mainly used to obtain RGB image data and radar point cloud data, and perform feature extraction on the input video frame data and radar point cloud data. Specifically, it includes:

[0047] S101, RGB image data includes video frame data captured by six cameras on the hull, including six directions: front, rear, left front, left rear, right front, and right rear. Swin-T is used as the RGB image feature extractor, and the input RGB image data is N*H*W*C (6*256*704*3). The image is sent to the RGB image feature extractor and multi-scale fusion is performed through FPN and down-sampled to output image features (down-sampled 8x, the output image feature size is 32*88).

[0048] S102, perform image imaging feature conversion on the acquired image features, convert the information in the plane into BEV space and process it by BEV pooling. Specifically, first clearly predict the discrete depth distribution of each pixel of the image feature, decompose the image feature pixel points into D discrete points along the camera ray, and then rescale the relevant features according to the corresponding depth probability. Then generate a visual feature point cloud of size N*H*W*D, where N is the number and (H, W) is the visual feature mapping size. Then quantize the three-dimensional feature point cloud along the x and y axes with a step size of r, use the BEV pooling operation to aggregate all features in each r×r BEV grid and expand the features along the z axis. At this time, the BEV feature map generated by the RGB image is obtained.

[0049] S103, the radar data feature extractor of the present method uses VoxelNet. The radar modal data is in the form of 3D point cloud, and does not need to be converted from 2D to 3D like RGB images. The radar feature map output by the radar data feature extractor is directly voxelized into BEV grid features with a step size r, and expanded along the z axis, and then the BEV feature map of the radar data is generated.

[0050] S2. Encode and fuse the visual BEV features and the radar BEV features to generate a fused BEV feature. This step is mainly used to generate a fused BEV feature map based on the BEV feature map generated by RGB and the BEV feature map generated by radar data. Specifically, it includes:

[0051] S201, the BEV feature map generated by the obtained RGB and the BEV feature map generated by the radar data are spliced ​​in the feature dimension.

[0052] S202: Although in the same feature space, the radar BEV features and the visual BEV features may still be misaligned to a certain extent in space due to the inaccurate depth in the view transformer. To this end, this method applies a convolution-based BEV encoder (including some residual blocks) to compensate for this local misalignment. Finally, a fused and complementary BEV feature map is generated for subsequent back-end tasks.

[0053] S3. Target detection is performed on the fused BEV features through a target sensor to obtain valid target features, each of which represents a valid target, including the current unmanned ship and other targets. This step is mainly used to perform target detection and tracking on the generated fused BEV feature map, and explore the depth information of the unmanned ship and other targets. Specifically, it includes:

[0054] S301, obtaining a fused BEV feature map B of size W*H*D (200*200*256) t (representing the fused BEV feature map corresponding to the RGB data of the tth frame), it is sent to TrackFormer for joint target detection and multi-target tracking. Specifically, in addition to using regular detection queries in TrackFormer, this method also introduces additional tracking queries to the tracking targets across frames. At each time step, the initialization detection query is responsible for detecting the newly perceived new targets for the first time, while the tracking query continues to model the valid targets detected in the previous frame. As the RGB video frames are continuously updated, the tracking query interacts with the previous frames in the current frame to aggregate temporal information in a self-attention module until the corresponding valid target completely disappears (has not been tracked for a period of time).

[0055] S302, Track Former includes N T Layer Transformer Encoder, at the beginning of each training sequence (i.e., the first frame), all queries are detection queries, and the GrandTurth corresponding to the detection query is matched through the Hungarian algorithm. The query interaction module (QIM) stores and updates the successfully matched detection queries and uses them as tracking queries for the next timestamp of TrackFormer. In the next timestamp, the detection query is still matched with the GrandTruth target through the Hungarian algorithm, while the tracking query will directly match a part of the GrandTruth target according to the corresponding track ID.

[0056] S303, to ensure the accuracy of TrackFormer in detecting and tracking other targets, the loss used in this method is as follows:

[0057] L track =λ focal L focal+λ l1 L l1 (3)

[0058] Among them, λ focal =2,λ l2 =0.25, L focal It is the Hungarian loss, which is a linear combination of focal loss and is a loss function used to detect and calculate category labels. l1 The loss function (L1 loss) used for three-dimensional detection coordinate positioning. After TrackFormer, the final output state Q A Provide N for downstream prediction tasks a characteristics of an effective target. It means N a The current frame contains valid target features for detection query and tracking query.

[0059] S4, the effective target features and fused BEV features are processed by the motion prediction module to predict the future trajectory of the effective target. This step is used to represent the effective target features of the current frame obtained in TrackFormer as Q A Put it into the ActionFormer of this method to predict the future motion information of all valid targets. Specifically include:

[0060] S401, ActionFormer contains N A Layers, each layer contains two perceptrons, target-target and target-whole, Q a is the output of the target-target sensor of ActionFormer, which can be expressed by the formula:

[0061] Q a =MHCA(MHSA(Q),Q A ) (4)

[0062] MHCA, MHSA represent multi-head cross attention module and multi-head self-attention module, Q represents the initialization learnable query, Q a It is to seek the interactive perception characteristics between the current unmanned ship and other targets.

[0063] S402, Q g is the output result after the target of ActionFormer - the overall perceptron, which can be expressed by the formula:

[0064]

[0065] DeformAttn(q,r,x) is a deformable attention module. The reference point r and the spatial feature x are used to obtain the learnable query q, which is spatially represented by sparse attention to the features around the reference point. cur Represents the fused BEV features corresponding to the current frame (t=cur).

[0066] S403, the interaction between the two perceptrons is modeled in parallel, and then the generated Q a and Q g After splicing, it is passed to two different multi-layer perceptrons (MLP) to generate predicted trajectory points respectively. Query context Then Q ctx Sent to the next layer as input when Q is initialized.

[0067] S404, the input query Q of each layer is called motion query, which includes two parts: the query context Q generated by the previous layer ctx And query location Q pos Specifically, Q pos The four types of position information are integrated together as shown in the following formula: (1) Scene-level anchor position I s ; (2) Target level anchor position I a ; (3) Current location of valid target (4) Valid target position predicted by the previous layer

[0068]

[0069] PE(·) is the sinusoidal position encoding, and MLP is used to encode the position points. Scene-level anchor point position I s represents the previous motion statistics in the global view, while the target-level anchor I a Captures possible intent in local coordinates. Provide a fixed position embedding for each target, the predicted endpoint As a dynamic anchor point, it is optimized layer by layer in a coarse-to-fine manner to make the predicted position more accurate. pos It represents the prior position information of itself and all other valid targets.

[0070] S405, after N AAfter the action-aware prediction of the ActionFormer layer, the multimodal future motion trajectories of all valid targets are predicted, i.e., the top k possible trajectories. This approach generates multiple valid target trajectories in a frame through a single forward pass, greatly saving the computational cost of aligning the entire scene with the coordinates of each valid target. While considering the future movements of all targets, we use ActionFormer to allow the own ship query obtained from TrackFormer to interact with other valid targets. Formally, the output motion state is represented as Where i is the valid target index, k is the trajectory number index (retaining the number of trajectories with the highest probability before K), and T is the length of the predicted field of view. In layman's terms, The top view coordinates of the predicted trajectory with the highest probability for the i-th valid target. Therefore, the final output of ActionFormer is And contextual queries They represent all target prediction point coordinate information and prediction query context features respectively.

[0071] S5. Generate a first self-query of the unmanned ship based on the target sensor, and a second self-query of the current unmanned ship based on the motion prediction module. Use the first self-query and the second self-query as inputs of the planner module, interact the aggregated plan query generated by the combination with the fused BEV features, and predict the final trajectory points of the unmanned ship. This step is mainly used to obtain the feature representation obtained through all the above perception processes, which is used as input in the planning stage, so as to obtain the track points of the unmanned ship itself. Specifically including:

[0072] S501, the input of the planner module (Planner) uses the own ship query generated by the tracking module (TrackFormer) and the motion prediction module (ActionFormer). Both queries are encoded using the MLP layer, followed by a maximum pooling layer of the modality dimension, in which the most prominent modal features are selected and aggregated. The formula is as follows:

[0073]

[0074] in, is from N a Query your own ship in the query, is from N a When adding the own ship query in the query, Expand according to the modal dimension. When performing maximum pooling, perform maximum pooling according to the modal dimension and generate an aggregate plan query

[0075] S502, BEV feature interaction module is constructed by standard TransFormer decoder layer and stacked N p Layer and set N p =3. Specifically, the decoder converts the dense BEV feature B cur Query Q with aggregate plan p Cross-sensing. In order to embed the position information, the BEV features are combined with the sinusoidal position embedding. Then the MLP layer is used to regress the planned trajectory, and the prediction is recorded as (Set T p =6), indicating the future T p The predicted path track point position information of the frame.

[0076] S6. Based on the final trajectory point of the unmanned ship, the unmanned ship operation instruction is generated through PID control. This step is mainly used to obtain the future T p Predicted path track point position information of the frame Used for PID control to generate navigation instructions for unmanned ships. Specifically include:

[0077] S601, after the previous stage is completed, all parts of the network framework have been completed, and the track point generated from the ship is the final output of the network framework of the present invention. By sending it into the PID control, the automatic driving instructions (steering, throttle, etc.) of the unmanned ship at the current moment can be generated.

[0078] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. An unmanned ship automatic control method based on multimodal feature fusion, characterized in that: include: S1, acquiring a multi-view RGB image and a radar data image, performing perspective conversion on the multi-view RGB image and the radar data image, and then extracting BEV features respectively, thereby obtaining visual BEV features and radar BEV features; S2, encoding and fusing the visual BEV feature and the radar BEV feature to generate a fused BEV feature; S3, performing target detection on the fused BEV feature through a target sensor, thereby obtaining valid target features, each of which represents a valid target, and the valid targets include the current unmanned ship and other targets; S4, processing the effective target features and the fused BEV features through a motion prediction module, thereby predicting the future trajectory of the effective target; S5, generating a first self-query of the unmanned ship based on the target sensor, and a second self-query of the current unmanned ship based on the motion prediction module, taking the first self-query and the second self-query as inputs of the planner module, interacting the combined generated aggregated plan query with the fused BEV feature, and predicting the final trajectory point of the unmanned ship; S6. Based on the final trajectory point of the unmanned ship, the unmanned ship operation instruction is generated through PID control.

2. The unmanned ship automatic control method based on multimodal feature fusion according to claim 1 is characterized in that: A multi-view RGB image and a radar data image are acquired, and after performing view conversion on the multi-view RGB image and the radar data image, BEV feature extraction is performed respectively, so as to obtain visual BEV features and radar BEV features, including: S101, using Swin-T as an RGB image feature extractor to process the input RGB image, performing multi-scale fusion and downsampling to output RGB image features through FPN; converting the acquired RGB image features into BEV space and performing BEVpooling processing to obtain visual BEV features; S102, using VoxelNet as a radar data feature extractor to process the input radar data image, directly performing BEV conversion processing on the radar feature map output by the radar data feature extractor, and then generating radar BEV features.

3. The unmanned ship automatic control method based on multimodal feature fusion according to claim 1 is characterized in that: Encoding and fusing the visual BEV feature and the radar BEV feature to generate a fused BEV feature, including: S201, performing dimension splicing on the visual BEV features and the radar BEV features; S202: Using a convolution-based BEV encoder to compensate for the misalignment in the process of splicing visual BEV features and radar BEV features, thereby generating a fused BEV feature.

4. The unmanned ship automatic control method based on multimodal feature fusion according to claim 1 is characterized in that: The target sensor includes N T Layer Transformer Encoder, at the beginning of each training sequence, all queries are detection queries. The GrandTurth corresponding to the detection query is matched through the Hungarian algorithm. The query interaction module stores and updates the successfully matched detection queries and uses them as tracking queries for the next timestamp of TrackFormer. In the next timestamp, the detection query is still matched with the GrandTruth target through the Hungarian algorithm, while the tracking query will directly match a part of the GrandTruth target according to the corresponding track ID.

5. The unmanned ship automatic control method based on multimodal feature fusion according to claim 1 is characterized in that: The motion prediction module includes N A Layers, each layer contains two perceptrons: target-target and target-whole; The target-target sensor is: Q a =MHCA(MHSA(Q),Q A ) Among them, MHCA and MHSA represent the multi-head cross attention module and the multi-head self-attention module, Q represents the initialization learnable query, and Q a It seeks the interactive perception characteristics between the current unmanned ship and other targets; The target-holistic perceptron is: in, DeformAttn(q,r,x) represents the predicted trajectory point of the previous layer. DeformAttn(q,r,x) is a deformable attention module. The reference point r and the spatial feature x are used to obtain the learnable query q. It is spatially represented by sparsely paying attention to the features around the reference point. cur Represents the fused BEV features corresponding to the current frame (t=cur).

6. The unmanned ship automatic control method based on multimodal feature fusion according to claim 5 is characterized in that: The effective target features and the fused BEV features are processed by a motion prediction module to predict the future trajectory of the effective target, including: S401, the Q generated by the two perceptrons of target-target and target-overall a and Q g After splicing, it is passed to two different multi-layer perceptrons (MLP) to generate predicted trajectory points respectively. Query context Then Q ctx Send to the next layer as input for Q initialization; S402: The input query Q of each layer includes two parts: the query context Q generated by the previous layer ctx And query location Q pos , where Q pos Obtained by the following calculation Among them, PE(·) is the sinusoidal position encoding, MLP is used to encode the position points, and I s is the scene-level anchor position, I a is the target-level anchor, It is the starting point. is the predicted endpoint; S403, after N A After the motion prediction module of the layer performs action perception prediction, it predicts the multimodal future motion trajectories of all valid targets.

7. The unmanned ship automatic control method based on multimodal feature fusion according to claim 1 is characterized in that: The first self-query and the second self-query are combined as inputs of the planner module to generate an aggregated plan query and interact with the fused BEV features to predict the final trajectory point of the unmanned ship, including: The decoder cross-perceives the aggregated plan query and the fused BEV features, combines the BEV features with the sinusoidal position embedding, and then regresses the planned trajectory using the MLP layer. The prediction is recorded as Indicates the future T p The predicted path track point position information of the frame.

8. The unmanned ship automatic control method based on multimodal feature fusion according to claim 1 is characterized in that: Based on the final track point of the unmanned ship, the unmanned ship operation instructions are generated through PID control, including: By sending it into PID control, the automatic control instructions for the unmanned ship at the current moment can be generated.

Images