Autonomous cruising method and system for unmanned ships based on multi-module fusion

Abstract

The present application relates to the field of autonomous cruising of unmanned ships, and specifically to a method and system for autonomous cruising of unmanned ships based on multi-module fusion. It collects navigation visible light images through camera monitoring, and collects navigation millimeter wave detection images through millimeter wave radar monitoring, and then introduces image processing and analysis algorithms based on artificial intelligence and deep learning at the back end to perform feature analysis and extraction on the navigation visible light images and navigation millimeter wave detection images, so as to capture and characterize the environmental perception features and navigation status multi-module cross-domain fusion features of the unmanned ship in the navigation visible light images and the navigation millimeter wave detection images, so as to perform environmental perception of the cruising process to determine whether to avoid obstacles, which can help the unmanned ship identify obstacles, optimize its obstacle avoidance strategy for autonomous cruising, and plan an optimized cruising route that effectively avoids obstacles and saves power.

Description

Technical Field

[0001] The present application relates to the field of autonomous cruising of unmanned ships, and specifically to an autonomous cruising method and system for unmanned ships based on multi-module fusion. Background Art

[0002] With the help of precise positioning technology, multi-sensor technology and communication technology, unmanned ships can autonomously cruise on the water surface according to preset routes and avoid obstacles according to preset tasks. Autonomous cruising unmanned ships have played an important role in many fields such as water conservancy environmental monitoring and environmental water sampling. The key to the unmanned ship's ability to automatically complete the predetermined task lies in its autonomous navigation capability. The autonomous navigation capability of the unmanned ship not only depends on its efficient power system and control algorithm, but also is closely related to its environmental perception ability. Environmental perception is the basis for unmanned ships to safely and efficiently navigate in complex waters, especially in the presence of obstacles.

[0003] For example, a Chinese patent with application number 202111528825.7 discloses an unmanned ship obstacle avoidance system based on image recognition technology, including an image acquisition module, an image filtering module, an image recognition module and a control module; the image acquisition module is used to acquire an image in the direction of the unmanned ship's advance; the image filtering module is used to determine whether the image in the direction of the unmanned ship's advance meets the preset filtering conditions; the image recognition module is used to perform image recognition processing on the image in the direction of the unmanned ship's advance transmitted by the image filtering module, and determine whether the image in the direction of the unmanned ship's advance contains an obstacle, and if so, calculate the positional relationship between the obstacle and the unmanned ship; the control module is used to control the unmanned ship according to the positional relationship. This invention is conducive to avoiding low-quality images from entering the subsequent image recognition process, and is conducive to improving the accuracy of the obstacle recognition results, and avoiding shortening the unmanned ship's response time to obstacles due to invalid operations in image processing.

[0004] However, traditional environmental perception methods mainly rely on single sensors, such as cameras, lidar or sonar, which often face many challenges in the changing water environment. For example, cameras may not perform well in low light or bad weather conditions, and millimeter-wave radars may also have limitations in detecting small or low-reflective objects, have certain blind spots, and are not sensitive enough when avoiding obstacles.

[0005] Therefore, an unmanned ship autonomous cruising solution based on multi-module fusion is needed to solve the above technical problems. Summary of the invention

[0006] This application is made in consideration of the above problems. One object of this application is to provide an unmanned ship autonomous cruising method and system based on multi-module fusion.

[0007] The embodiment of the present application provides an unmanned ship autonomous cruising method based on multi-module fusion, which includes:

[0008] Acquire navigation visible light images collected by a camera;

[0009] Acquire navigation millimeter-wave detection images collected by millimeter-wave radar;

[0010] Performing a multi-scale feature scan based on the navigation condition on the navigation visible light image to obtain a navigation condition visible light shallow feature map and a navigation condition visible light deep feature map;

[0011] Inputting the navigation condition visible light shallow feature map and the navigation condition visible light deep feature map into a bidirectional global fine-grained attention feature joint module to obtain a navigation condition visible light shallow-deep joint perception feature map;

[0012] Based on the navigation millimeter wave detection image, a depth explicit depiction of the heading condition is performed to obtain a semantic feature vector of the navigation condition depth millimeter wave detection;

[0013] Input the navigation status visible light shallow-deep joint perception feature map and the navigation status deep millimeter wave detection semantic feature vector into a gated Transformer structure encoder based on cross-modal information fusion to obtain a navigation status multi-module cross-domain fusion feature map;

[0014] Based on the multi-module cross-domain fusion feature map of the navigation status, a cruise instruction is generated, and the cruise instruction is used to indicate whether to perform obstacle avoidance.

[0015] For example, according to the unmanned ship autonomous cruising method based on multi-module fusion of the embodiment of the present application, wherein the navigation visible light image is subjected to a multi-scale feature scanning based on the navigation condition to obtain a navigation condition visible light shallow feature map and a navigation condition visible light deep feature map, comprising: inputting the navigation visible light image into a navigation condition multi-scale feature scanner based on a hollow pyramid network to obtain the navigation condition visible light shallow feature map and the navigation condition visible light deep feature map.

[0016] For example, according to the unmanned ship autonomous cruising method based on multi-module fusion according to an embodiment of the present application, the navigation condition visible light shallow feature map and the navigation condition visible light deep feature map are input into a two-way global fine-grained attention feature joint module to obtain a navigation condition visible light shallow-deep joint perception feature map, including:

[0017] Performing feature fine-grained deconstruction on the navigation condition visible light shallow feature map and the navigation condition visible light deep feature map to obtain a set of navigation condition visible light shallow channel dimension local feature matrices and a set of navigation condition visible light deep channel dimension local feature matrices;

[0018] Using each local feature matrix of the shallow channel dimension of visible light for navigation conditions in the set of local feature matrices of the shallow channel dimension of visible light for navigation conditions as a query feature matrix, using the set of local feature matrices of the deep channel dimension of visible light for navigation conditions as a set of key feature matrices, inputting the query feature matrix and the set of key feature matrices into a one-way global attention interaction module based on the first converter structure to obtain a set of local feature matrices of the shallow channel dimension of visible light for navigation conditions optimized by one-way global attention;

[0019] Using each local feature matrix of the deep channel dimension of visible light for navigation conditions in the set of local feature matrices of the deep channel dimension of visible light for navigation conditions as a query feature matrix, using the set of local feature matrices of the shallow channel dimension of visible light for navigation conditions as a set of key feature matrices, inputting the query feature matrix and the set of key feature matrices into a one-way global attention interaction module based on a second converter structure to obtain a set of local feature matrices of the shallow channel dimension of visible light for navigation conditions optimized by one-way global attention;

[0020] The set of local feature matrices of the shallow channel dimension of the visible light of the one-way global attention optimization navigation condition and the set of local feature matrices of the shallow channel dimension of the visible light of the one-way global attention optimization navigation condition are coupled along the channel dimension to obtain a shallow feature map of the visible light of the one-way global interaction optimization navigation condition and a deep feature map of the visible light of the one-way global interaction optimization navigation condition;

[0021] The position-weighted sum between the one-way global interactive optimization of the navigation condition visible light shallow feature map and the one-way global interactive optimization of the navigation condition visible light deep feature map is calculated to obtain the navigation condition visible light shallow-deep joint perception feature map.

[0022] For example, according to the unmanned ship autonomous cruising method based on multi-module fusion of an embodiment of the present application, each local feature matrix of the shallow channel dimension of the visible light of the navigation condition in the set of local feature matrices of the shallow channel dimension of the visible light of the navigation condition is used as a query feature matrix, and the set of local feature matrices of the deep channel dimension of the visible light of the navigation condition is used as a set of key feature matrices, and the query feature matrix and the set of key feature matrices are input into a one-way global attention interaction module based on a first converter structure to obtain a one-way global attention optimized navigation condition visible light shallow channel dimension local feature matrix set, including:

[0023] Selecting a predetermined navigation condition visible light shallow channel dimension local feature matrix from the set of navigation condition visible light shallow channel dimension local feature matrix as a query feature matrix;

[0024] Calculate the product of the predetermined local feature matrix of the shallow channel dimension of visible light of the navigation condition and the transposed matrix of each local feature matrix of the deep channel dimension of visible light of the navigation condition in the set of local feature matrices of the deep channel dimension of visible light of the navigation condition to obtain a set of local semantic interaction feature matrices of the shallow-deep channel dimension of visible light of the navigation condition;

[0025] After dividing the scale square root of the predetermined local feature matrix of the shallow channel dimension of visible light for navigation conditions by each local semantic interaction feature matrix of the shallow-deep channel dimension of visible light for navigation conditions in the set of local semantic interaction feature matrices of the shallow-deep channel dimension of visible light for navigation conditions at their respective positions, a softmax function is used to perform soft maximum normalization processing on each feature matrix in the obtained set of feature matrices to obtain a set of local weight matrices of the shallow channel dimension of visible light for navigation conditions;

[0026] Taking the set of local weight matrices of the navigation condition visible light shallow channel dimension as weighted weights, the position-weighted sum of the local feature matrices of the navigation condition visible light shallow channel dimension in the set of local feature matrices of the navigation condition visible light shallow channel dimension is calculated to obtain the one-way global attention optimized navigation condition visible light shallow channel dimension local feature matrix.

[0027] For example, according to the unmanned ship autonomous cruising method based on multi-module fusion of an embodiment of the present application, each local feature matrix of the deep channel dimension of the visible light of the navigation condition in the set of the local feature matrix of the deep channel dimension of the visible light of the navigation condition is used as a query feature matrix, and the set of the local feature matrix of the shallow channel dimension of the visible light of the navigation condition is used as a set of key feature matrices, and the query feature matrix and the set of the key feature matrix are input into a one-way global attention interaction module based on the second converter structure to obtain a one-way global attention optimized navigation condition visible light shallow channel dimension local feature matrix set, including:

[0028] Selecting a predetermined navigation condition visible light deep channel dimension local feature matrix from the set of navigation condition visible light deep channel dimension local feature matrices as a query feature matrix;

[0029] Calculate the product of the predetermined local feature matrix of the deep channel dimension of visible light for navigation conditions and the transposed matrix of each local feature matrix of the shallow channel dimension of visible light for navigation conditions in the set of local feature matrices of the shallow channel dimension of visible light for navigation conditions to obtain a set of local semantic interaction feature matrices of the deep-shallow channel dimension of visible light for navigation conditions;

[0030] After dividing the scale square root of the predetermined local feature matrix of the deep channel dimension of visible light for navigation conditions by each local semantic interaction feature matrix of the deep-shallow channel dimension of visible light for navigation conditions at each position point, each feature matrix in the obtained set of feature matrices is subjected to soft maximum normalization processing using a softmax function to obtain a set of local weight matrices of the deep channel dimension of visible light for navigation conditions;

[0031] Taking the set of local weight matrices of the navigation condition visible light deep channel dimension as weighted weights, the position-weighted sum of the local feature matrices of the navigation condition visible light deep channel dimension in the set of local feature matrices of the navigation condition visible light deep channel dimension is calculated to obtain the one-way global attention optimized local feature matrix of the navigation condition visible light deep channel dimension.

[0032] For example, according to the unmanned ship autonomous cruising method based on multi-module fusion of an embodiment of the present application, a deep explicit depiction of the heading condition is performed based on the navigation millimeter-wave detection image to obtain a navigation condition deep millimeter-wave detection semantic feature vector, comprising: inputting the navigation millimeter-wave detection image into a heading condition deep explicit depictor based on a deep separable convolutional network to obtain the navigation condition deep millimeter-wave detection semantic feature vector.

[0033] For example, according to the unmanned ship autonomous cruising method based on multi-module fusion of an embodiment of the present application, the navigation status visible light shallow-deep joint perception feature map and the navigation status deep millimeter wave detection semantic feature vector are input into a gated Transformer structure encoder based on cross-modal information fusion to obtain a navigation status multi-module cross-domain fusion feature map, including:

[0034] Performing layer normalization and convolution coding processing on the navigation condition visible light shallow layer-deep layer joint perception feature map to obtain the navigation condition visible light shallow layer-deep layer joint perception spatial deepening feature map;

[0035] Copying the navigation condition visible light shallow-deep joint perception space deepening feature map to obtain a backup navigation condition visible light shallow-deep joint perception space deepening feature map;

[0036] Reshape the backup navigation condition visible light shallow-deep joint perception space deepening feature map and the navigation condition visible light shallow-deep joint perception space deepening feature map respectively to obtain the backup navigation condition visible light shallow-deep joint perception space deepening feature matrix as the simulated query feature matrix and the navigation condition visible light shallow-deep joint perception space deepening feature matrix as the simulated key feature matrix;

[0037] Calculating the product of the pseudo-query feature matrix and the transposed matrix of the pseudo-key feature matrix to obtain a navigation condition visible light shallow-deep joint perception local autocorrelation library matrix;

[0038] Inputting the navigation condition visible light shallow-deep joint perception local autocorrelation library matrix into the attention gating unit based on the GELU function to obtain the navigation condition visible light shallow-deep joint perception gated local autocorrelation library weight matrix;

[0039] Taking the navigation condition depth millimeter wave detection semantic feature vector as the value feature vector, calculating the product between the navigation condition visible light shallow-deep joint perception gated local autocorrelation library weight matrix and the navigation condition depth millimeter wave detection semantic feature vector to obtain the navigation condition depth millimeter wave detection semantic cross-modal interactive fusion condition weight vector;

[0040] The navigation status deep millimeter wave detection semantic cross-modal interactive fusion condition weight vector is used as the weight vector, and the navigation status visible light shallow-deep joint perception feature map is weighted channel by channel to obtain the navigation status multi-module cross-domain fusion feature map.

[0041] For example, according to the unmanned ship autonomous cruising method based on multi-module fusion of an embodiment of the present application, the navigation condition visible light shallow-deep joint perception feature map is layer normalized and convolutionally encoded to obtain the navigation condition visible light shallow-deep joint perception spatial deepening feature map, including:

[0042] Performing layer normalization processing on the navigation condition visible light shallow layer-deep layer joint perception feature map to obtain a layer normalized navigation condition visible light shallow layer-deep layer joint perception feature map;

[0043] The layer-normalized navigation condition visible light shallow-layer-deep joint perception feature map is point-convolutionally encoded and then spatially convolutionally encoded to obtain the navigation condition visible light shallow-layer-deep joint perception spatial deepening feature map.

[0044] For example, according to the unmanned ship autonomous cruising method based on multi-module fusion of an embodiment of the present application, a cruise instruction is generated based on the multi-module cross-domain fusion feature map of the navigation state, and the cruise instruction is used to indicate whether to perform obstacle avoidance, including: inputting the multi-module cross-domain fusion feature map of the navigation state into a cruise instruction generation module based on a classifier to obtain the cruise instruction.

[0045] The embodiment of the present application also provides an unmanned ship autonomous cruising system based on multi-module fusion, which includes:

[0046] A navigation visible light image acquisition module is used to acquire the navigation visible light image acquired by the camera;

[0047] A navigation millimeter wave detection image acquisition module is used to obtain a navigation millimeter wave detection image collected by a millimeter wave radar;

[0048] A multi-scale feature scanning module, used for performing a multi-scale feature scanning on the navigation visible light image based on the navigation condition to obtain a navigation condition visible light shallow feature map and a navigation condition visible light deep feature map;

[0049] A shallow-deep joint perception module is used to input the navigation condition visible light shallow feature map and the navigation condition visible light deep feature map into a two-way global fine-grained attention feature joint module to obtain a navigation condition visible light shallow-deep joint perception feature map;

[0050] A heading condition depth explicit depiction module, used for performing heading condition depth explicit depiction based on the navigation millimeter wave detection image to obtain a navigation condition depth millimeter wave detection semantic feature vector;

[0051] A cross-modal information fusion module is used to input the navigation status visible light shallow-deep joint perception feature map and the navigation status deep millimeter wave detection semantic feature vector into a gated Transformer structure encoder based on cross-modal information fusion to obtain a navigation status multi-module cross-domain fusion feature map;

[0052] A cruise instruction generation module is used to generate a cruise instruction based on the multi-module cross-domain fusion feature map of the navigation status, and the cruise instruction is used to indicate whether to perform obstacle avoidance.

[0053] The present application has at least the following technical effects: Compared with the prior art, the unmanned ship autonomous cruising method and system based on multi-module fusion of the embodiments of the present application monitor and collect navigation visible light images through cameras, and monitor and collect navigation millimeter-wave detection images through millimeter-wave radars, and then introduce image processing and analysis algorithms based on artificial intelligence and deep learning at the back end to perform feature analysis and extraction on the navigation visible light images and navigation millimeter-wave detection images, so as to capture and characterize the environmental perception features and navigation status multi-module cross-domain fusion features of the unmanned ship in the navigation visible light images and the navigation millimeter-wave detection images, so as to perform environmental perception of the cruising process to determine whether to avoid obstacles, which can help the unmanned ship identify obstacles, optimize its obstacle avoidance strategy for autonomous cruising, and plan an optimized cruising route that effectively avoids obstacles and saves power. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings of the embodiments of the present application are briefly introduced below. Obviously, the drawings described below only relate to some embodiments of the present application, and are not intended to limit the present application.

[0055] Figure 1 A schematic diagram of the application architecture of the unmanned ship autonomous cruising method based on multi-module fusion in an embodiment of the present application is shown;

[0056] Figure 2 A flow chart of an unmanned ship autonomous cruising method based on multi-module fusion in an embodiment of the present application is shown;

[0057] Figure 3 A flowchart of sub-step S540 of the unmanned ship autonomous cruising method based on multi-module fusion in an embodiment of the present application is shown;

[0058] Figure 4 A schematic diagram of the structure of an unmanned ship autonomous cruising system based on multi-module fusion in an embodiment of the present application is shown. DETAILED DESCRIPTION

[0059] The technical solutions in the embodiments of the present application will be described clearly and completely below in conjunction with the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, not all embodiments. Based on the embodiments of the present application, all other embodiments obtained by ordinary technicians in this field without creative work also fall within the scope of protection of the present application.

[0060] The terms used in this specification are those common terms currently widely used in the art in consideration of the functions of the present application, but these terms may vary according to the intentions of those skilled in the art, precedents, or new technologies in the art. In addition, specific terms may be selected, and in this case, their detailed meanings will be described in the detailed description of the present application. Therefore, the terms used in the specification should not be understood as simple names, but rather as a general description based on the meaning of the terms and the present application.

[0061] Although the present application makes various references to certain modules in the system according to the embodiments of the present application, any number of different modules can be used and run on the user terminal and / or server. The modules are only illustrative, and different aspects of the system and method can use different modules.

[0062] Flowcharts are used in the present application to illustrate the operations performed by the system according to the embodiments of the present application. It should be understood that the preceding or following operations are not necessarily performed accurately in order. On the contrary, various steps may be processed in reverse order or simultaneously as required. At the same time, other operations may also be added to these processes, or a certain step or several steps of operations may be removed from these processes.

[0063] Figure 1 A schematic diagram of the application architecture of an unmanned ship autonomous cruising method based on multi-module fusion in an embodiment of the present application is shown, including a server 100 and a terminal device 200.

[0064] The terminal device 200 and the server 100 can be connected via the Internet to achieve mutual communication. Optionally, the above-mentioned Internet uses standard communication technology and / or protocol. The Internet is usually the Internet, but it can also be any network, including but not limited to any combination of a local area network (LAN), a metropolitan area network (MAN), a wide area network (WAN), a mobile, wired or wireless network, a private network or a virtual private network. In some embodiments, the data exchanged through the network can be represented by technologies and / or formats including Hyper Text Markup Language (HTML), Extensible Markup Language (XML), etc. In addition, conventional encryption technologies such as Secure Socket Layer (SSL), Transport Layer Security (TLS), Virtual Private Network (VPN), Internet Protocol Security (IPsec), etc. can be used to encrypt all or some links. In other embodiments, customized and / or dedicated data communication technologies can also be used to replace or supplement the above-mentioned data communication technologies.

[0065] The server 100 can provide various network services for the terminal device 200, wherein the server 100 can be a single server, a server cluster consisting of several servers, or a cloud computing center. Specifically, the server 100 may include a processor 110 (Center Processing Unit, CPU), a memory 120, an input device 130, and an output device 140, etc. The input device 130 may include a keyboard, a mouse, a touch screen, etc., and the output device 140 may include a display device, such as a liquid crystal display (Liquid Crystal Display, LCD), a cathode ray tube (Cathode Ray Tube, CRT), etc.

[0066] The memory 120 may include a read-only memory (ROM) and a random access memory (RAM), and provides the processor 110 with program instructions and data stored in the memory 120. In an embodiment of the present application, the memory 120 may be used to store a program of the unmanned ship autonomous cruising method based on multi-module fusion in an embodiment of the present application.

[0067] The processor 110 calls the program instructions stored in the memory 120, and the processor 110 is used to execute the steps of any unmanned ship autonomous cruising method based on multi-module fusion in the embodiments of the present application according to the obtained program instructions.

[0068] In addition, the application architecture diagram in the embodiment of the present application is intended to more clearly illustrate the technical solution in the embodiment of the present application, and does not constitute a limitation on the technical solution provided in the embodiment of the present application. Of course, for other application architectures and business applications, the technical solution provided in the embodiment of the present application is also applicable to similar problems.

[0069] The following is a non-restrictive description of the unmanned ship autonomous cruising method based on multi-module fusion provided according to at least one embodiment of the present application through several examples or embodiments. As described below, different features in these specific examples or embodiments can be combined with each other without conflicting with each other to obtain new examples or embodiments, and these new examples or embodiments also fall within the scope of protection of the present application.

[0070] In response to the above technical problems, in the technical solution of the present application, a method for autonomous cruising of an unmanned ship based on multi-module fusion is proposed, which can adopt a variety of sensor fusion technologies to autonomously realize environmental perception and cruise auxiliary control of the unmanned ship, so as to achieve effective obstacle avoidance when the unmanned ship is cruising.

[0071] Among them, since the camera can capture high-resolution visual information, it helps to identify important features such as static or dynamic obstacles, other ships, and shore boundaries; while the millimeter-wave radar can provide reliable detection information in various weather conditions, especially in low visibility conditions, the millimeter-wave radar can still maintain effective target detection capabilities. These sensors can provide different perception information to help unmanned ships understand the surrounding environment. Therefore, combining the information of cameras and millimeter-wave radars can improve the adaptability and robustness of unmanned ships to complex environments.

[0072] Based on this, the technical concept of the present application is to monitor and collect navigation visible light images through cameras, and to monitor and collect navigation millimeter wave detection images through millimeter wave radars, and then introduce image processing and analysis algorithms based on artificial intelligence and deep learning at the back end to perform feature analysis and extraction on the navigation visible light images and navigation millimeter wave detection images, so as to capture and characterize the environmental perception features and multi-module cross-domain fusion features of the unmanned ship in the navigation visible light images and the navigation millimeter wave detection images, so as to perform environmental perception of the cruising process to determine whether to avoid obstacles. By adopting a variety of sensor fusion technologies, the environmental perception information of the unmanned ship, such as obstacles, can be obtained in real time to generate cruise instructions to optimize the speed and direction of the unmanned ship to achieve obstacle avoidance. In this way, it can not only help the unmanned ship identify obstacles, but also optimize its obstacle avoidance strategy for autonomous cruising, plan an optimized cruise route that effectively avoids obstacles and saves power, and improve the cruising capability of the unmanned ship.

[0073] Figure 2 The flowchart of the unmanned ship autonomous cruising method based on multi-module fusion in the embodiment of the present application is shown. For example, the unmanned ship autonomous cruising method based on multi-module fusion can be executed by a server, which can be Figure 1 The server 100 shown in FIG. Figure 2 As shown, according to the embodiment of the present application, the autonomous cruising method of an unmanned ship based on multi-module fusion includes the following steps: S510, obtaining a navigation visible light image collected by a camera; S520, obtaining a navigation millimeter-wave detection image collected by a millimeter-wave radar; S530, performing a multi-scale feature scan based on the navigation conditions on the navigation visible light image to obtain a navigation condition visible light shallow feature map and a navigation condition visible light deep feature map; S540, inputting the navigation condition visible light shallow feature map and the navigation condition visible light deep feature map into a two-way global fine-grained attention feature joint module to obtain a navigation condition visible light shallow feature map. -deep joint perception feature map; S550, based on the navigation millimeter-wave detection image, perform a deep explicit depiction of the heading status to obtain a navigation status deep millimeter-wave detection semantic feature vector; S560, input the navigation status visible light shallow-deep joint perception feature map and the navigation status deep millimeter-wave detection semantic feature vector into a gated Transformer structure encoder based on cross-modal information fusion to obtain a navigation status multi-module cross-domain fusion feature map; S570, based on the navigation status multi-module cross-domain fusion feature map, generate a cruise instruction, and the cruise instruction is used to indicate whether to perform obstacle avoidance.

[0074] Specifically, in the technical solution of the present application, first, a navigation visible light image collected by a camera is obtained, and a navigation millimeter wave detection image collected by a millimeter wave radar is obtained. Then, in order to more fully capture the navigation status features in the navigation visible light image, in the technical solution of the present application, the navigation visible light image is input into a navigation status multi-scale feature scanner based on a dilated pyramid network to obtain a navigation status visible light shallow feature map and a navigation status visible light deep feature map. Among them, the dilated pyramid network is a network structure commonly used for semantic segmentation and object detection. It captures multi-scale context information in the image by using dilated convolutions (also called expanded convolutions) with different sampling rates. The advantage of doing so is that objects or scene details of different sizes can be effectively processed, thereby improving the detection accuracy of the model for targets of different scales. Therefore, through the processing of the navigation status multi-scale feature scanner based on the dilated pyramid network, the navigation visible light image can be scanned based on multi-scale features, so as to capture multi-scale and multi-level feature information about the navigation status of the unmanned ship in the image. The shallow feature map of the navigation status visible light is usually a feature map containing more local information such as edges and textures, which helps to quickly locate the boundaries and other details of objects in the image. The deep feature map of the navigation status visible light contains more abstract and global feature information, which helps to identify and classify objects in the image, such as other ships, obstacles, etc.

[0075] Correspondingly, in step S530, the navigation visible light image is subjected to a multi-scale feature scan based on the navigation conditions to obtain a navigation condition visible light shallow feature map and a navigation condition visible light deep feature map, including: inputting the navigation visible light image into a navigation condition multi-scale feature scanner based on a hollow pyramid network to obtain the navigation condition visible light shallow feature map and the navigation condition visible light deep feature map.

[0076] Among them, since the navigation condition visible light shallow feature map and the navigation condition visible light deep feature map respectively contain the shallow local information and deep global semantic information about the navigation condition in the navigation visible light image. In order to integrate the information in these two feature maps, so as to obtain a more complete and accurate environmental perception result, in the technical solution of the present application, the navigation condition visible light shallow feature map and the navigation condition visible light deep feature map are further input into the bidirectional global fine-grained attention feature joint module to obtain the navigation condition visible light shallow-deep joint perception feature map. The bidirectional global fine-grained attention feature joint module aims to improve the semantic joint perception ability of the model through feature fine-grained deconstruction, global fine-grained interaction based on attention mechanism, feature coupling and feature fusion, so as to better perceive the navigation condition and identify obstacles. Specifically, the process of fine-grained deconstruction along the channel dimension allows the model to perform more detailed segmentation and semantic understanding of the navigation condition visible light shallow feature map and the navigation condition visible light deep feature map, so as to capture more local details of the environmental condition, which helps the subsequent attention mechanism to more accurately identify important cruising environmental condition features. Then, after fine-grained deconstruction of the feature map, the unidirectional global attention interaction module based on the transformer structure can strengthen the relationship between different features in the navigation visible light image by utilizing the multi-head attention mechanism in the Transformer structure to measure the correlation between different feature matrices, so that the model can pay more attention to the information-rich feature areas and ignore irrelevant background information, so that the model can more accurately identify key information in complex environments, such as obstacle-related objects.

[0077] In particular, in the technical solution of the present application, each local feature matrix of the shallow channel dimension of the navigation condition visible light of the navigation condition shallow feature map can respectively perform one-way semantic interaction with all local feature matrices of the deep channel dimension of the navigation condition visible light of the navigation condition deep feature map so as to more comprehensively capture the global semantic association information between each local feature matrix of the shallow channel dimension of the navigation condition visible light of the navigation condition shallow feature map and the deep feature map of the navigation condition visible light. Similarly, each local feature matrix of the deep channel dimension of the navigation condition visible light of the navigation condition deep feature map can respectively perform one-way semantic interaction with all local feature matrices of the shallow channel dimension of the navigation condition visible light of the navigation condition shallow feature map so as to more comprehensively capture the global semantic association information between each local feature matrix of the deep channel dimension of the navigation condition visible light of the navigation condition deep feature map and the shallow feature map of the navigation condition visible light. Among them, the effect of the one-way global attention interaction is to enhance the fine-grained channel relationship within the feature map, so that the model can pay more attention to the key information related to the navigation status and obstacle recognition tasks in the feature map, improve the quality and discrimination of feature representation, and thus help improve the performance of the model in subsequent obstacle recognition and obstacle avoidance visual tasks.

[0078] Then, through the coupling processing along the channel dimension, the local feature matrices of the shallow and deep layers of the visible light of the navigation condition that have been optimized by attention can be recombined along the channel dimension to form a complete feature map, providing a complete feature representation for the final fusion. Finally, the weighted sum by position between the shallow feature map of the visible light of the navigation condition optimized by the one-way global interaction and the deep feature map of the visible light of the navigation condition optimized by the one-way global interaction is calculated to obtain the shallow-deep joint perception feature map of the visible light of the navigation condition. That is, the weighted sum operation integrates the key information in the two feature maps and generates a joint perception feature representation of the navigation condition containing rich interactive information. In particular, the final generated "shallow-deep joint perception feature map of the visible light of the navigation condition" is a result of the fusion of shallow and deep feature information. This joint perception method allows the system to perceive specific objects and details in the process of environmental perception, as well as the overall environmental layout and structure, so as to make more accurate obstacle avoidance and navigation decisions, which helps to improve the safety and reliability of cruising.

[0079] Accordingly, in step S540, if Figure 3As shown, the navigation condition visible light shallow feature map and the navigation condition visible light deep feature map are input into a bidirectional global fine-grained attention feature joint module to obtain a navigation condition visible light shallow-deep joint perception feature map, including: S541, feature fine-grained deconstruction of the navigation condition visible light shallow feature map and the navigation condition visible light deep feature map to obtain a set of navigation condition visible light shallow channel dimension local feature matrices and a set of navigation condition visible light deep channel dimension local feature matrices; S542, each navigation condition visible light shallow channel dimension local feature matrix in the set of navigation condition visible light shallow channel dimension local feature matrices is used as a query feature matrix, and the set of the navigation condition visible light deep channel dimension local feature matrix is ​​used as a set of key feature matrices, and the query feature matrix and the set of the key feature matrix are input into a unidirectional global attention interaction module based on the first converter structure to obtain a set of unidirectional global attention optimized navigation condition visible light shallow channel dimension local feature matrices; S543, each navigation condition visible light deep channel dimension local feature matrix is ​​used as a query feature matrix, and the set of the navigation condition visible light deep channel dimension local feature matrix is ​​used as a key feature matrix. Each local feature matrix of the deep channel dimension of visible light for navigation conditions in the set of local feature matrices is used as a query feature matrix, and the set of local feature matrices of the shallow channel dimension of visible light for navigation conditions is used as a set of key feature matrices, and the query feature matrix and the set of key feature matrices are input into a one-way global attention interaction module based on a second converter structure to obtain a set of local feature matrices of the shallow channel dimension of visible light for navigation conditions that are optimized by one-way global attention; S544, respectively, the set of local feature matrices of the shallow channel dimension of visible light for navigation conditions that are optimized by one-way global attention and the set of local feature matrices of the shallow channel dimension of visible light for navigation conditions that are optimized by one-way global attention are coupled along the channel dimension to obtain a one-way global interaction optimization of the shallow feature map of visible light for navigation conditions and a one-way global interaction optimization of the deep feature map of visible light for navigation conditions; S545, the position-weighted sum between the one-way global interaction optimization of the shallow feature map of visible light for navigation conditions and the one-way global interaction optimization of the deep feature map of visible light for navigation conditions is calculated to obtain the shallow-deep joint perception feature map of visible light for navigation conditions.

[0080] Wherein, in step S542, each local feature matrix of the shallow channel dimension of visible light under navigation conditions in the set of local feature matrices of the shallow channel dimension of visible light under navigation conditions is used as a query feature matrix, and the set of local feature matrices of the deep channel dimension of visible light under navigation conditions is used as a set of key feature matrices, and the query feature matrix and the set of key feature matrices are input into a one-way global attention interaction module based on the first converter structure to obtain a set of local feature matrices of the shallow channel dimension of visible light under navigation conditions optimized by one-way global attention, including: selecting a predetermined local feature matrix of the shallow channel dimension of visible light under navigation conditions from the set of local feature matrices of the shallow channel dimension of visible light under navigation conditions as a query feature matrix; calculating the transposed matrices of the predetermined local feature matrix of the shallow channel dimension of visible light under navigation conditions and each local feature matrix of the deep channel dimension of visible light under navigation conditions in the set of local feature matrices of the deep channel dimension of visible light under navigation conditions; to obtain a set of local semantic interaction feature matrices of the shallow-deep channel dimension of visible light for navigation conditions; to divide the scale square root of the predetermined local feature matrix of the shallow channel dimension of visible light for navigation conditions in the set of local semantic interaction feature matrices of the shallow-deep channel dimension of visible light for navigation conditions by each local semantic interaction feature matrix of the shallow-deep channel dimension of visible light for navigation conditions according to the position point, and then to perform soft maximum normalization processing on each feature matrix in the obtained set of feature matrices using the softmax function to obtain a set of local weight matrices of the shallow channel dimension of visible light for navigation conditions; to use the set of local weight matrices of the shallow channel dimension of visible light for navigation conditions as the weighted weights, and to calculate the weighted sum by position between the local feature matrices of the shallow channel dimension of visible light for navigation conditions in the set of local feature matrices of the shallow channel dimension of visible light for navigation conditions to obtain the one-way global attention optimized local feature matrix of the shallow channel dimension of visible light for navigation conditions.

[0081] Wherein, in step S543, each local feature matrix of the deep channel dimension of visible light for navigation conditions in the set of local feature matrices of the deep channel dimension of visible light for navigation conditions is used as a query feature matrix, and the set of local feature matrices of the shallow channel dimension of visible light for navigation conditions is used as a set of key feature matrices, and the query feature matrix and the set of key feature matrices are input into a one-way global attention interaction module based on a second converter structure to obtain a set of local feature matrices of the shallow channel dimension of visible light for navigation conditions optimized by one-way global attention, including: selecting a predetermined local feature matrix of the deep channel dimension of visible light for navigation conditions from the set of local feature matrices of the deep channel dimension of visible light for navigation conditions as a query feature matrix; calculating the transposed matrices of the predetermined local feature matrix of the deep channel dimension of visible light for navigation conditions and each local feature matrix of the shallow channel dimension of visible light for navigation conditions in the set of local feature matrices of the shallow channel dimension of visible light for navigation conditions to obtain a set of local semantic interaction feature matrices of the deep-shallow channel dimension of visible light for navigation conditions; to divide the scale square root of the predetermined local feature matrix of the deep channel dimension of visible light for navigation conditions in the set of local semantic interaction feature matrices of the deep-shallow channel dimension of visible light for navigation conditions by each local semantic interaction feature matrix of the deep-shallow channel dimension of visible light for navigation conditions according to the position point, and then to perform soft maximum normalization processing on each feature matrix in the obtained set of feature matrices using the softmax function to obtain a set of local weight matrices of the deep channel dimension of visible light for navigation conditions; to use the set of local weight matrices of the deep channel dimension of visible light for navigation conditions as the weighted weights, and to calculate the weighted sum by position between the local feature matrices of the deep channel dimension of visible light for navigation conditions in the set of local feature matrices of the deep channel dimension of visible light for navigation conditions to obtain the one-way global attention optimized local feature matrix of the deep channel dimension of visible light for navigation conditions.

[0082] In a specific example, the navigation condition visible light shallow feature map and the navigation condition visible light deep feature map are input into a two-way global fine-grained attention feature joint module to obtain a navigation condition visible light shallow-deep joint perception feature map, including: inputting the navigation condition visible light shallow feature map and the navigation condition visible light deep feature map into the two-way global fine-grained attention feature joint module to process them with the following two-way global fine-grained feature joint formula to obtain the navigation condition visible light shallow-deep joint perception feature map; wherein the two-way global fine-grained feature joint formula is:

[0083]

[0084]

[0085]

[0086]

[0087]

[0088]

[0089]

[0090] in, and respectively represent the visible light shallow characteristic map of the navigation condition and the visible light deep characteristic map of the navigation condition, It is a feature fine-grained deconstruction operation. They are respectively the first, second, and third dimensions along the channel dimension in the visible light shallow feature map of the navigation condition. and The local feature matrix of the visible light shallow channel dimension for each navigation condition, They are respectively the first, second, and third dimensions along the channel dimension in the visible light deep feature map of the navigation condition. and The local feature matrix of the visible light deep channel dimension for each navigation condition, For the The scale of the local feature matrix of the visible light shallow channel dimension for each navigation condition, is matrix multiplication, is the softmax function, is the number of local feature matrices of the navigation condition visible light shallow channel dimension in the navigation condition visible light shallow feature map, For the said The one-way global attention optimization of the local feature matrix of the shallow channel dimension of the visible light in the navigation condition corresponds to the local feature matrix of the shallow channel dimension of the visible light in the navigation condition, For the The one-way global attention optimization of the local feature matrix of the shallow channel dimension of the visible light in the navigation condition corresponds to the local feature matrix of the shallow channel dimension of the visible light in the navigation condition, Indicates that the features are coupled along the channel dimension. and They are respectively the visible light shallow feature map of one-way global interactive optimization of navigation conditions and the visible light deep feature map of one-way global interactive optimization of navigation conditions. and are weighted hyper parameters of the shallow feature map of visible light under the one-way global interactive optimization of navigation conditions and the deep feature map of visible light under the one-way global interactive optimization of navigation conditions, respectively. It is a joint perception feature map of shallow and deep layers of visible light for the navigation condition.

[0091] Furthermore, since the operating frequency of the millimeter wave radar is relatively high, usually between 30GHz and 300GHz, it has a higher resolution and can detect smaller targets. In addition, the millimeter wave radar has the ability to penetrate rain and fog and can work in adverse weather conditions, which makes it an important component of the unmanned ship environment perception system. Therefore, in order to be able to extract meaningful features from the data provided by the millimeter wave radar so as to better understand and describe the navigation environment, in the technical solution of the present application, the navigation millimeter wave detection image is further input into the heading condition deep explicit depictor based on the deep separable convolutional network to obtain the navigation condition deep millimeter wave detection semantic feature vector. Through the processing of the heading condition deep explicit depictor based on the deep separable convolutional network, useful navigation condition features can be efficiently extracted from the navigation millimeter wave detection image. These features can describe the target objects and their attributes in the environment. These features are crucial for subsequent decision-making (such as obstacle avoidance, path planning, etc.).

[0092] Accordingly, in step S550, a depth explicit depiction of the heading condition is performed based on the navigation millimeter-wave detection image to obtain a navigation condition depth millimeter-wave detection semantic feature vector, including: inputting the navigation millimeter-wave detection image into a depth explicit depictor of the heading condition based on a deep separable convolutional network to obtain the navigation condition depth millimeter-wave detection semantic feature vector.

[0093] Among them, since the navigation condition visible light shallow-deep joint perception feature map and the navigation condition deep millimeter wave detection semantic feature vector respectively contain the visible light shallow-deep joint perception features and deep millimeter wave detection semantic features related to the navigation condition, these two features belong to information of different modalities, in order to be able to effectively fuse these different modal data information, thereby complementing their respective deficiencies and improving the unmanned ship's comprehensive understanding of the environment, so as to provide more comprehensive and reliable information, in the technical solution of the present application, the navigation condition visible light shallow-deep joint perception feature map and the navigation condition deep millimeter wave detection semantic feature vector are further input into the gated Transformer structure encoder based on cross-modal information fusion to obtain the navigation state multi-module cross-domain fusion feature map. The gated Transformer structure encoder based on cross-modal information fusion can achieve effective fusion of different modalities, such as visible light image features and millimeter wave detection semantic features, by imitating the self-attention mechanism of the Transformer structure and combining the gating unit.

[0094] Specifically, the gated Transformer structure encoder based on cross-modal information fusion first performs point convolution encoding and spatial convolution encoding on the navigation condition visible light shallow-deep joint perception feature map to extract deep spatial features. This can enhance the spatial expression ability of the feature map, allowing the model to better understand the spatial layout and structure in the visible light image, such as the positional relationship between the unmanned ship and other static or dynamic obstacles.

[0095] Subsequently, the shallow-deep joint perception feature map of the visible light of the navigation condition is copied and reshaped to obtain the query and key feature matrices. Next, by calculating the product of the query feature matrix and the transposed key feature matrix, a local autocorrelation library matrix is ​​constructed, which realizes the modal internal correlation of the perception features of the visible light image of the navigation condition, which can help the model identify which areas are important and which areas can be ignored. If there are obstacles in a certain area, the model will give a higher attention weight to better capture the information of these areas. The generated local autocorrelation library matrix constructs the correlation between the various features in the image, which helps the model understand the relationship between different objects.

[0096] Furthermore, the attention gating unit based on the GELU function is used to dynamically adjust the autocorrelation weights to achieve adaptive selection and enhancement of features. This attention gating mechanism not only enhances the expressive power of the shallow-deep joint perception features of the visible light of the navigation condition, but also provides adaptive weight adjustment for cross-modal fusion. The adaptive selection process ensures that the model can select the most valuable information for processing according to the specific conditions of the current environment, and can ensure that the model can flexibly allocate attention resources when dealing with different types of obstacles. For example, in a densely populated area, the model may pay more attention to the gaps between ships, while in open waters, it may pay more attention to fixed obstacles in the distance.

[0097] Then, considering that the information provided by the millimeter-wave radar can supplement the deficiencies of visible light images in low visibility conditions. By combining radar feature information with visible light image feature information, the model can more comprehensively understand obstacles in the environment, thereby making more accurate obstacle avoidance decisions. Therefore, by calculating the product of the navigation condition visible light shallow-deep joint perception gated local autocorrelation library weight matrix and the navigation condition deep millimeter-wave detection semantic feature vector, conditional fusion between modalities is achieved, and the cross-modal correlation of features is enhanced. Finally, the navigation condition deep millimeter-wave detection semantic cross-modal interactive fusion condition weight vector is used to weight the navigation condition visible light shallow-deep joint perception feature map channel by channel to generate a navigation state multi-module cross-domain fusion feature map. This feature map integrates feature information of different modalities, enabling the unmanned ship to more comprehensively understand the navigation environment, improve the reliability and safety of obstacle avoidance decisions, and enhance the generalization ability of the model.

[0098] Correspondingly, in step S560, the navigation status visible light shallow-deep joint perception feature map and the navigation status deep millimeter wave detection semantic feature vector are input into a gated Transformer structure encoder based on cross-modal information fusion to obtain a navigation status multi-module cross-domain fusion feature map, including: performing layer normalization and convolution coding processing on the navigation status visible light shallow-deep joint perception feature map to obtain a navigation status visible light shallow-deep joint perception space deepening feature map; copying the navigation status visible light shallow-deep joint perception space deepening feature map to obtain a backup navigation status visible light shallow-deep joint perception space deepening feature map; and reshaping the backup navigation status visible light shallow-deep joint perception space deepening feature map and the navigation status visible light shallow-deep joint perception space deepening feature map respectively to obtain a backup navigation status visible light shallow-deep joint perception space deepening feature matrix as a simulated query feature matrix and a navigation status visible light shallow-deep joint perception space deepening feature matrix. The feature matrix is ​​used as the simulated key feature matrix; the product between the simulated query feature matrix and the transposed matrix of the simulated key feature matrix is ​​calculated to obtain the navigation condition visible light shallow-deep joint perception local autocorrelation library matrix; the navigation condition visible light shallow-deep joint perception local autocorrelation library matrix is ​​input into the attention gating unit based on the GELU function to obtain the navigation condition visible light shallow-deep joint perception gated local autocorrelation library weight matrix; the navigation condition deep millimeter wave detection semantic feature vector is used as the value feature vector, and the product between the navigation condition visible light shallow-deep joint perception gated local autocorrelation library weight matrix and the navigation condition deep millimeter wave detection semantic feature vector is calculated to obtain the navigation condition deep millimeter wave detection semantic cross-modal interactive fusion condition weight vector; the navigation condition deep millimeter wave detection semantic cross-modal interactive fusion condition weight vector is used as the weight vector, and the navigation condition visible light shallow-deep joint perception feature map is weighted channel by channel to obtain the navigation status multi-module cross-domain fusion feature map.

[0099] Among them, the navigation condition visible light shallow-deep joint perception feature map is subjected to layer normalization and convolution coding processing to obtain the navigation condition visible light shallow-deep joint perception spatial deepening feature map, including: the navigation condition visible light shallow-deep joint perception feature map is subjected to layer normalization processing to obtain the layer normalized navigation condition visible light shallow-deep joint perception feature map; the layer normalized navigation condition visible light shallow-deep joint perception feature map is subjected to point convolution coding and then spatial convolution coding to obtain the navigation condition visible light shallow-deep joint perception spatial deepening feature map.

[0100] In a specific example, the navigation status visible light shallow-deep joint perception feature map and the navigation status deep millimeter wave detection semantic feature vector are input into a gated Transformer structure encoder based on cross-modal information fusion to obtain a navigation status multi-module cross-domain fusion feature map, including: inputting the navigation status visible light shallow-deep joint perception feature map and the navigation status deep millimeter wave detection semantic feature vector into the gated Transformer structure encoder based on cross-modal information fusion and processing them with the following multi-modal fusion formula to obtain the navigation status multi-module cross-domain fusion feature map; wherein the multi-modal fusion formula is:

[0101]

[0102]

[0103]

[0104]

[0105]

[0106] in, is the visible light shallow-deep joint perception feature map of the navigation condition, is the layer normalization operation, is point convolutional coding, is a convolution operation with a convolution kernel of 3×3. The visible light shallow-deep joint perception space deepening feature map for the navigation condition, For copy operation, The backup navigation condition visible light shallow-deep joint perception space deepening feature map, For the reshape operation, is the transpose of the feature matrix, is matrix multiplication, is the GELU function, is the weight matrix of the visible light shallow-deep joint perception gated local autocorrelation library for the navigation condition, is the semantic feature vector of the navigation status depth millimeter wave detection, is the navigation condition depth millimeter wave detection semantic cross-modal interactive fusion condition weight vector, It is the multi-module cross-domain fusion feature map of the navigation status.

[0107] Then, the multi-module cross-domain fusion feature map of the navigation state is input into the cruise instruction generation module based on the classifier to obtain the cruise instruction, which is used to indicate whether to perform obstacle avoidance. That is, the multi-modal cross-domain fusion features of the navigation state are classified and processed to perform environmental perception of the cruising process to determine whether to perform obstacle avoidance. By adopting a variety of sensor fusion technologies, the environmental perception information of the unmanned ship, such as obstacles, can be obtained in real time to generate cruise instructions to optimize the speed and direction of the unmanned ship to achieve obstacle avoidance. In this way, it can not only help the unmanned ship identify obstacles, but also optimize its obstacle avoidance strategy for autonomous cruising, and plan an optimized cruise route that effectively avoids obstacles and saves power.

[0108] Accordingly, in step S570, a cruise instruction is generated based on the navigation status multi-module cross-domain fusion feature map, and the cruise instruction is used to indicate whether to perform obstacle avoidance, including: inputting the navigation status multi-module cross-domain fusion feature map into a classifier-based cruise instruction generation module to obtain the cruise instruction.

[0109] The role of the classifier is to use the given categories and known training data to learn classification rules and classifiers, and then classify (or predict) unknown data. Logistic regression and SVM are often used to solve binary classification problems. For multi-class classification problems, logistic regression or SVM can also be used, but multiple binary classifications are required to form a multi-classification. However, this is prone to errors and is inefficient. Commonly used multi-classification methods include the Softmax classification function.

[0110] In a preferred example, the navigation state multi-module cross-domain fusion feature map is input into a classifier-based cruise instruction generation module to obtain the cruise instruction, including: calculating the sum of the absolute values ​​of each eigenvalue of the navigation state multi-module cross-domain fusion feature map to obtain a first navigation state multi-module cross-domain fusion and modulation value, and calculating the square root of the sum of the squares of each eigenvalue of the navigation state multi-module cross-domain fusion feature map to obtain a second navigation state multi-module cross-domain fusion and modulation value; performing dot subtraction between the navigation state multi-module cross-domain fusion feature map and the second navigation state multi-module cross-domain fusion and modulation value, and then performing dot multiplication with the number of eigenvalues ​​of the navigation state multi-module cross-domain fusion feature map and the reciprocal of the first navigation state multi-module cross-domain fusion and modulation value, and taking the reciprocal of each eigenvalue to obtain the first navigation state A multi-module cross-domain fusion phase conversion feature map; after point-wise subtraction of the navigation state multi-module cross-domain fusion feature map and the first navigation state multi-module cross-domain fusion and modulation value, respectively perform point multiplication with the square root of the number of eigenvalues ​​of the navigation state multi-module cross-domain fusion feature map and the reciprocal of the second navigation state multi-module cross-domain fusion and modulation value, and take the reciprocal of each eigenvalue to obtain the second navigation state multi-module cross-domain fusion phase conversion feature map; point-wise subtraction of the first navigation state multi-module cross-domain fusion phase conversion feature map from the second navigation state multi-module cross-domain fusion phase conversion feature map and the dot product vector of the weighted hyperparameter to obtain an optimized navigation state multi-module cross-domain fusion feature map; the optimized navigation state multi-module cross-domain fusion feature map is passed through the classifier-based cruise instruction generation module to obtain a cruise instruction.

[0111] Here, the multi-module cross-domain fusion feature map of the navigation state is recorded as The optimization process is expressed as follows:

[0112]

[0113]

[0114]

[0115]

[0116]

[0117]

[0118]

[0119] in, is the multi-module cross-domain fusion feature map of the navigation state, are the characteristic values ​​of the multi-module cross-domain fusion feature map of the navigation status, represents absolute value calculation, It is the multi-module cross-domain fusion and modulation value of the first navigation state. For the second navigation state multi-module cross-domain fusion and modulation value, Indicates point reduction, represents dot product, represents the number of eigenvalues ​​of the multi-module cross-domain fusion feature graph of the navigation status, means taking the inverse of each eigenvalue, It is the phase transition feature map of multi-module cross-domain fusion in the first navigation state. Represents the phase transition feature map of multi-module cross-domain fusion in the second navigation state, represents the weighted hyperparameter, To optimize the navigation status multi-module cross-domain fusion feature map, represents the width of the feature map, represents the height of the feature map, Indicates the number of channels of the feature map.

[0120] That is, considering that the navigation condition visible light shallow-deep joint perception feature map and the navigation condition deep millimeter wave detection semantic feature vector respectively represent the joint perception features of shallow image features and deep image features of the navigation visible light image and the navigation condition deep millimeter wave detection semantic features determined based on deep separable convolutional coding of the navigation millimeter wave detection image, when they are passed through a gated Transformer structure encoder based on cross-modal information fusion, the different dimensional association structures based on the cross-modal feature distribution will also have cross-modal fusion context association encoding differences. Therefore, it is expected to improve the detail semantic aggregation expression effect based on the multi-modal semantic association representation differences of the navigation status multi-module cross-domain fusion feature map.

[0121] Therefore, in the preferred example, the difference of the difference and modulation representation of the feature value of the navigation state multi-module cross-domain fusion feature map relative to the overall feature set of the navigation state multi-module cross-domain fusion feature map is used as the semantic change intensity information, and a phase-like conversion corresponding to the position-based intensity modulation is performed through different and modulation representation forms, so that the aggregation enhancement of semantic change phase perception can improve the axial aggregation receptive field along the feature aggregation direction by performing a spatial translation operation based on alternating stacking under the set scale balance of the navigation state multi-module cross-domain fusion feature map, thereby improving the perception effect of the aggregated semantics of the navigation state multi-module cross-domain fusion feature map on the detailed semantic changes, so as to improve the expression effect of the navigation state multi-module cross-domain fusion feature map, and improve the accuracy of the cruise instruction obtained by the cruise instruction generation module based on the classifier, so that the environmental perception of the cruising process can be more accurately performed to generate cruise instructions, so as to optimize the travel speed and direction of the unmanned ship to achieve obstacle avoidance.

[0122] Based on the above embodiments, see Figure 4 As shown, it is a structural schematic diagram of an unmanned ship autonomous cruising system 800 based on multi-module fusion in an embodiment of the present application. The unmanned ship autonomous cruising system 800 based on multi-module fusion includes: a navigation visible light image acquisition module 810, used to obtain a navigation visible light image acquired by a camera; a navigation millimeter wave detection image acquisition module 820, used to obtain a navigation millimeter wave detection image acquired by a millimeter wave radar; a multi-scale feature scanning module 830, used to perform a multi-scale feature scanning of the navigation visible light image based on the navigation condition to obtain a navigation condition visible light shallow feature map and a navigation condition visible light deep feature map; a shallow-deep joint perception module 840, used to input the navigation condition visible light shallow feature map and the navigation condition visible light deep feature map into a two-way global fine-grained attention feature joint module to obtain a navigation condition visible light. A shallow-deep joint perception feature map; a heading status depth explicit depiction module 850, used for performing a heading status depth explicit depiction based on the navigation millimeter-wave detection image to obtain a navigation status depth millimeter-wave detection semantic feature vector; a cross-modal information fusion module 860, used for inputting the navigation status visible light shallow-deep joint perception feature map and the navigation status depth millimeter-wave detection semantic feature vector into a gated Transformer structure encoder based on cross-modal information fusion to obtain a navigation status multi-module cross-domain fusion feature map; a cruise instruction generation module 870, used for generating a cruise instruction based on the navigation status multi-module cross-domain fusion feature map, the cruise instruction being used to indicate whether to perform obstacle avoidance.

[0123] The specific functions and operations of each module in the unmanned ship autonomous cruising system 800 based on multi-module fusion have been referred to above. Figure 2 to Figure 3The description of the unmanned ship autonomous cruising method based on multi-module fusion has been introduced in detail, and therefore, its repeated description will be omitted.

[0124] The control system is the brain of the unmanned ship intelligent control platform. The control logic and intelligent algorithm of the unmanned ship are implemented by the control system. The unmanned ship platform reflects the autonomous navigation, transmission signal status, satellite signal, navigation speed, monitoring status, onboard battery status and early warning prompts, low voltage reminder protection, etc. of the unmanned ship in real time. The navigation speed can be set according to the use requirements or during the navigation process. If the signal is interrupted during the operation, the unmanned ship will automatically return to the original starting point after 15 seconds (the time can be set) to ensure the safety of the ship and equipment.

[0125] Based on the above embodiment, another exemplary embodiment of an electronic device is also provided in the embodiment of the present application. In some possible implementations, the electronic device in the embodiment of the present application may include a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor can implement the steps of the autonomous cruising method of an unmanned ship based on multi-module fusion in the above embodiment when executing the program.

[0126] For example, in the case of electronic equipment Figure 1 Taking the server 100 in the example as an example, the processor in the electronic device is the processor 110 in the server 100, and the memory in the electronic device is the memory 120 in the server 100.

[0127] The embodiment of the present application also provides a computer-readable storage medium, on which computer-executable instructions are stored. When the computer-executable instructions are executed by a processor, the unmanned ship autonomous cruising method based on multi-module fusion according to the embodiment of the present application described with reference to the above figures can be executed. The computer-readable storage medium includes, but is not limited to, for example, volatile memory and / or non-volatile memory. The volatile memory may, for example, include a random access memory (RAM) and / or a cache memory (cache), etc. The non-volatile memory may, for example, include a read-only memory (ROM), a hard disk, a flash memory, etc.

[0128] The embodiment of the present application also provides a computer program product or a computer program, which includes computer executable instructions, and the computer executable instructions are stored in a computer-readable storage medium. The processor of the computer device reads the computer executable instructions from the computer-readable storage medium, and the processor executes the computer executable instructions, so that the computer device executes the unmanned ship autonomous cruising method based on multi-module fusion according to the embodiment of the present application.

[0129] Those skilled in the art will appreciate that the contents disclosed in this application may be subject to various variations and improvements. For example, the various devices or components described above may be implemented by hardware, or by software, firmware, or a combination of some or all of the three.

[0130] In addition, although the present application makes various references to certain units in the system according to embodiments of the present application, any number of different units can be used and run on the client and / or server. The units are only illustrative, and different aspects of the system and method can use different units.

[0131] Those skilled in the art will appreciate that all or part of the steps in the above method can be completed by instructing related hardware through a program, and the program can be stored in a computer-readable storage medium, such as a read-only memory, a disk or an optical disk. Optionally, all or part of the steps in the above embodiment can also be implemented using one or more integrated circuits. Accordingly, each module / unit in the above embodiment can be implemented in the form of hardware or in the form of software function modules. The present application is not limited to any particular form of combination of hardware and software.

[0132] Unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by those of ordinary skill in the art to which this application belongs. It should also be understood that terms such as those defined in common dictionaries should be interpreted as having a meaning consistent with their meaning in the context of the relevant technology, and should not be interpreted in an idealized or extremely formal sense, unless explicitly defined as such herein.

[0133] The above is an explanation of the present application and should not be considered as limiting thereof. Although several exemplary embodiments of the present application have been described, those skilled in the art will readily appreciate that many modifications may be made to the exemplary embodiments without departing from the novel teachings and advantages of the present application.

Claims

1. An unmanned ship autonomous cruising method based on multi-module fusion, characterized in that: include: Acquire navigation visible light images collected by a camera; Acquire navigation millimeter-wave detection images collected by millimeter-wave radar; Performing a multi-scale feature scan based on the navigation condition on the navigation visible light image to obtain a navigation condition visible light shallow feature map and a navigation condition visible light deep feature map; Inputting the navigation condition visible light shallow feature map and the navigation condition visible light deep feature map into a bidirectional global fine-grained attention feature joint module to obtain a navigation condition visible light shallow-deep joint perception feature map; Based on the navigation millimeter wave detection image, a depth saliency description of the heading condition is performed to obtain a semantic feature vector of the navigation condition depth millimeter wave detection; Input the navigation status visible light shallow-deep joint perception feature map and the navigation status deep millimeter wave detection semantic feature vector into a gated Transformer structure encoder based on cross-modal information fusion to obtain a navigation status multi-module cross-domain fusion feature map; Generate a cruise instruction based on the multi-module cross-domain fusion feature map of the navigation state, wherein the cruise instruction is used to indicate whether to perform obstacle avoidance; The navigation condition visible light shallow feature map and the navigation condition visible light deep feature map are input into a two-way global fine-grained attention feature joint module to obtain a navigation condition visible light shallow-deep joint perception feature map, including: Performing feature fine-grained deconstruction on the navigation condition visible light shallow feature map and the navigation condition visible light deep feature map to obtain a set of navigation condition visible light shallow channel dimension local feature matrices and a set of navigation condition visible light deep channel dimension local feature matrices; Using each local feature matrix of the shallow channel dimension of visible light for navigation conditions in the set of local feature matrices of the shallow channel dimension of visible light for navigation conditions as a query feature matrix, using the set of local feature matrices of the deep channel dimension of visible light for navigation conditions as a set of key feature matrices, inputting the query feature matrix and the set of key feature matrices into a one-way global attention interaction module based on the first converter structure to obtain a set of local feature matrices of the shallow channel dimension of visible light for navigation conditions optimized by one-way global attention; Using each local feature matrix of the deep channel dimension of visible light for navigation conditions in the set of local feature matrices of the deep channel dimension of visible light for navigation conditions as a query feature matrix, using the set of local feature matrices of the shallow channel dimension of visible light for navigation conditions as a set of key feature matrices, inputting the query feature matrix and the set of key feature matrices into a one-way global attention interaction module based on a second converter structure to obtain a set of local feature matrices of the deep channel dimension of visible light for navigation conditions optimized by one-way global attention; The set of local feature matrices of the shallow channel dimension of the visible light of the one-way global attention optimized navigation condition and the set of local feature matrices of the deep channel dimension of the visible light of the one-way global attention optimized navigation condition are coupled along the channel dimension to obtain a shallow feature map of the visible light of the one-way global interactive optimized navigation condition and a deep feature map of the visible light of the one-way global interactive optimized navigation condition; The position-weighted sum between the one-way global interactive optimization of the navigation condition visible light shallow feature map and the one-way global interactive optimization of the navigation condition visible light deep feature map is calculated to obtain the navigation condition visible light shallow-deep joint perception feature map.

2. The unmanned ship autonomous cruising method based on multi-module fusion according to claim 1 is characterized in that: The navigation visible light image is subjected to a multi-scale feature scanning based on the navigation conditions to obtain a navigation condition visible light shallow feature map and a navigation condition visible light deep feature map, comprising: inputting the navigation visible light image into a navigation condition multi-scale feature scanner based on a hollow pyramid network to obtain the navigation condition visible light shallow feature map and the navigation condition visible light deep feature map.

3. The unmanned ship autonomous cruising method based on multi-module fusion according to claim 2 is characterized in that: Each local feature matrix of the shallow channel dimension of visible light in the navigation condition in the set of local feature matrices of the shallow channel dimension of visible light in the navigation condition is used as a query feature matrix, and the set of local feature matrices of the deep channel dimension of visible light in the navigation condition is used as a set of key feature matrices, and the query feature matrix and the set of key feature matrices are input into a one-way global attention interaction module based on the first converter structure to obtain a set of local feature matrices of the shallow channel dimension of visible light in the navigation condition optimized by one-way global attention, including: Selecting a predetermined navigation condition visible light shallow channel dimension local feature matrix from the set of navigation condition visible light shallow channel dimension local feature matrix as a query feature matrix; Calculate the product of the predetermined local feature matrix of the shallow channel dimension of visible light of the navigation condition and the transposed matrix of each local feature matrix of the deep channel dimension of visible light of the navigation condition in the set of local feature matrices of the deep channel dimension of visible light of the navigation condition to obtain a set of local semantic interaction feature matrices of the shallow-deep channel dimension of visible light of the navigation condition; After dividing the scale square root of the predetermined local feature matrix of the shallow channel dimension of visible light for navigation conditions by each local semantic interaction feature matrix of the shallow-deep channel dimension of visible light for navigation conditions in the set of local semantic interaction feature matrices of the shallow-deep channel dimension of visible light for navigation conditions at their respective positions, a softmax function is used to perform soft maximum normalization processing on each feature matrix in the obtained set of feature matrices to obtain a set of local weight matrices of the shallow channel dimension of visible light for navigation conditions; Taking the set of local weight matrices of the navigation condition visible light shallow channel dimension as weighted weights, the position-weighted sum of the local feature matrices of the navigation condition visible light shallow channel dimension in the set of local feature matrices of the navigation condition visible light shallow channel dimension is calculated to obtain the one-way global attention optimized navigation condition visible light shallow channel dimension local feature matrix.

4. The unmanned ship autonomous cruising method based on multi-module fusion according to claim 3 is characterized in that: Each local feature matrix of the deep channel dimension of visible light in the navigation condition in the set of local feature matrices of the deep channel dimension of visible light in the navigation condition is used as a query feature matrix, and the set of local feature matrices of the shallow channel dimension of visible light in the navigation condition is used as a set of key feature matrices, and the query feature matrix and the set of key feature matrices are input into a one-way global attention interaction module based on a second converter structure to obtain a set of local feature matrices of the shallow channel dimension of visible light in the navigation condition optimized by one-way global attention, including: Selecting a predetermined navigation condition visible light deep channel dimension local feature matrix from the set of navigation condition visible light deep channel dimension local feature matrices as a query feature matrix; Calculate the product of the predetermined local feature matrix of the deep channel dimension of visible light for navigation conditions and the transposed matrix of each local feature matrix of the shallow channel dimension of visible light for navigation conditions in the set of local feature matrices of the shallow channel dimension of visible light for navigation conditions to obtain a set of local semantic interaction feature matrices of the deep-shallow channel dimension of visible light for navigation conditions; After dividing the scale square root of the predetermined local feature matrix of the deep channel dimension of visible light for navigation conditions by each local semantic interaction feature matrix of the deep-shallow channel dimension of visible light for navigation conditions at each position point, each feature matrix in the obtained set of feature matrices is subjected to soft maximum normalization processing using a softmax function to obtain a set of local weight matrices of the deep channel dimension of visible light for navigation conditions; Taking the set of local weight matrices of the navigation condition visible light deep channel dimension as weighted weights, the position-weighted sum of the local feature matrices of the navigation condition visible light deep channel dimension in the set of local feature matrices of the navigation condition visible light deep channel dimension is calculated to obtain the one-way global attention optimized local feature matrix of the navigation condition visible light deep channel dimension.

5. The unmanned ship autonomous cruising method based on multi-module fusion according to claim 4 is characterized in that: Based on the navigation millimeter-wave detection image, a heading condition depth saliency depiction is performed to obtain a navigation condition depth millimeter-wave detection semantic feature vector, including: inputting the navigation millimeter-wave detection image into a heading condition depth saliency depictor based on a deep separable convolutional network to obtain the navigation condition depth millimeter-wave detection semantic feature vector.

6. The unmanned ship autonomous cruising method based on multi-module fusion according to claim 5 is characterized in that: The navigation status visible light shallow-deep joint perception feature map and the navigation status deep millimeter wave detection semantic feature vector are input into a gated Transformer structure encoder based on cross-modal information fusion to obtain a navigation status multi-module cross-domain fusion feature map, including: Performing layer normalization and convolution coding processing on the navigation condition visible light shallow layer-deep layer joint perception feature map to obtain the navigation condition visible light shallow layer-deep layer joint perception spatial deepening feature map; Copying the navigation condition visible light shallow-deep joint perception space deepening feature map to obtain a backup navigation condition visible light shallow-deep joint perception space deepening feature map; Reshape the backup navigation condition visible light shallow-deep joint perception space deepening feature map and the navigation condition visible light shallow-deep joint perception space deepening feature map respectively to obtain the backup navigation condition visible light shallow-deep joint perception space deepening feature matrix as the simulated query feature matrix and the navigation condition visible light shallow-deep joint perception space deepening feature matrix as the simulated key feature matrix; Calculating the product of the pseudo-query feature matrix and the transposed matrix of the pseudo-key feature matrix to obtain a navigation condition visible light shallow-deep joint perception local autocorrelation library matrix; Inputting the navigation condition visible light shallow-deep joint perception local autocorrelation library matrix into the attention gating unit based on the GELU function to obtain the navigation condition visible light shallow-deep joint perception gated local autocorrelation library weight matrix; Taking the navigation condition depth millimeter wave detection semantic feature vector as the value feature vector, calculating the product between the navigation condition visible light shallow-deep joint perception gated local autocorrelation library weight matrix and the navigation condition depth millimeter wave detection semantic feature vector to obtain the navigation condition depth millimeter wave detection semantic cross-modal interactive fusion condition weight vector; The navigation status deep millimeter wave detection semantic cross-modal interactive fusion condition weight vector is used as the weight vector, and the navigation status visible light shallow-deep joint perception feature map is weighted channel by channel to obtain the navigation status multi-module cross-domain fusion feature map.

7. The unmanned ship autonomous cruising method based on multi-module fusion according to claim 6 is characterized in that: The navigation condition visible light shallow layer-deep layer joint perception feature map is subjected to layer normalization and convolution coding processing to obtain the navigation condition visible light shallow layer-deep layer joint perception spatial deepening feature map, including: Performing layer normalization processing on the navigation condition visible light shallow layer-deep layer joint perception feature map to obtain a layer normalized navigation condition visible light shallow layer-deep layer joint perception feature map; The layer-normalized navigation condition visible light shallow-layer-deep joint perception feature map is point-convolutionally encoded and then spatially convolutionally encoded to obtain the navigation condition visible light shallow-layer-deep joint perception spatial deepening feature map.

8. The unmanned ship autonomous cruising method based on multi-module fusion according to claim 7 is characterized in that: Based on the multi-module cross-domain fusion feature map of the navigation status, a cruise instruction is generated, and the cruise instruction is used to indicate whether to perform obstacle avoidance, including: inputting the multi-module cross-domain fusion feature map of the navigation status into a cruise instruction generation module based on a classifier to obtain the cruise instruction.

9. The unmanned ship autonomous cruising system based on multi-module fusion is characterized by: include: A navigation visible light image acquisition module is used to acquire the navigation visible light image acquired by the camera; A navigation millimeter wave detection image acquisition module is used to obtain a navigation millimeter wave detection image collected by a millimeter wave radar; A multi-scale feature scanning module, used for performing a multi-scale feature scanning on the navigation visible light image based on the navigation condition to obtain a navigation condition visible light shallow feature map and a navigation condition visible light deep feature map; A shallow-deep joint perception module is used to input the navigation condition visible light shallow feature map and the navigation condition visible light deep feature map into a two-way global fine-grained attention feature joint module to obtain a navigation condition visible light shallow-deep joint perception feature map; A heading condition depth saliency depiction module, used for performing heading condition depth saliency depiction based on the navigation millimeter wave detection image to obtain a navigation condition depth millimeter wave detection semantic feature vector; A cross-modal information fusion module is used to input the navigation status visible light shallow-deep joint perception feature map and the navigation status deep millimeter wave detection semantic feature vector into a gated Transformer structure encoder based on cross-modal information fusion to obtain a navigation status multi-module cross-domain fusion feature map; A cruise instruction generation module, used to generate a cruise instruction based on the multi-module cross-domain fusion feature map of the navigation state, wherein the cruise instruction is used to indicate whether to perform obstacle avoidance; The navigation condition visible light shallow feature map and the navigation condition visible light deep feature map are input into a two-way global fine-grained attention feature joint module to obtain a navigation condition visible light shallow-deep joint perception feature map, including: Performing feature fine-grained deconstruction on the navigation condition visible light shallow feature map and the navigation condition visible light deep feature map to obtain a set of navigation condition visible light shallow channel dimension local feature matrices and a set of navigation condition visible light deep channel dimension local feature matrices; Using each local feature matrix of the shallow channel dimension of visible light for navigation conditions in the set of local feature matrices of the shallow channel dimension of visible light for navigation conditions as a query feature matrix, using the set of local feature matrices of the deep channel dimension of visible light for navigation conditions as a set of key feature matrices, inputting the query feature matrix and the set of key feature matrices into a one-way global attention interaction module based on the first converter structure to obtain a set of local feature matrices of the shallow channel dimension of visible light for navigation conditions optimized by one-way global attention; Using each local feature matrix of the deep channel dimension of visible light for navigation conditions in the set of local feature matrices of the deep channel dimension of visible light for navigation conditions as a query feature matrix, using the set of local feature matrices of the shallow channel dimension of visible light for navigation conditions as a set of key feature matrices, inputting the query feature matrix and the set of key feature matrices into a one-way global attention interaction module based on a second converter structure to obtain a set of local feature matrices of the deep channel dimension of visible light for navigation conditions optimized by one-way global attention; The set of local feature matrices of the shallow channel dimension of the visible light of the one-way global attention optimized navigation condition and the set of local feature matrices of the deep channel dimension of the visible light of the one-way global attention optimized navigation condition are coupled along the channel dimension to obtain a shallow feature map of the visible light of the one-way global interactive optimized navigation condition and a deep feature map of the visible light of the one-way global interactive optimized navigation condition; The position-weighted sum between the one-way global interactive optimization of the navigation condition visible light shallow feature map and the one-way global interactive optimization of the navigation condition visible light deep feature map is calculated to obtain the navigation condition visible light shallow-deep joint perception feature map.

Images