Multi-sensor adaptive anti-collision method and device based on deep learning and medium

Abstract

The invention provides a multi-sensor self-adaptive anti-collision method and device based on deep learning and a medium, and relates to the field of ship anti-collision, and the method comprises the steps: obtaining the ship position of a target ship and multi-class obstacle data; a self-attention mechanism is adopted, data fusion is carried out on the multiple pieces of obstacle data, and a fusion feature vector is obtained; processing the fusion feature vector by adopting YOLOv5, and determining a bounding box of the obstacle, a bounding box position corresponding to the bounding box and a bounding box speed; the bounding box position and the bounding box speed are processed through a convolutional neural network, and future trajectory data of the obstacle and the collision probability of the obstacle and the target ship are obtained; and determining the course and speed of the target ship based on the future trajectory data, the ship position and the collision probability. According to the invention, through fusion of multiple sensors and application of a deep learning algorithm, high-precision collision prediction and risk assessment can be realized under complex sea conditions and low visibility.

Description

Technical Field

[0001] The present application relates to the field of ship collision avoidance, and more specifically, to a multi-sensor adaptive collision avoidance method, device and medium based on deep learning. Background Art

[0002] During the voyage of ships at sea, collision accidents often lead to serious losses. However, most traditional collision avoidance systems rely on single sensors, such as radar, AIS or cameras, which cannot provide sufficient accurate data in certain environments (such as dense fog, nighttime, complex sea conditions, etc.). Although the existing collision avoidance systems can provide early warning of collision risks to a certain extent, they still have problems such as insufficient data processing capabilities, slow response speed, false alarms and missed reports in actual applications. Summary of the invention

[0003] The purpose of the embodiments of the present application is to provide a multi-sensor adaptive collision avoidance method, device and medium based on deep learning, so as to solve the above-mentioned problems existing in the prior art and realize high-precision ship collision prediction and risk assessment under complex sea conditions and low visibility.

[0004] In a first aspect, a multi-sensor adaptive collision avoidance method based on deep learning is provided, and the method may include:

[0005] Obtain the ship position and multi-category obstacle data of the target ship; the multi-category obstacle data is collected by multi-category sensors installed on the target ship;

[0006] The self-attention mechanism is used to fuse multiple obstacle data to obtain a fused feature vector;

[0007] Using YOLOv5, the fused feature vector is processed to determine the bounding box of the obstacle, the bounding box position corresponding to the bounding box, and the bounding box velocity;

[0008] The bounding box position and the bounding box velocity are processed by a convolutional neural network to obtain future trajectory data of the obstacle and a collision probability between the obstacle and the target ship;

[0009] The heading and speed of the target ship are determined based on the future trajectory data, the ship position and the collision probability.

[0010] In one possible implementation, the multiple categories of sensors include: radar, LiDAR sensor, and camera.

[0011] In a possible implementation, before fusing the multi-category obstacle data using a self-attention mechanism, the method further includes:

[0012] Perform time synchronization, denoising and coordinate conversion on obstacle data collected by sensors of various categories to obtain processed obstacle data;

[0013] The self-attention mechanism is used to fuse the processed multi-category obstacle data.

[0014] In a possible implementation, after obtaining the future trajectory data of the obstacle and the collision probability between the obstacle and the target ship, the method includes:

[0015] The future trajectory data and ship position are processed through the configured risk assessment model to determine the risk assessment value;

[0016] A collision avoidance strategy is determined according to the risk assessment value.

[0017] In a possible implementation, determining a collision avoidance strategy according to the risk assessment value includes:

[0018] If the risk assessment value is greater than a preset threshold value, the collision avoidance strategy is to control the target ship to travel according to the heading and the speed.

[0019] In a possible implementation, determining the heading and speed of the target ship based on the future trajectory data, the ship position, and the collision probability includes:

[0020] The configured deep reinforcement learning model is used to process the collision probability, the future trajectory data, the ship speed of the target ship and the ship position to obtain the heading and speed of the target ship.

[0021] In a second aspect, a multi-sensor adaptive collision avoidance device based on deep learning is provided, which may include:

[0022] An acquisition unit, used to acquire the ship position of the target ship and multi-category obstacle data; the multi-category obstacle data is collected by multi-category sensors installed on the target ship;

[0023] A fusion unit is used to fuse multiple obstacle data using a self-attention mechanism to obtain a fused feature vector;

[0024] A processing unit, configured to process the fused feature vector using YOLOv5 to determine a bounding box of the obstacle, a bounding box position corresponding to the bounding box, and a bounding box velocity;

[0025] and, processing the bounding box position and the bounding box velocity through a convolutional neural network to obtain future trajectory data of the obstacle and a collision probability between the obstacle and the target ship;

[0026] A determination unit is used to determine the heading and speed of the target ship based on the future trajectory data, the ship position and the collision probability.

[0027] In a third aspect, an electronic device is provided, the electronic device comprising a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other via the communication bus;

[0028] Memory, used to store computer programs;

[0029] The processor is used to implement any method step described in the first aspect when executing the program stored in the memory.

[0030] In a fourth aspect, a computer-readable storage medium is provided, wherein a computer program is stored in the computer-readable storage medium, and when the computer program is executed by a processor, any method step described in the first aspect is implemented.

[0031] The present application provides a multi-sensor adaptive collision avoidance method based on deep learning, the method comprising: obtaining a ship position and multi-category obstacle data of a target ship; the multi-category obstacle data is collected by multi-category sensors installed on the target ship; using a self-attention mechanism to fuse multiple obstacle data to obtain a fused feature vector; using YOLOv5 to process the fused feature vector to determine a bounding box of the obstacle, a bounding box position corresponding to the bounding box, and a bounding box speed; using a convolutional neural network to process the bounding box position and the bounding box speed to obtain future trajectory data of the obstacle and a collision probability between the obstacle and the target ship; and determining a heading and a speed of the target ship based on the future trajectory data, the ship position, and the collision probability. Through the fusion of multiple sensors and the application of deep learning algorithms, this application can achieve high-precision collision prediction and risk assessment in complex sea conditions and low visibility; it can make real-time adjustments and decisions based on the current navigation status of the ship, environmental changes, and dynamic changes of the target, avoiding the limitations of traditional collision avoidance systems; through intelligent data fusion and target recognition algorithms, it greatly reduces the risk of false alarms and missed alarms and improves the reliability of the system; when a potential collision occurs, the system can quickly generate the optimal collision avoidance path, reduce the operating pressure of the crew, and ensure navigation safety; on the premise of ensuring the safety of the crew, it can automatically perform risk prediction, path planning, and collision avoidance decisions, significantly improving the degree of automation of the ship. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments of the present application will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0033] Figure 1 A system architecture diagram of a multi-sensor adaptive collision avoidance method based on deep learning provided in an embodiment of the present application;

[0034] Figure 2 A schematic diagram of a flow chart of a multi-sensor adaptive collision avoidance method based on deep learning provided in an embodiment of the present application;

[0035] Figure 3 A schematic diagram of the structure of a multi-sensor adaptive collision avoidance device based on deep learning provided in an embodiment of the present application;

[0036] Figure 4 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0037] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments of the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0038] The multi-sensor adaptive collision avoidance method based on deep learning provided in the embodiment of the present application can be applied to Figure 1 In the system architecture shown in Figure 1 As shown, the system may include: a processor and multiple types of sensors.

[0039] Among them, multiple categories of sensors may include: radar, LiDAR sensors, and cameras;

[0040] Radar, used to collect the distance of the obstacle from the target ship, the speed of the obstacle and the azimuth of the obstacle;

[0041] LiDAR sensor, used to collect point cloud data of the corresponding area;

[0042] The camera is used to collect images of the corresponding area.

[0043] Various types of sensors send the collected data to the processor.

[0044] The processor is used to receive data sent by each sensor to execute a multi-sensor adaptive collision avoidance method based on deep learning provided in the present application.

[0045] During the voyage of ships at sea, collision accidents often lead to serious losses. However, most traditional collision avoidance systems rely on single sensors, such as radar, AIS or cameras, which cannot provide sufficient accurate data in certain environments (such as dense fog, nighttime, complex sea conditions, etc.). Although the existing collision avoidance systems can provide early warning of collision risks to a certain extent, they still have problems such as insufficient data processing capabilities, slow response speed, false alarms and missed reports in actual applications.

[0046] 1. Single sensor reliance: Traditional collision avoidance systems often rely on a single sensor (such as radar or AIS). This single sensor has low accuracy in complex weather or sea conditions and cannot provide comprehensive environmental data, resulting in inaccurate warnings of collision risks.

[0047] 2. Lack of adaptive capabilities: Most existing collision avoidance systems fail to adaptively adjust their strategies according to environmental changes, and have the defect of being unable to accurately judge and respond to complex dynamic environments.

[0048] 3. Calculation and response delay: Traditional algorithms have high computational complexity, resulting in insufficient real-time response capabilities, especially in multi-ship sailing or rapidly changing dynamic environments, where the course cannot be adjusted in time to avoid risks.

[0049] 4. False alarm and missed alarm problems: Due to insufficient sensor data processing, existing systems are prone to false alarms and missed alarms in complex environments, affecting system reliability and efficiency.

[0050] Therefore, the present application provides a multi-sensor adaptive collision avoidance method based on deep learning to solve the above-mentioned problems existing in the prior art, and can achieve high-precision ship collision prediction and risk assessment under complex sea conditions and low visibility.

[0051] The preferred embodiments of the present application are described below in conjunction with the drawings in the specification. It should be understood that the preferred embodiments described herein are only used to illustrate and explain the present application, and are not used to limit the present application. In addition, the embodiments and features in the embodiments of the present application may be combined with each other if there is no conflict.

[0052] Figure 2 A flowchart of a multi-sensor adaptive collision avoidance method based on deep learning is provided in an embodiment of the present application. Figure 2 As shown, the method may include:

[0053] Step S210: Acquire the ship position of the target ship and multiple categories of obstacle data.

[0054] The ship's position is obtained through the positioning system on the target ship.

[0055] Various types of sensors are installed at the corresponding positions of the target ship, such as radar, LiDAR sensor and camera.

[0056] The radar collects the distance of the obstacle from the target ship, the speed of the obstacle, and the azimuth of the obstacle;

[0057] Obstacle data collected by radar D r It can be expressed as: D r ={d r ,θ r ,v r}

[0058] Among them, dr is the distance between the obstacle and the target ship, θ r is the azimuth angle, and vr is the obstacle speed.

[0059] The LiDAR sensor collects point cloud data of the first area, which reflects the shape and structure of the surrounding environment. The first area is the largest environmental area where the LiDAR sensor can collect point cloud data.

[0060] The data collected by the LiDAR sensor can be expressed as: L ={p L1 ,p L2 ,…,p Ln};

[0061] Among them, P L The LiDAR sensor collects the spatial coordinates of each point corresponding to the environment, so that the spatial coordinates of the obstacle can be determined.

[0062] The camera captures an image of the second area; the second area is the maximum environment area in which the camera can capture images;

[0063] The data collected by the camera can be expressed as: C = {I C1 ,I C2 ,…,I CM};

[0064] Among them, I C is the feature region extracted from the image.

[0065] Afterwards, since the time intervals for collecting obstacle data by sensors of different categories are different, the obstacle data collected by sensors of different categories are time synchronized to obtain obstacle data of different categories at the same time point.

[0066] Since the target ship is located at a different position for each category of sensors, the obstacle data collected is not in the same coordinate system. It is necessary to perform coordinate transformation on the obstacle data of each category to obtain multiple categories of obstacle data in the same coordinate system. For example, the obstacle data of each category is transformed into the geodetic coordinate system for representation.

[0067] In some embodiments, in order to obtain more accurate obstacle data, denoising is performed on obstacle data of each category.

[0068] After that, the obstacle data of multiple categories of the same obstacle are determined, that is, the standardized obstacle data X is obtained. r , X L , X C :X r , X L , X C They are all non-fixed values, but dynamic coordinate values, representing the coordinate values ​​of the obstacle position measured by the radar; the coordinate values ​​of the environmental point cloud data captured by the LiDAR sensor, and the coordinate values ​​of the obstacle in the three-dimensional space in the image captured by the camera.

[0069] Step S220: adopt a self-attention mechanism to fuse the processed obstacle data of multiple categories of the same obstacle to obtain a fused feature vector.

[0070] Specifically, a Transformer-based self-attention mechanism is used for data fusion, that is, the dependency between sensor data is captured through a multi-head attention mechanism.

[0071] The input of the Transformer's self-attention mechanism is X r , X L and X C .

[0072] Attention mechanism: Calculate query Q, key K, value V:

[0073] Q=W q ·X;

[0074] K=W k ·X;

[0075] V=W v ·X;

[0076] Among them, W q , W k , W v is the learned weight matrix and X is the input data.

[0077] Calculate the attention weights:

[0078] Among them, d k is the dimension of the key, which is a constant.

[0079] The final fused feature vector X fused :

[0080] X fused =Concat(Attention1,Attention2,…,Attention h )

[0081] Output: The fused feature vector Xfused represents the obstacle position, obstacle speed, obstacle category and other information of the obstacle. In other words, the fused feature vector represents the obstacle position, obstacle speed and obstacle category.

[0082] Step S230: Use YOLOv5 to process the fused feature vector to determine the bounding box of the obstacle, the bounding box position corresponding to the bounding box, and the bounding box speed.

[0083] Specifically, this step can also be understood as the detection and classification of obstacles. YOLOv5 is used to detect the fused feature vector, and the detected obstacles are assigned a bounding box b = (x, y, w, h), where x, y are the center coordinates of the box, and w, h are the width and height of the box.

[0084] L YOLO =L box +L cls +L obj

[0085] Among them, L box is the bounding box regression loss, L cls is the classification loss, L obj is the target confidence loss;

[0086]

[0087] Output: Bounding box category C, bounding box position b and o is the confidence of the existence of obstacles, indicating whether each cell contains obstacles; x i ,y i Respectively represent the predicted position coordinates of the obstacles; They respectively represent the actual position coordinates of the obstacles determined by the data detected by multiple categories of sensors.

[0088] Step S240: Process the bounding box position and bounding box velocity through a convolutional neural network to obtain future trajectory data of the obstacle and the collision probability between the obstacle and the target ship.

[0089] Specifically, Transformer or LSTM is used to predict the future trajectory of obstacles.

[0090] Input: Obtain the bounding box position and bounding box velocity corresponding to the obstacle at the historical moment, that is, the historical trajectory data of the obstacle:

[0091] {(x1, y1, v x1 , v y1 ), (x2, y2, v x2 , v y2 ),…,(x n ,y n , v xn , v yn )}

[0092] Among them, x n ,y n They represent the coordinates corresponding to the position of the bounding box at the historical moment, v xn , v yn They represent the bounding box velocities in different directions at the historical moments.

[0093] Model structure:

[0094] Use LSTM for time series modeling to extract time-dependent features of historical trajectories:

[0095] h t =LSTM(h t－1 , x t )

[0096] Among them, h t－1 represents the vector of LSTM at time t-1, which includes the network's understanding of the input at the current moment and also includes the information transmitted from the previous moment. t is the input data at time t.

[0097] Use Transformer's self-attention mechanism to capture long-term dependencies:

[0098] h t =Attention(Q, K, V)

[0099] Output the predicted point of the future trajectory (x t+1 ,y t+1 ) and the collision probability P collide .

[0100] Output: Predicted position information and collision probability of future trajectories.

[0101] Step S250: Determine the heading and speed of the target ship based on the future trajectory data, the ship position and the collision probability.

[0102] Specifically, the configured deep reinforcement learning model is used to process the collision probability, future trajectory data, the ship speed and ship position of the target ship to obtain the heading and speed of the target ship.

[0103] This deep reinforcement learning model is a DRL model, which is used to optimize the ship's path planning, ensuring collision avoidance while optimizing navigation efficiency.

[0104] State space: including the ship position s of the target ship t =(x t ,y t , v t ) and obstacle positions (can also be bounding box positions).

[0105] Action space: The target ship can choose the heading adjustment angle θ t and speed adjustment v t .

[0106] Q-value function: Learning the optimal strategy through a deep Q network (DQN):

[0107]

[0108] Among them, R t is the reward at the current moment, and γ is the discount factor.

[0109] Reward function: Combined with collision risk P collide , path length L path And energy consumption E, design reward function:

[0110] R=-P collide -L path -αE

[0111] Output: Optimal heading and speed adjustment instructions

[0112] a * =(θ * , v * )

[0113] In some embodiments, after obtaining the future trajectory data of the obstacle and the collision probability between the obstacle and the target ship, the method may further include:

[0114] The future trajectory data and ship position are processed through the configured risk assessment model to determine the risk assessment value.

[0115] Specifically, the risk assessment value P is calculated based on the intersection probability between the future position prediction of the obstacle and the ship's track. collide :

[0116]

[0117] Among them, d collision is the minimum distance between the obstacle trajectory and the ship track, and σ is the standard deviation of the risk assessment.

[0118] Decision logic: Based on collision risk value P collide and the preset threshold P threshold , determine the collision avoidance strategy.

[0119] If P collide >P threshold , then the collision avoidance strategy is to control the target ship to travel according to the said heading and speed.

[0120] If P collide ≤P threshold , the collision avoidance strategy is to continue to maintain the current navigation status.

[0121] Decision logic formula:

[0122]

[0123] Output: Control instructions based on decision logic, such as adjusting heading, speed, or initiating emergency collision avoidance maneuvers.

[0124] Afterwards, when the processor determines that the risk assessment value exceeds the preset threshold, it automatically takes over the control of the target ship and adjusts the course and speed in real time to avoid a collision.

[0125] A. You can use PID controller to achieve smooth adjustment of heading and speed:

[0126]

[0127] Among them, K p , K d , K i are proportional, differential and integral gains, s t and target are the current state of the target ship and the state of the obstacle respectively.

[0128] B. Human control is possible: When the risk assessment value is lower than the preset threshold, the system will allow the crew to manually control it, or provide auxiliary control to improve decision-making efficiency.

[0129] Manual intervention rules:

[0130] When the collision risk P collide ≤P threshold And when there is no emergency, the crew is allowed to manually intervene in the control.

[0131] When the crew is in control, real-time data and auxiliary decision support such as collision prediction and collision avoidance advice are provided.

[0132] In another embodiment, the processor will automatically enter redundant mode when sensor data fails (such as radar signal loss, LiDAR data anomaly), switch to other valid sensors, and restart the data fusion and decision-making mechanism. Under extreme conditions (the optimal collision avoidance path cannot be fully determined), emergency avoidance strategies (such as deceleration) will be activated to minimize the risk of collision and ensure the safety of crew and ship.

[0133] The present application provides a multi-sensor adaptive collision avoidance method based on deep learning, the method comprising: obtaining a ship position and multi-category obstacle data of a target ship; the multi-category obstacle data is collected by multi-category sensors installed on the target ship; using a self-attention mechanism to fuse multiple obstacle data to obtain a fused feature vector; using YOLOv5 to process the fused feature vector to determine a bounding box of the obstacle, a bounding box position corresponding to the bounding box, and a bounding box speed; using a convolutional neural network to process the bounding box position and the bounding box speed to obtain future trajectory data of the obstacle and a collision probability between the obstacle and the target ship; and determining a heading and a speed of the target ship based on the future trajectory data, the ship position, and the collision probability. Through the fusion of multiple sensors and the application of deep learning algorithms, this application can achieve high-precision collision prediction and risk assessment in complex sea conditions and low visibility; it can make real-time adjustments and decisions based on the current navigation status of the ship, environmental changes, and dynamic changes of the target, avoiding the limitations of traditional collision avoidance systems; through intelligent data fusion and target recognition algorithms, it greatly reduces the risk of false alarms and missed alarms and improves the reliability of the system; when a potential collision occurs, the system can quickly generate the optimal collision avoidance path, reduce the operating pressure of the crew, and ensure navigation safety; on the premise of ensuring the safety of the crew, it can automatically perform risk prediction, path planning, and collision avoidance decisions, significantly improving the degree of automation of the ship.

[0134] Corresponding to the above method, the embodiment of the present application also provides a multi-sensor adaptive collision avoidance device based on deep learning, such as Figure 3 As shown, the device comprises:

[0135] The acquisition unit 310 is used to acquire the ship position of the target ship and multiple types of obstacle data; the multiple types of obstacle data are collected by multiple types of sensors installed on the target ship;

[0136] A fusion unit 320 is used to fuse multiple obstacle data using a self-attention mechanism to obtain a fusion feature vector;

[0137] A processing unit 330 is used to process the fused feature vector using YOLOv5 to determine a bounding box of the obstacle, a bounding box position corresponding to the bounding box, and a bounding box velocity;

[0138] and, processing the bounding box position and the bounding box velocity through a convolutional neural network to obtain future trajectory data of the obstacle and a collision probability between the obstacle and the target ship;

[0139] The determination unit 340 is used to determine the heading and speed of the target ship based on the future trajectory data, the ship position and the collision probability.

[0140] The functions of each functional unit of a multi-sensor adaptive collision avoidance device based on deep learning provided in the above-mentioned embodiment of the present application can be realized through the above-mentioned method steps. Therefore, the specific working process and beneficial effects of each unit in a multi-sensor adaptive collision avoidance device based on deep learning provided in the embodiment of the present application will not be repeated here.

[0141] The present application also provides an electronic device, such as Figure 4 As shown, it includes a processor 410 , a communication interface 420 , a memory 430 and a communication bus 440 , wherein the processor 410 , the communication interface 420 , and the memory 430 communicate with each other via the communication bus 440 .

[0142] Memory 430, for storing computer programs;

[0143] The processor 410 is used to execute the program stored in the memory 430 to implement the following steps:

[0144] Obtain the ship position and multi-category obstacle data of the target ship; the multi-category obstacle data is collected by multi-category sensors installed on the target ship;

[0145] The self-attention mechanism is used to fuse multiple obstacle data to obtain a fused feature vector;

[0146] Using YOLOv5, the fused feature vector is processed to determine the bounding box of the obstacle, the bounding box position corresponding to the bounding box, and the bounding box velocity;

[0147] The bounding box position and the bounding box velocity are processed by a convolutional neural network to obtain future trajectory data of the obstacle and a collision probability between the obstacle and the target ship;

[0148] The heading and speed of the target ship are determined based on the future trajectory data, the ship position and the collision probability.

[0149] The communication bus mentioned above can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.

[0150] The communication interface is used for communication between the above electronic device and other devices.

[0151] The memory may include a random access memory (RAM) or a non-volatile memory (NVM), such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.

[0152] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0153] The implementation methods and beneficial effects of the components of the electronic device in the above embodiments to solve the problems can be seen in Figure 2 The various steps in the illustrated embodiment are implemented, therefore, the specific working process and beneficial effects of the electronic device provided by the embodiment of the present application are not repeated here.

[0154] In another embodiment provided in the present application, a computer-readable storage medium is also provided, in which instructions are stored. When the computer-readable storage medium is executed on a computer, the computer executes a multi-sensor adaptive collision avoidance method based on deep learning as described in any of the above embodiments.

[0155] In another embodiment provided in the present application, a computer program product including instructions is also provided. When the computer program product is executed on a computer, the computer executes a multi-sensor adaptive collision avoidance method based on deep learning as described in any one of the above embodiments.

[0156] Those skilled in the art will appreciate that the embodiments in the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt a complete hardware embodiment, a complete software embodiment, or a form of an embodiment combining software and hardware. Moreover, the present application may adopt a form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0157] The embodiments of the present application are described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0158] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0159] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0160] Unless otherwise defined, the technical terms or scientific terms used in this application should be understood by people with ordinary skills in the field to which the present invention belongs. "First", "second" and similar words used in this application do not indicate any order, quantity or importance, but are only used to distinguish different components. "Include" or "comprise" and similar words mean that the elements or objects appearing before the word cover the elements or objects listed after the word and their equivalents, without excluding other elements or objects. "Connect", "couple" or "connected" and similar words are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. "Up", "down", "left", "right" and the like are only used to indicate relative positional relationships. When the absolute position of the described object changes, the relative positional relationship may also change accordingly.

[0161] Although the preferred embodiments of the present application have been described, those skilled in the art may make additional changes and modifications to these embodiments once they are aware of the basic creative concepts. Therefore, the present application embodiments are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present application embodiments.

[0162] Obviously, those skilled in the art can make various changes and modifications to the embodiments in the present application without departing from the spirit and scope of the embodiments in the present application. Thus, if these modifications and variations of the embodiments in the present application are within the scope of the embodiments in the present application and their equivalents, the embodiments in the present application are also intended to include these modifications and variations.

Claims

1. A multi-sensor adaptive collision avoidance method based on deep learning, characterized in that: The method comprises: Obtain the ship position and multi-category obstacle data of the target ship; the multi-category obstacle data is collected by multi-category sensors installed on the target ship; The self-attention mechanism is used to fuse multiple obstacle data to obtain a fused feature vector; Using YOLOv5, the fused feature vector is processed to determine the bounding box of the obstacle, the bounding box position corresponding to the bounding box, and the bounding box velocity; The bounding box position and the bounding box velocity are processed by a convolutional neural network to obtain future trajectory data of the obstacle and a collision probability between the obstacle and the target ship; The heading and speed of the target ship are determined based on the future trajectory data, the ship position and the collision probability.

2. The method according to claim 1, characterized in that Multiple categories of sensors include: radar, LiDAR sensors and cameras.

3. The method according to claim 1, characterized in that Before fusing the multi-category obstacle data using the self-attention mechanism, the method further includes: Perform time synchronization, denoising and coordinate conversion on obstacle data collected by sensors of various categories to obtain processed obstacle data; The self-attention mechanism is used to fuse the processed multi-category obstacle data.

4. The method according to claim 1, characterized in that After obtaining the future trajectory data of the obstacle and the collision probability between the obstacle and the target ship, the method includes: The future trajectory data and ship position are processed through the configured risk assessment model to determine the risk assessment value; A collision avoidance strategy is determined according to the risk assessment value.

5. The method according to claim 4, characterized in that Determine a collision avoidance strategy based on the risk assessment value, including: If the risk assessment value is greater than a preset threshold value, the collision avoidance strategy is to control the target ship to travel according to the heading and the speed.

6. The method according to claim 1, characterized in that Determining the heading and speed of the target ship based on the future trajectory data, the ship position and the collision probability includes: The configured deep reinforcement learning model is used to process the collision probability, the future trajectory data, the ship speed of the target ship and the ship position to obtain the heading and speed of the target ship.

7. A multi-sensor adaptive collision avoidance device based on deep learning, characterized in that: The device comprises: An acquisition unit, used to acquire the ship position of the target ship and multi-category obstacle data; the multi-category obstacle data is collected by multi-category sensors installed on the target ship; A fusion unit is used to fuse multiple obstacle data using a self-attention mechanism to obtain a fused feature vector; A processing unit, configured to process the fused feature vector using YOLOv5 to determine a bounding box of the obstacle, a bounding box position corresponding to the bounding box, and a bounding box velocity; and, processing the bounding box position and the bounding box velocity through a convolutional neural network to obtain future trajectory data of the obstacle and a collision probability between the obstacle and the target ship; A determination unit is used to determine the heading and speed of the target ship based on the future trajectory data, the ship position and the collision probability.

8. An electronic device, characterized in that: The electronic device comprises a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other via the communication bus; Memory, used to store computer programs; A processor, for implementing the method steps described in any one of claims 1 to 6 when executing a program stored in a memory.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method steps described in any one of claims 1 to 6 are implemented.

Images