Multimodal fusion underwater environment intelligent perception system and method

Abstract

The present invention discloses a multi-modal fusion underwater environment intelligent perception system and method thereof, and relates to the technical field of multi-modal data processing. A sonar device is used to detect whether there is an obstacle in front of a detector. If so, depth image data in front is collected to obtain entity data, and a movement prediction model of the obstacle is trained based on the depth image data. The movement data of the obstacle is obtained based on the movement prediction model prediction. Based on the entity data and movement data of the obstacle and the current movement path of the detector, it is determined whether a collision will occur. If a collision will occur, based on the time when the collision is determined to occur, it is determined whether the obstacle needs to be bypassed. If bypassing is required, a new movement path is planned based on the entity data and movement data and the target location and current position of the detector. The intelligence level and task adaptability of the underwater detector are improved to ensure the smooth completion of the operation and the safe operation of the equipment.

Description

Technical Field

[0001] The present invention relates to the technical field of multimodal data processing, and in particular to a multimodal fusion underwater environment intelligent perception system and method thereof. Background Art

[0002] At present, underwater detectors play an indispensable role in the fields of marine environment monitoring, pipeline inspection, underwater archaeology, etc. They can penetrate into complex underwater environments, perform various detection, monitoring and operation tasks, and provide humans with valuable seabed information and work support. However, the underwater environment is complex and dynamic, which brings great challenges to the safe operation of detectors.

[0003] The distribution of underwater obstacles is highly uncertain and dynamic. Fixed obstacles such as shipwrecks and rocks are randomly distributed and it is difficult to obtain accurate information in advance; the movement trajectories of mobile obstacles such as floating objects in ocean currents and biological communities are more complex and changeable. In addition, there are a large number of unknown hidden risks in the underwater environment, such as sudden mudslides and fragile ecological structures, which will lead to serious consequences once a collision occurs. Therefore, how to ensure that the detector moves safely and effectively in this high-risk unknown environment is a major challenge that needs to be solved urgently.

[0004] Existing technologies have many deficiencies in obstacle detection and path planning, making it difficult to achieve accurate and effective obstacle avoidance capabilities. Traditional perception technologies, such as sonar and vision, can only detect static obstacle information at the current moment and cannot predict future motion trajectories, resulting in lag and passivity in obstacle avoidance plans. In addition, most existing path planning algorithms only consider the geometric shape of obstacles, ignoring important factors such as size and motion, and the generated paths have a greater risk of collision. Although some improved planning methods take motion into account, they can only handle simple motion models and cannot accurately describe complex nonlinear motion trajectories, and the obstacle avoidance effect cannot be guaranteed.

[0005] To this end, the present invention proposes a multi-modal fusion underwater environment intelligent perception system and method. Summary of the invention

[0006] The present invention aims to solve at least one of the technical problems existing in the prior art. To this end, the present invention proposes a multi-modal fusion underwater environment intelligent perception system and method thereof, which improves the intelligence level and task adaptability of underwater detectors, ensures the smooth completion of operations and the safe operation of equipment.

[0007] To achieve the above objectives, a multi-modal fusion underwater environment intelligent perception method is proposed, which includes the following steps:

[0008] Step 1: Use the sonar device to detect whether there is an obstacle in front of the detector. If there is, go to step 2, otherwise repeat step 1;

[0009] Step 2: Collect the depth image data in front, obtain the body data based on the depth image data, train the obstacle movement prediction model based on the depth image data, and obtain the obstacle movement data based on the movement prediction model prediction;

[0010] Step 3: Based on the obstacle's body data and movement data and the detector's current movement path, determine whether a collision will occur. If a collision will occur, go to step 4; otherwise, loop steps 2 to 3;

[0011] Step 4: Based on the time when the collision is determined to have occurred, determine whether the obstacle needs to be bypassed. If it needs to be bypassed, execute steps 5 and 6; otherwise, execute step 6;

[0012] Step 5: Plan a new movement path based on the entity data, movement data, and the target location and current location of the detector;

[0013] Step 6: Repeat steps 2 to 6 until the target location is reached;

[0014] The method of detecting whether there is an obstacle in front of the detector by using a sonar device comprises the following steps:

[0015] Step 11: The sonar device carried by the detector transmits a sound wave pulse signal s(t) forward and receives the returned reflected echo signal r(t);

[0016] Step 12: Define the delay variable , represents the time delay between the transmitted signal and the received signal;

[0017] Step 13: Calculate the cross-correlation function R( ), R( )The time delay at which the peak value occurs is marked as τ_p;

[0018] Step 14: Calculate the distance d to the nearest obstacle based on the speed of sound c;

[0019] Step 15: If d is greater than or equal to the preset distance threshold, it is determined that there is no obstacle; otherwise, it is determined that there is an obstacle;

[0020] The obtaining of the body data based on the depth image data comprises the following steps:

[0021] Step 21: performing image preprocessing on the depth image data;

[0022] Step 22: Use a depth image segmentation algorithm to segment obstacles in the depth image data, and reproject the depth values ​​of the obstacle depth image obtained by segmentation into a three-dimensional space to obtain a three-dimensional point cloud of the obstacle;

[0023] Step 23: Use the trained 3D shape classification model to classify the point cloud and output the obstacle category and size estimation as the ontology data;

[0024] The step of training a movement prediction model of an obstacle based on the depth image data and obtaining movement data of the obstacle based on the movement prediction model comprises the following steps:

[0025] Step 31: Using a visual tracking system, record the actual movement trajectory of the obstacle within a preset monitoring time;

[0026] Step 32: During the monitoring period, use the depth camera to synchronously collect depth image data of obstacles at different times to form time series depth data;

[0027] Step 33: Associating the actual motion trajectory of the obstacle with the corresponding time series depth data as a training data set;

[0028] Step 34: Using a 3D convolutional neural network as a feature extractor, inputting the temporal depth data into the 3D convolutional neural network, and the 3D convolutional neural network finally outputs a sequence of spatiotemporal feature maps;

[0029] Step 35: Use the convolutional LSTM or the attention-based sequence prediction model as the trajectory prediction model, and use the spatiotemporal feature map sequence as the input of the trajectory prediction model. At each time step, the trajectory prediction model outputs the trajectory state of the next moment according to the historical spatiotemporal feature map sequence. The trajectory state includes position and speed. After accumulating the predictions of multiple time steps, the motion trajectory of the next T time steps is output; T is the preset prediction duration.

[0030] Step 36: Take the subsequent speed and position of the obstacle within the monitoring time as the prediction target, take the Euclidean distance between the trajectory state and the prediction target as the prediction error, and use the average value of the prediction error at each time step as the loss function to train the trajectory prediction model and obtain a trained mobile prediction model;

[0031] Step 37: Based on the latest actual motion trajectory of the obstacle, the sequence prediction model outputs the trajectory state at the next T time as the input of the 3D convolutional neural network to form the movement data of the obstacle;

[0032] The step of determining whether a collision will occur based on the obstacle's body data and movement data and the detector's current movement path comprises the following steps:

[0033] Step 41: mark the length, width and height of the obstacle as Zl, Zw and Zh respectively; mark the position sequence and velocity sequence of the obstacle in the future T time steps as {p_obj(t), p_obj(t+1), ..., p_obj(t+T)} and {v_obj(t), v_obj(t+1), ..., v_obj(t+T)} respectively;

[0034] Step 42: mark the radius size of the detector as r, the current position as p_c, the direction as dir, the position sequence corresponding to the moving path as {p_0, p_1, ..., p_n} and the velocity sequence as {v_0, v_1, ..., v_n}; n is the time required to reach the target location from the current position according to the moving path;

[0035] Step 43: For each time step k, k=0, 1...n:

[0036] Based on the position p_c(k) and velocity v(k) of the detector at the kth time step, calculate its motion trajectory in the next T time steps: {p_exp(k), p_exp(k+1), ..., p_exp(k+T)};

[0037] For each time step i, i=0,...T in the obstacle position and velocity sequence, perform:

[0038] Get the position p_obj(k+i) and velocity v_obj(k+i) of the obstacle at time k+i;

[0039] Construct a three-dimensional bounding box of the obstacle at time k+i, with p_obj(k+i) as the center and length, width and height of the box being (Zl+vx, Zw+vy, Zh+vz), where (vx, vy, vz) are the components of v_obj(t+i) in the x, y and z dimensions, and the x, y and z axes are the three coordinate axes of the spatial rectangular coordinate system;

[0040] Determine whether the position p_exp(k+i) of the detector at time k+i is within the three-dimensional bounding box; if so, determine that a collision will occur and mark the time step k+i as k0; otherwise, continue to determine the next time step;

[0041] Step 44: If no collision is detected in all time steps k, it is determined that no collision will occur;

[0042] The method of judging whether the obstacle needs to be bypassed based on the time when the collision is judged to occur is:

[0043] If it is determined that the time step k0 at which a collision will occur is less than the preset avoidance time threshold; otherwise, it is determined that no detour is required;

[0044] The planning of a new moving path based on the body data and the moving data as well as the target location and the current location of the detector comprises the following steps:

[0045] Step 51: construct a detector position optimization variable set S = {p_0, p_1, ..., p_m}, p_j = (x_j, y_j, z_j), j = 1, 2, 3 ... m; m is the preset maximum time to reach the target location;

[0046] Step 52: Construct the optimization objective function. ;in;

[0047] Step 53: Construct a set of constraints:

[0048] p_0=p0; p0 is the current position of the detector;

[0049] p_m=pm; pm is the target location of the detector;

[0050] ||p_j - p_{j+1}|| <= d_max, j=0,...,m-1; where d_max is the maximum moving speed of the detector;

[0051] min_dist(p_j) > R, ; Where min_dist(p_j) is the minimum distance from point p to the center of the obstacle's 3D bounding box at the jth moment; R is half of the maximum value of the length, width, and height of the 3D bounding box;

[0052] Step 54: Taking the optimization objective function as the optimization objective and the constraint condition set as the constraint, it is formalized into a constrained optimization problem. The constrained optimization problem is solved by using the optimization algorithm to obtain the solution set of the optimization variable set S. According to the position of the detector per unit time in the solution set, a new moving path is constructed.

[0053] A multi-modal fusion underwater environment intelligent perception system is proposed, which includes an obstacle detection module, an obstacle data collection module, a collision detection module and a detour module; wherein all modules are electrically connected;

[0054] The obstacle detection module detects whether there is an obstacle in front of the detector through the sonar device. If there is, the obstacle data collection module is started;

[0055] The obstacle data collection module collects the depth image data in front, obtains the main body data based on the depth image data, trains the movement prediction model of the obstacle based on the depth image data, obtains the movement data of the obstacle based on the movement prediction model, and sends the main body data and the movement data to the collision detection module;

[0056] The collision detection module determines whether a collision will occur based on the obstacle's body data and movement data and the detector's current movement path. If a collision is determined to occur, the time when the collision is determined to occur and the obstacle's body data and movement data are sent to the detour module.

[0057] The bypass module determines whether the obstacle needs to be bypassed based on the time when the collision is determined to have occurred. If bypassing is required, a new movement path is planned based on the body data and movement data as well as the target location and current position of the detector.

[0058] An electronic device comprises: a processor and a memory, wherein the memory stores a computer program that can be called by the processor; the processor executes the above-mentioned multimodal fusion underwater environment intelligent perception system and method by calling the computer program stored in the memory.

[0059] A computer-readable storage medium stores instructions, which, when executed on a computer, enable the computer to execute the above-mentioned multi-modal fusion underwater environment intelligent perception system and method.

[0060] Compared with the prior art, the present invention has the following beneficial effects:

[0061] The present invention detects whether there is an obstacle in front of the detector through a sonar device. If there is, the depth image data in front is collected, and the body data is obtained based on the depth image data. The movement prediction model of the obstacle is trained based on the depth image data, and the movement data of the obstacle is predicted based on the movement prediction model. Based on the body data and movement data of the obstacle and the current movement path of the detector, it is judged whether a collision will occur. If a collision will occur, it is judged whether the obstacle needs to be bypassed based on the time when the collision is judged to occur. If bypassing is required, a new movement path is planned based on the body data and movement data and the target location and current position of the detector; firstly, based on the body data and movement trajectory data of the obstacle and the current movement path of the detector, through distance calculation, it is judged whether there is a collision risk. If there is a risk, it will comprehensively consider the obstacle size, future precise position and speed change, and generate an optimal new movement path that meets the safety distance requirement through the distance constraint between the dynamic three-dimensional bounding box and the movement trajectory. The obstacle data, movement data and the detector's own status data are integrated to model the constrained optimization problem, which improves the level of intelligent detour decision-making. The technical route unifies the three links of collision detection, path prediction and detour planning, improves the intelligence level and task adaptability of the detector, thereby improving the intelligence level and task adaptability of the underwater detector, ensuring the smooth completion of the operation and the safe operation of the equipment. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] Figure 1 This is a flow chart of the multi-modal fusion underwater environment intelligent perception method in Example 1 of the present invention;

[0063] Figure 2 This is a module connection relationship diagram of the multi-modal fusion underwater environment intelligent perception system in Example 2 of the present invention;

[0064] Figure 3 This is a schematic diagram of the structure of an electronic device in Embodiment 3 of the present invention;

[0065] Figure 4 This is a schematic diagram of the computer-readable storage medium structure in Example 4 of the present invention. DETAILED DESCRIPTION

[0066] The technical solution of the present invention will be clearly and completely described below in conjunction with the embodiments. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention. Example 1

[0067] like Figure 1 As shown, the multi-modal fusion underwater environment intelligent perception method includes the following steps:

[0068] Step 1: Use the sonar device to detect whether there is an obstacle in front of the detector. If there is, go to step 2, otherwise repeat step 1;

[0069] Step 2: Collect the depth image data in front, obtain the body data based on the depth image data, train the obstacle movement prediction model based on the depth image data, and obtain the obstacle movement data based on the movement prediction model prediction;

[0070] Step 3: Based on the obstacle's body data and movement data and the detector's current movement path, determine whether a collision will occur. If a collision will occur, go to step 4; otherwise, loop steps 2 to 3;

[0071] Step 4: Based on the time when the collision is determined to have occurred, determine whether the obstacle needs to be bypassed. If it needs to be bypassed, execute steps 5 and 6; otherwise, execute step 6;

[0072] Step 5: Plan a new movement path based on the entity data, movement data, and the target location and current location of the detector;

[0073] Step 6: Repeat steps 2 to 6 until the target location is reached;

[0074] Specifically, the method of detecting whether there is an obstacle in front of the detector by using a sonar device includes the following steps:

[0075] Step 11: The sonar device carried by the detector transmits a sound wave pulse signal s(t) to the front and receives the returned reflected echo signal r(t); specifically, s(t) is a time function, which represents the amplitude value of the transmitted signal at different time points; r(t) is also a time function, which represents the amplitude value of the received signal at different time points;

[0076] Step 12: Define the delay variable , represents the time delay between the transmitted signal and the received signal;

[0077] Step 13: Calculate the cross-correlation function R( ), R( )The time delay of the peak is marked as τ_p; wherein the cross-correlation function R(t) is calculated as: , is the integral symbol;

[0078] Step 14: Calculate the distance d to the nearest obstacle based on the speed of sound c; the calculation formula for the distance d to the nearest obstacle is: ;

[0079] Step 15: If d is greater than or equal to the preset distance threshold, it means that the nearest obstacle is still far away and is not dangerous, and it is determined that there is no obstacle; otherwise, it is determined that there is an obstacle;

[0080] Furthermore, the method of collecting the front depth image data may be to collect an underwater three-dimensional depth image through a depth camera (such as a structured light or ToF camera);

[0081] It should be noted that the depth image data provides three-dimensional geometric information for the scene, can generate three-dimensional point cloud or mesh data, reconstruct the three-dimensional model of the scene, and estimate the three-dimensional size (such as length, width and height) and volume of the detected target;

[0082] Furthermore, the obtaining of the body data based on the depth image data comprises the following steps:

[0083] Step 21: performing image preprocessing on the depth image data; the image preprocessing includes but is not limited to denoising filtering (such as Gaussian filtering) to reduce the noise impact of the depth data, spatial filtering (such as median filtering) to remove isolated noise points in the depth image, and depth image registration: correcting lens distortion to synchronize with other data;

[0084] Step 22: Use a depth image segmentation algorithm to segment obstacles in the depth image data, and reproject the depth values ​​of the obstacle depth image obtained by segmentation into a three-dimensional space to obtain a three-dimensional point cloud of the obstacle;

[0085] Step 23: Use the trained 3D shape classification model to classify the point cloud and output the obstacle category and size estimation value as the ontology data; the 3D shape classification model can be a 3D convolutional network, PointNet++, etc.; the size estimation value includes the length, width and height of the obstacle;

[0086] Furthermore, the training of the obstacle movement prediction model based on the depth image data and obtaining the obstacle movement data based on the movement prediction model includes the following steps:

[0087] Step 31: Use a visual tracking system to record the actual movement trajectory of the obstacle within a preset monitoring time. Specifically, the visual tracking system can calculate and analyze the movement trajectory of the obstacle within a unit time by using the position change of the obstacle in adjacent unit time by a depth camera;

[0088] Step 32: During the monitoring period, use the depth camera to synchronously collect depth image data of obstacles at different times to form time series depth data;

[0089] Step 33: Associating the actual motion trajectory of the obstacle with the corresponding time series depth data as a training data set;

[0090] Step 34: Using a 3D convolutional neural network as a feature extractor, the temporal depth data is input into the 3D convolutional neural network, and the 3D convolutional neural network finally outputs a sequence of spatiotemporal feature maps, encoding the shape and motion information of the obstacle at different times; it can be understood that the 3D convolution kernel slides in both time and space dimensions to extract the comprehensive features of motion and shape, and the 3D pooling layer in the network gradually reduces the feature resolution, thereby obtaining a multi-scale, high-level feature representation;

[0091] Step 35: Use the convolutional LSTM or the attention-based sequence prediction model as the trajectory prediction model, and use the spatiotemporal feature map sequence as the input of the trajectory prediction model. At each time step, the trajectory prediction model outputs the trajectory state of the next moment according to the historical spatiotemporal feature map sequence. The trajectory state includes position and speed. After accumulating the predictions of multiple time steps, the motion trajectory of the next T time steps is output. T is the preset prediction duration, which indicates the duration of the future trajectory of the obstacle that needs to be predicted.

[0092] Step 36: Take the subsequent speed and position of the obstacle within the monitoring time as the prediction target, take the Euclidean distance between the trajectory state and the prediction target as the prediction error, and use the average value of the prediction error at each time step as the loss function to train the trajectory prediction model and obtain a trained mobile prediction model;

[0093] Step 37: Based on the latest actual motion trajectory of the obstacle, the sequence prediction model outputs the trajectory state at the next T time as the input of the 3D convolutional neural network to form the movement data of the obstacle;

[0094] Furthermore, judging whether a collision will occur based on the obstacle's body data and movement data and the detector's current movement path includes the following steps:

[0095] Step 41: mark the length, width and height of the obstacle as Zl, Zw and Zh respectively; mark the position sequence and velocity sequence of the obstacle in the future T time steps as {p_obj(t), p_obj(t+1), ..., p_obj(t+T)} and {v_obj(t), v_obj(t+1), ..., v_obj(t+T)} respectively;

[0096] Step 42: mark the radius size of the detector as r, the current position as p_c, the direction as dir, the position sequence corresponding to the moving path as {p_0, p_1, ..., p_n} and the velocity sequence as {v_0, v_1, ..., v_n}; n is the time required to reach the target location from the current position according to the moving path;

[0097] Step 43: For each time step k, k=0, 1...n:

[0098] Based on the position p_c(k) and velocity v(k) of the detector at the kth time step, calculate its motion trajectory in the next T time steps: {p_exp(k), p_exp(k+1), ..., p_exp(k+T)};

[0099] For each time step i, i=0,...T in the obstacle position and velocity sequence, perform:

[0100] Get the position p_obj(k+i) and velocity v_obj(k+i) of the obstacle at time k+i;

[0101] Construct a three-dimensional bounding box of the obstacle at time k+i, wherein the three-dimensional bounding box is centered at p_obj(k+i), and its length, width and height are (Zl+vx, Zw+vy, Zh+vz), respectively, wherein (vx, vy, vz) are the components of v_obj(t+i) in the three dimensions of x, y and z axes, and the x, y and z axes are the three coordinate axes of the spatial rectangular coordinate system; it can be understood that in the process of the movement of the obstacle, the coverage space range will change, because the actual space range occupied by it will change with the change of speed. For example, if an obstacle with a length of 2 meters and a width of 1 meter moves at a speed of 1 meter / second in the x direction, then after a unit of time, the range it covers in the x direction will increase to 2+1=3 meters.

[0102] Determine whether the position p_exp(k+i) of the detector at time k+i is within the three-dimensional bounding box; if so, determine that a collision will occur and mark the time step k+i as k0; otherwise, continue to determine the next time step;

[0103] Step 44: If no collision is detected in all time steps k, it is determined that no collision will occur;

[0104] Furthermore, the method of judging whether the obstacle needs to be bypassed based on the time when the collision is judged to occur is:

[0105] If it is determined that the time step k0 at which a collision will occur is less than the preset avoidance time threshold, that is, a collision may occur in a short time, and therefore timely avoidance is required, then it is determined that a detour is required; otherwise, it is determined that a detour is not required, that is, the running trajectory of the obstacle can be continuously observed to determine whether there will be any new changes;

[0106] Furthermore, planning a new moving path based on the body data and the moving data as well as the target location and the current location of the detector includes the following steps:

[0107] Step 51: mark the spatial coordinates of the target location of the detector as (x_g, y_g, z_g), and the current position of the detector as (x_s, y_s, z_s);

[0108] Step 52: construct a detector position optimization variable set S = {p_0, p_1, ..., p_m}, p_j = (x_j, y_j, z_j), j = 1, 2, 3 ... m; m is the preset maximum time to reach the target location;

[0109] Step 53: Construct the optimization objective function. ; Wherein, || represents the Euclidean distance between two points in three-dimensional space; thus, the optimization objective function represents minimizing the total moving path length;

[0110] Step 54: Construct a set of constraints:

[0111] p_0=p0; p0 is the current position of the detector;

[0112] p_m=pm; pm is the target location of the detector; it can be understood that these two restrictions are used to limit the detector to reach the target location after a time length m from the current position;

[0113] ||p_j - p_{j+1}|| <= d_max, j=0,...,m-1; where d_max is the maximum moving speed of the detector; it can be understood that this limit is used to limit the distance moved by the detector per unit time to not exceed its speed limit;

[0114] min_dist(p_j) > R, ; Wherein, min_dist(p_j) is the minimum distance from point p to the center point of the three-dimensional bounding box of the obstacle at the jth moment; R is half of the maximum value of the length, width, height and height of the three-dimensional bounding box; It can be understood that the center point of the three-dimensional bounding box of the obstacle at the jth moment is p_obj(j); The length, width and height of the three-dimensional bounding box are (Zl+vx_j, Zw+vy_j, Zh+vz_j), where (vx_j, vy_j, vz_j) are the components of v_obj(j) in the three dimensions of the x, y and z axes;

[0115] Step 55: Taking the optimization objective function as the optimization objective and the constraint condition set as the constraint, it is formalized into a constrained optimization problem. The constrained optimization problem is solved by using the optimization algorithm to obtain the solution set of the optimization variable set S. According to the position of the detector per unit time in the solution set, a new moving path is constructed. Example 2

[0116] like Figure 2 As shown, the multi-modal fusion underwater environment intelligent perception system includes an obstacle detection module, an obstacle data collection module, a collision detection module and a detour module; wherein all modules are electrically connected;

[0117] The obstacle detection module detects whether there is an obstacle in front of the detector through the sonar device. If there is, the obstacle data collection module is started;

[0118] The obstacle data collection module collects the depth image data in front, obtains the main body data based on the depth image data, trains the movement prediction model of the obstacle based on the depth image data, obtains the movement data of the obstacle based on the movement prediction model, and sends the main body data and the movement data to the collision detection module;

[0119] The collision detection module determines whether a collision will occur based on the obstacle's body data and movement data and the detector's current movement path. If a collision is determined to occur, the time when the collision is determined to occur and the obstacle's body data and movement data are sent to the detour module.

[0120] The bypass module determines whether the obstacle needs to be bypassed based on the time when the collision is determined to have occurred. If bypassing is required, a new movement path is planned based on the body data and movement data as well as the target location and current position of the detector. Example 3

[0121] See also Figure 3 As shown, according to another aspect of the present application, an electronic device 500 is also provided. The electronic device 500 may include one or more processors and one or more memories. The memories store computer readable codes, and when the computer readable codes are run by one or more processors, the multi-modal fusion underwater environment intelligent perception system and method thereof as described above may be executed.

[0122] The method or system according to the embodiment of the present application can also be used by Figure 3 The electronic device architecture shown in FIG. Figure 3 As shown, the electronic device 500 may include a bus 501, one or more CPUs 502, a read-only memory (ROM) 503, a random access memory (RAM) 504, a communication port 505 connected to a network, an input / output component 506, a hard disk 507, etc. The storage device in the electronic device 500, such as the ROM 503 or the hard disk 507, may store the multi-modal fusion underwater environment intelligent perception system and method thereof provided by the present application. Furthermore, the electronic device 500 may also include a user interface 508. Of course, Figure 3 The architecture shown is only exemplary and can be omitted according to actual needs when implementing different devices. Figure 3One or more components of an electronic device are shown. Example 4

[0123] See also Figure 4 As shown, a computer-readable storage medium 600 according to an embodiment of the present application is shown. Computer-readable instructions are stored on the computer-readable storage medium 600. When the computer-readable instructions are executed by the processor, the multi-modal fusion underwater environment intelligent perception system and method according to the embodiment of the present application described with reference to the above figures can be executed. The storage medium 600 includes, but is not limited to, for example, volatile memory and / or non-volatile memory. Volatile memory may include, for example, random access memory (RAM) and cache memory (cache). Non-volatile memory may include, for example, read-only memory (ROM), hard disk, flash memory, etc.

[0124] In addition, according to the embodiments of the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the present application provides a non-transitory machine-readable storage medium, which stores machine-readable instructions, and the machine-readable instructions can be executed by a processor to execute instructions corresponding to the method steps provided by the present application;

[0125] The above embodiments can be implemented in whole or in part by software, hardware, firmware or any other combination. When implemented by software, the above embodiments can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, the process or function described in the embodiment of the present invention is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from one website site, computer, server or data center to another website site, computer, server or data center through a wired network or a wireless network. The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server or data center that contains one or more available media sets. The available medium can be a magnetic medium (for example, a floppy disk, a hard disk, a tape), an optical medium (for example, a DVD), or a semiconductor medium. The semiconductor medium can be a solid-state hard disk.

[0126] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed in the present invention can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0127] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0128] In the several embodiments provided by the present invention, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a multimodal fusion underwater environment intelligent perception system and method thereof. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0129] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0130] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0131] The above embodiments are only used to illustrate the technical method of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical method of the present invention may be modified or replaced by equivalents without departing from the spirit and scope of the technical method of the present invention.

Claims

1. A multi-modal fusion underwater environment intelligent perception method, characterized in that: The following steps are involved: Step 1: Use the sonar device to detect whether there is an obstacle in front of the detector. If there is, go to step 2, otherwise repeat step 1; Step 2: Collect the depth image data in front, obtain the body data based on the depth image data, train the obstacle movement prediction model based on the depth image data, and obtain the obstacle movement data based on the movement prediction model prediction; Step 3: Based on the obstacle's body data and movement data and the detector's current movement path, determine whether a collision will occur. If a collision will occur, go to step 4; otherwise, loop steps 2 to 3; Step 4: Based on the time when the collision is determined to have occurred, determine whether the obstacle needs to be bypassed. If it needs to be bypassed, execute steps 5 and 6; otherwise, execute step 6; Step 5: Plan a new movement path based on the entity data, movement data, and the target location and current location of the detector; Step 6: Repeat steps 2 to 6 until the target location is reached; The obtaining of the body data based on the depth image data comprises the following steps: Step 21: performing image preprocessing on the depth image data; Step 22: Use a depth image segmentation algorithm to segment obstacles in the depth image data, and reproject the depth values ​​of the obstacle depth image obtained by segmentation into a three-dimensional space to obtain a three-dimensional point cloud of the obstacle; Step 23: Use the trained 3D shape classification model to classify the point cloud and output the obstacle category and size estimation as the ontology data; The step of training a movement prediction model of an obstacle based on the depth image data and obtaining movement data of the obstacle based on the movement prediction model comprises the following steps: Step 31: Using a visual tracking system, record the actual movement trajectory of the obstacle within a preset monitoring time; Step 32: During the monitoring period, use the depth camera to synchronously collect depth image data of obstacles at different times to form time series depth data; Step 33: Associating the actual motion trajectory of the obstacle with the corresponding time series depth data as a training data set; Step 34: Using a 3D convolutional neural network as a feature extractor, inputting the temporal depth data into the 3D convolutional neural network, and the 3D convolutional neural network finally outputs a sequence of spatiotemporal feature maps; Step 35: Use the convolutional LSTM or the attention-based sequence prediction model as the trajectory prediction model, and use the spatiotemporal feature map sequence as the input of the trajectory prediction model. At each time step, the trajectory prediction model outputs the trajectory state of the next moment according to the historical spatiotemporal feature map sequence. The trajectory state includes position and speed. After accumulating the predictions of multiple time steps, the motion trajectory of the next T time steps is output; T is the preset prediction duration. Step 36: Take the subsequent speed and position of the obstacle within the monitoring time as the prediction target, take the Euclidean distance between the trajectory state and the prediction target as the prediction error, and use the average value of the prediction error at each time step as the loss function to train the trajectory prediction model and obtain a trained mobile prediction model; Step 37: Based on the latest actual motion trajectory of the obstacle, the sequence prediction model outputs the trajectory state at the next T time as the input of the 3D convolutional neural network to form the movement data of the obstacle; The step of determining whether a collision will occur based on the obstacle's body data and movement data and the detector's current movement path comprises the following steps: Step 41: mark the length, width and height of the obstacle as Zl, Zw and Zh respectively; mark the position sequence and velocity sequence of the obstacle in the future T time steps as {p_obj(t), p_obj(t+1), ..., p_obj(t+T)} and {v_obj(t), v_obj(t+1), ..., v_obj(t+T)} respectively; Step 42: mark the radius size of the detector as r, the current position as p_c, the direction as dir, the position sequence corresponding to the moving path as {p_0, p_1, ..., p_n} and the velocity sequence as {v_0, v_1, ..., v_n}; n is the time required to reach the target location from the current position according to the moving path; Step 43: For each time step k, k=0, 1, ..., n: Based on the position p_c(k) and velocity v(k) of the detector at the kth time step, calculate its motion trajectory in the next T time steps: {p_exp(k), p_exp(k+1), ..., p_exp(k+T)}; For each time step i, i=0,...,T in the obstacle position and velocity sequence, perform: Get the position p_obj(k+i) and velocity v_obj(k+i) of the obstacle at time k+i; Construct a three-dimensional bounding box of the obstacle at time k+i, with p_obj(k+i) as the center and length, width and height of the box being (Zl+vx, Zw+vy, Zh+vz), where (vx, vy, vz) are the components of v_obj(t+i) in the x, y and z dimensions, and the x, y and z axes are the three coordinate axes of the spatial rectangular coordinate system; Determine whether the position p_exp(k+i) of the detector at time k+i is within the three-dimensional bounding box; if so, determine that a collision will occur and mark the time step k+i as k0; otherwise, continue to determine the next time step; Step 44: If no collision is detected in all time steps k, it is determined that no collision will occur; The planning of a new moving path based on the body data and the moving data as well as the target location and the current location of the detector comprises the following steps: Step 51: construct a detector position optimization variable set S = {p_0, p_1, ..., p_m}, p_j = (x_j, y_j, z_j), j = 1, 2, 3, ..., m; m is the preset maximum time to reach the target location; Step 52: construct an optimization objective function; Step 53: Construct a set of constraints: Step 54: Taking the optimization objective function as the optimization objective and the constraint condition set as the constraint, it is formalized into a constrained optimization problem. The constrained optimization problem is solved by using the optimization algorithm to obtain the solution set of the optimization variable set S. According to the position of the detector per unit time in the solution set, a new moving path is constructed.

2. The multi-modal fusion underwater environment intelligent perception method according to claim 1 is characterized in that: The method of detecting whether there is an obstacle in front of the detector by using a sonar device comprises the following steps: Step 11: The sonar device carried by the detector transmits a sound wave pulse signal s(t) forward and receives the returned reflected echo signal r(t); Step 12: Define a delay variable T, which represents the time delay between the transmitted signal and the received signal; Step 13: Calculate the cross-correlation function between the transmitted acoustic pulse signal and the reflected echo signal , The time delay at which the peak occurs is marked as τ_p; Step 14: Calculate the distance d to the nearest obstacle based on the speed of sound c; Step 15: If d is greater than or equal to the preset distance threshold, it is determined that there is no obstacle; otherwise, it is determined that there is an obstacle.

3. The multi-modal fusion underwater environment intelligent perception method according to claim 2 is characterized in that: The method of judging whether the obstacle needs to be bypassed based on the time when the collision is judged to occur is: If it is determined that the time step k0 at which a collision will occur is less than the preset avoidance time threshold; otherwise, it is determined that no detour is required.

4. A multi-modal fusion underwater environment intelligent perception system, which is used to implement the multi-modal fusion underwater environment intelligent perception method according to any one of claims 1 to 3, characterized in that: It includes an obstacle detection module, an obstacle data collection module, a collision detection module and a detour module; wherein all modules are electrically connected; The obstacle detection module detects whether there is an obstacle in front of the detector through the sonar device. If there is, the obstacle data collection module is started; The obstacle data collection module collects the depth image data in front, obtains the main body data based on the depth image data, trains the movement prediction model of the obstacle based on the depth image data, obtains the movement data of the obstacle based on the movement prediction model, and sends the main body data and the movement data to the collision detection module; The obtaining of the body data based on the depth image data comprises the following steps: Step 21: performing image preprocessing on the depth image data; Step 22: Use a depth image segmentation algorithm to segment obstacles in the depth image data, and reproject the depth values ​​of the obstacle depth image obtained by segmentation into a three-dimensional space to obtain a three-dimensional point cloud of the obstacle; Step 23: Use the trained 3D shape classification model to classify the point cloud and output the obstacle category and size estimation as the ontology data; The step of training a movement prediction model of an obstacle based on the depth image data and obtaining movement data of the obstacle based on the movement prediction model comprises the following steps: Step 31: Using a visual tracking system, record the actual movement trajectory of the obstacle within a preset monitoring time; Step 32: During the monitoring period, use the depth camera to synchronously collect depth image data of obstacles at different times to form time series depth data; Step 33: Associating the actual motion trajectory of the obstacle with the corresponding time series depth data as a training data set; Step 34: Using a 3D convolutional neural network as a feature extractor, inputting the temporal depth data into the 3D convolutional neural network, and the 3D convolutional neural network finally outputs a sequence of spatiotemporal feature maps; Step 35: Use the convolutional LSTM or the attention-based sequence prediction model as the trajectory prediction model, and use the spatiotemporal feature map sequence as the input of the trajectory prediction model. At each time step, the trajectory prediction model outputs the trajectory state of the next moment according to the historical spatiotemporal feature map sequence. The trajectory state includes position and speed. After accumulating the predictions of multiple time steps, the motion trajectory of the next T time steps is output; T is the preset prediction duration. Step 36: Take the subsequent speed and position of the obstacle within the monitoring time as the prediction target, take the Euclidean distance between the trajectory state and the prediction target as the prediction error, and use the average value of the prediction error at each time step as the loss function to train the trajectory prediction model and obtain a trained mobile prediction model; Step 37: Based on the latest actual motion trajectory of the obstacle, the sequence prediction model outputs the trajectory state at the next T time as the input of the 3D convolutional neural network to form the movement data of the obstacle; The collision detection module determines whether a collision will occur based on the obstacle's body data and movement data and the detector's current movement path. If a collision is determined to occur, the time when the collision is determined to occur and the obstacle's body data and movement data are sent to the detour module. The step of determining whether a collision will occur based on the obstacle's body data and movement data and the detector's current movement path comprises the following steps: Step 41: mark the length, width and height of the obstacle as Zl, Zw and Zh respectively; mark the position sequence and velocity sequence of the obstacle in the future T time steps as {p_obj(t), p_obj(t+1), ..., p_obj(t+T)} and {v_obj(t), v_obj(t+1), ..., v_obj(t+T)} respectively; Step 42: mark the radius size of the detector as r, the current position as p_c, the direction as dir, the position sequence corresponding to the moving path as {p_0, p_1, ..., p_n} and the velocity sequence as {v_0, v_1, ..., v_n}; n is the time required to reach the target location from the current position according to the moving path; Step 43: For each time step k, k=0, 1, ..., n: Based on the position p_c(k) and velocity v(k) of the detector at the kth time step, calculate its motion trajectory in the next T time steps: {p_exp(k), p_exp(k+1), ..., p_exp(k+T)}; For each time step i, i=0,...,T in the obstacle position and velocity sequence, perform: Get the position p_obj(k+i) and velocity v_obj(k+i) of the obstacle at time k+i; Construct a three-dimensional bounding box of the obstacle at time k+i, with p_obj(k+i) as the center and length, width and height of the box being (Zl+vx, Zw+vy, Zh+vz), where (vx, vy, vz) are the components of v_obj(t+i) in the x, y and z dimensions, and the x, y and z axes are the three coordinate axes of the spatial rectangular coordinate system; Determine whether the position p_exp(k+i) of the detector at time k+i is within the three-dimensional bounding box; if so, determine that a collision will occur and mark the time step k+i as k0; otherwise, continue to determine the next time step; Step 44: If no collision is detected in all time steps k, it is determined that no collision will occur; The bypass module determines whether the obstacle needs to be bypassed based on the time when the collision is determined to have occurred. If bypassing is required, a new movement path is planned based on the body data and movement data as well as the target location and current position of the detector; The planning of a new moving path based on the body data and the moving data as well as the target location and the current location of the detector comprises the following steps: Step 51: construct a detector position optimization variable set S = {p_0, p_1, ..., p_m}, p_j = (x_j, y_j, z_j), j = 1, 2, 3, ..., m; m is the preset maximum time to reach the target location; Step 52: construct an optimization objective function; Step 53: Construct a set of constraints: Step 54: Taking the optimization objective function as the optimization objective and the constraint condition set as the constraint, it is formalized into a constrained optimization problem. The constrained optimization problem is solved by using the optimization algorithm to obtain the solution set of the optimization variable set S. According to the position of the detector per unit time in the solution set, a new moving path is constructed.

5. An electronic device, characterized in that: include: A processor and a memory, wherein the memory stores a computer program that can be called by the processor; The processor executes the multimodal fusion underwater environment intelligent perception method according to any one of claims 1 to 3 by calling the computer program stored in the memory.

6. A computer-readable storage medium, characterized in that: Instructions are stored, and when the instructions are executed on a computer, the computer executes the multi-modal fusion underwater environment intelligent perception method as described in any one of claims 1 to 3.

Images