Method, device and computer equipment for generating water surface map of unmanned ship

Abstract

The present invention relates to a method, device and computer equipment for generating a water surface map of an unmanned ship. The method comprises: acquiring attitude sensor data, image data and laser radar data, and constructing water surface map prediction training data; constructing multiple backbone feature extraction networks to extract features from image data; establishing a water surface perspective conversion network, inputting image features and attitude sensor data into the water surface perspective conversion network; inputting bird's-eye view image features into a head prediction network, calculating a loss function to complete the iteration of a water surface map generation model; inputting image data and attitude sensor data to generate a water surface map in real time. The present invention can directly generate omnidirectional high-precision map data by collecting camera data and attitude sensor data of the unmanned ship and inputting them into a constructed vision-based water surface high-precision map generation network, thereby improving the prediction robustness and map prediction accuracy of the unmanned ship when it moves on the water surface.

Description

Technical Field

[0001] The present invention relates to the technical field of water surface map generation, and in particular to a method, device and computer equipment for generating a water surface map of an unmanned ship. Background Art

[0002] In recent years, unmanned ships have been widely used in marine research. At present, unmanned ships are also closely watched as ideal surface vehicles and surface autonomous operation equipment in inland river scenarios. When unmanned ships perform surface autonomous driving tasks in inland river scenarios, they often need to face complex water scenes, and the narrow river channels cause deviations in the positioning system of unmanned ships. Therefore, building high-precision maps on the water surface is of great significance for the safe navigation of unmanned ships, helping unmanned ships to plan paths in complex scenarios without GPS.

[0003] The traditional high-precision map construction method uses LiDAR for point cloud collection and mapping, but LiDAR is expensive and cannot work effectively in rainy and foggy days, which is not conducive to the widespread deployment of unmanned ships. With the development of deep learning technology, the performance of visual neural networks has been promoted. At present, pure visual autonomous driving perception solutions have attracted widespread attention because of their low price and strong adaptability. At present, road autonomous driving cars can achieve 360-degree visual perception by installing surround cameras, and transform image information from camera space to bird's-eye view space through neural network mapping. However, in water scenes, unmanned ships will continue to pitch with the fluctuation of water flow, and the observation angle of the camera will also change a lot. The use of visual neural networks to estimate the distance of the surrounding environment is easily affected by angle changes and produce large deviations, affecting the performance of the visual network. In addition, there are many angles of the surround-view cameras, such as telephoto cameras, short focal length cameras and fisheye cameras, and the distortion degree of different types of cameras is different. If the visual processing method of a single camera is used for fusion, the total computational load of the system will be too large, and in the multi-camera fusion network, different degrees of distortion have great difficulties in the fusion input of multiple cameras. Summary of the invention

[0004] The purpose of the present invention is to overcome the deficiencies of the prior art and provide a method, device and computer equipment for generating a water surface map of an unmanned ship.

[0005] In order to solve the above technical problems, the present invention adopts the following technical solutions:

[0006] In a first aspect, this embodiment provides a method for generating a water surface map of an unmanned ship, comprising the following steps:

[0007] Obtain the attitude sensor data of the unmanned boat, image data collected by multiple cameras, and lidar data, and construct water surface map prediction training data;

[0008] Construct multiple backbone feature extraction networks according to the number of cameras, extract features from image data to obtain image features;

[0009] Establish a water surface perspective conversion network for converting the camera space into a bird's-eye view space, input image features and attitude sensor data into the water surface perspective conversion network to obtain bird's-eye view image features;

[0010] Construct a map head prediction network, input the bird's-eye view image features into the head prediction network to generate a water surface map prediction result, and calculate the loss function based on the water surface map prediction result and the true value of the training data to complete the iteration of the water surface map generation model;

[0011] Obtain the water surface map generation model trained by the unmanned boat, input the image data collected by multiple cameras and attitude sensor data, and generate the water surface map in real time.

[0012] The further technical solution is: the acquisition of the attitude sensor data of the unmanned boat, the image data collected by multiple cameras and the laser radar data, and the construction of the water surface map prediction training data, includes the following steps:

[0013] Get the camera intrinsic parameter matrix;

[0014] A coordinate system is established with the position of the unmanned ship when it is powered on as the origin and the front and right directions of the unmanned ship as the positive directions;

[0015] The attitude sensor data, image data and lidar data of the unmanned ship are acquired at a set sampling frequency, and the water surface map prediction training data is constructed.

[0016] A further technical solution is: constructing multiple backbone feature extraction networks according to the number of cameras, extracting features from image data to obtain image features, including the following steps:

[0017] The image data includes image data of n cameras, and multiple backbone feature extraction networks are constructed to extract image features of the cameras to obtain extracted image features;

[0018] Adjust the extracted image features to obtain new image features;

[0019] The new image features are normalized and concatenated to obtain the image features.

[0020] A further technical solution is as follows: establishing a water surface perspective conversion network for converting the camera space into a bird's-eye view space, inputting image features and gesture sensor data into the water surface perspective conversion network to obtain bird's-eye view image features, comprises the following steps:

[0021] Construct a query matrix to save the water surface perspective conversion parameters from the camera space to the bird's-eye view space;

[0022] Get the transformation matrix according to the camera intrinsic parameter matrix;

[0023] Perform feature encoding according to the attitude sensor data to obtain the attitude sensor data features;

[0024] Obtain keyword matrix and parameter matrix according to image features;

[0025] According to the transformation matrix, the keyword matrix and the parameter matrix, the view image features are obtained;

[0026] The features of the attitude sensor data and the view image are fused to obtain the bird's-eye view image features.

[0027] The further technical solution is as follows: the map head prediction network is constructed, the bird's-eye view image features are input into the head prediction network to generate a water surface map prediction result, and the loss function is calculated according to the water surface map prediction result and the true value of the training data to complete the iteration of the water surface map generation model, including the following steps:

[0028] Construct a map head prediction network and input the bird's-eye view image features into the head prediction network to obtain prediction features;

[0029] Threshold filtering is performed on the predicted features to generate a water surface map prediction result;

[0030] The loss function is calculated by combining the water surface map prediction result with the true value of the training data to obtain a water surface map generation model;

[0031] The water surface map generation model performs back propagation and completes model training after continuous iteration.

[0032] In a second aspect, the present embodiment provides a water surface map generating device for an unmanned ship, comprising: an acquisition and construction unit, a construction and extraction unit, an establishment input unit, a construction input calculation unit, and an acquisition and input generation unit;

[0033] The acquisition and construction unit is used to acquire the attitude sensor data of the unmanned ship, the image data collected by multiple cameras and the laser radar data, and construct the water surface map prediction training data;

[0034] The construction and extraction unit is used to construct multiple backbone feature extraction networks according to the number of cameras, and perform feature extraction on the image data to obtain image features;

[0035] The input unit is used to establish a water surface perspective conversion network for converting the camera space into a surround bird's-eye view space, and input image features and gesture sensor data into the water surface perspective conversion network to obtain bird's-eye view image features;

[0036] The input calculation unit is used to construct a map head prediction network, input the bird's-eye view image features into the head prediction network to generate a water surface map prediction result, and calculate the loss function according to the water surface map prediction result and the true value of the training data to complete the iteration of the water surface map generation model;

[0037] The acquisition input generation unit is used to obtain the water surface map generation model trained by the unmanned ship, input image data collected by multiple cameras and attitude sensor data, and generate a water surface map in real time.

[0038] Its further technical solution is: the acquisition construction unit includes: an acquisition module, a building module and an acquisition construction module;

[0039] The acquisition module is used to obtain the camera intrinsic parameter matrix;

[0040] The establishment module is used to establish a coordinate system with the position of the unmanned ship at the time of power-on as the origin and the front direction and right direction of the unmanned ship as the positive direction;

[0041] The acquisition and construction module is used to acquire the attitude sensor data, image data and lidar data of the unmanned ship at a set sampling frequency, and to construct water surface map prediction training data.

[0042] Its further technical solution is: the construction and extraction unit includes: a construction and extraction module, an adjustment module and a normalization and splicing module;

[0043] The extraction module is used to construct multiple backbone feature extraction networks to extract image features of the cameras from image data containing n cameras, so as to obtain extracted image features;

[0044] The adjustment module is used to adjust the extracted image features to obtain new image features;

[0045] The normalization and splicing module is used to normalize and splice new image features to obtain image features.

[0046] In a third aspect, this embodiment provides a computer device, which includes a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the computer program, the method for generating a water surface map of an unmanned ship as described above is implemented.

[0047] In a fourth aspect, this embodiment provides a storage medium, wherein the storage medium stores a computer program, wherein the computer program includes program instructions, and when the program instructions are executed by a processor, the method for generating a water surface map of an unmanned ship as described above can be implemented.

[0048] The beneficial effects of the present invention compared with the prior art are as follows: by collecting the camera data and attitude sensor data of the unmanned boat, and then inputting them into the constructed vision-based water surface high-precision map generation network, omnidirectional high-precision map data can be directly generated. The network based on surround vision input can reduce computing resources and has a higher price advantage compared to the lidar mapping solution. In addition, in order to achieve better prediction effects on the water surface, the water surface perspective conversion network using attitude sensor data can effectively improve the prediction robustness of the unmanned boat when moving on the water surface, and improve the prediction accuracy of the high-precision map.

[0049] The present invention is further described below in conjunction with the accompanying drawings and specific embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0051] Figure 1 A schematic diagram of a flow chart of a method for generating a water surface map of an unmanned ship provided in an embodiment of the present invention;

[0052] Figure 2 Schematic diagram of a sub-process of a method for generating a water surface map of an unmanned ship provided in an embodiment of the present invention Figure 1 ;

[0053] Figure 3 Schematic diagram of a sub-process of a method for generating a water surface map of an unmanned ship provided in an embodiment of the present invention Figure 2 ;

[0054] Figure 4 A schematic diagram of an application scenario for extracting image features provided by an embodiment of the present invention;

[0055] Figure 5 Schematic diagram of a sub-process of a method for generating a water surface map of an unmanned ship provided in an embodiment of the present invention Figure 3 ;

[0056] Figure 6 A schematic diagram of a water surface perspective conversion network structure provided by an embodiment of the present invention;

[0057] Figure 7 A schematic diagram of a cross-attention module structure provided by an embodiment of the present invention;

[0058] Figure 8 Schematic diagram of a sub-process of a method for generating a water surface map of an unmanned ship provided in an embodiment of the present invention Figure 4 ;

[0059] Fig. 9 A schematic diagram of a head prediction network structure provided by an embodiment of the present invention;

[0060] Fig.10 A schematic block diagram of a water surface map generating device for an unmanned ship provided in an embodiment of the present invention;

[0061] Fig.11 A schematic diagram of a water surface map generating device for an unmanned ship provided in an embodiment of the present invention Figure 1 ;

[0062] Fig.12 A schematic diagram of a water surface map generating device for an unmanned ship provided in an embodiment of the present invention Figure 2 ;

[0063] Fig.13 A schematic diagram of a water surface map generating device for an unmanned ship provided in an embodiment of the present invention Figure 3 ;

[0064] Fig.14 A schematic diagram of a water surface map generating device for an unmanned ship provided in an embodiment of the present invention Figure 4 ;

[0065] Fig.15 A schematic block diagram of a computer device provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0066] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0067] It should be understood that when used in this specification and the appended claims, the terms "include" and "comprises" indicate the presence of described features, integers, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or combinations thereof.

[0068] It should also be understood that the terms used in this specification of the present invention are only for the purpose of describing specific embodiments and are not intended to limit the present invention. As used in the specification of the present invention and the appended claims, unless the context clearly indicates otherwise, the singular forms "a", "an" and "the" are intended to include plural forms.

[0069] It should be further understood that the term "and / or" used in the present description and the appended claims refers to any and all possible combinations of one or more of the associated listed items, and includes these combinations.

[0070] See also Figure 1 In the specific embodiment shown, the present invention discloses a method for generating a water surface map of an unmanned ship, comprising the following steps:

[0071] S1, obtains the attitude sensor data of the unmanned ship, the image data collected by multiple cameras and the lidar data, and constructs the water surface map prediction training data;

[0072] Specifically, the sensors carried by the unmanned ship include multiple cameras installed for surround view, a multi-line laser radar, an inertial measurement unit (IMU), etc.

[0073] In one embodiment, see Figure 2 As shown, the acquisition of the attitude sensor data of the unmanned boat, the image data collected by multiple cameras and the laser radar data, and the construction of the water surface map prediction training data include the following steps:

[0074] S11, obtaining the camera internal parameter matrix;

[0075] Specifically, the camera intrinsic parameter matrix T is obtained through a calibration algorithm (the calibration algorithm adopts an existing conventional algorithm).

[0076] S12, establishing a coordinate system with the position of the unmanned ship at the time of power-on as the origin and the front direction and right direction of the unmanned ship as the positive direction;

[0077] Specifically, the position of the unmanned ship at the time of power-on is the origin, and the unmanned boat is facing in front of Axis positive direction, right direction The unmanned ship coordinate system is established in the positive direction of the axis.

[0078] S13, acquiring the attitude sensor data, image data and lidar data of the unmanned boat at a set sampling frequency, and constructing water surface map prediction training data.

[0079] Specifically, the sampling rate is set to 10 Hz, and the inertial measurement unit IMU (attitude sensor) data P, image data I, and lidar data L of the unmanned ship are acquired at a sampling frequency of 10 Hz.

[0080] In one embodiment, after step S13, the process further includes: performing point cloud filtering FLITER on the laser radar data L to filter out adjacent noise points and discrete noise points to obtain a filtered point cloud. ;

[0081] ;

[0082] At time t, the filtered point cloud Fill in the grid map with the unmanned ship as the origin to get the grid map at time t , the point cloud location information is saved in a 100×100 matrix. Each grid contains points within 0.5m, which means that the resolution of the grid map is 0.5m. The grid map saves the obstacle location information of 50m×50m near the unmanned ship.

[0083] Save the accumulated raster map of multiple frames to get the raster map This solution uses 10 frames of accumulated LiDAR point cloud results, which can be selected according to the performance of the LiDAR:

[0084] ;

[0085] Use labeling tools to label raster maps Label the obstacles as semantic targets such as shore, ship, fountain, etc. to obtain the real results of high-precision map. ,All images, attitude sensor data, and real results of high precision maps constitute the ,water surface map prediction training data.

[0086] S2, constructing multiple backbone feature extraction networks according to the number of cameras, extracting features from the image data to obtain image features;

[0087] In one embodiment, see Figure 3 to Figure 4 As shown, the method of constructing multiple backbone feature extraction networks according to the number of cameras and extracting features from image data to obtain image features includes the following steps:

[0088] S21, the image data includes image data of n cameras, and multiple backbone feature extraction networks are constructed to extract image features of the cameras to obtain extracted image features;

[0089] See also Figure 4 As shown, the image data I includes image data of n cameras. Considering the different distortion conditions of different types of cameras, the long focal length camera encoder Long_Encoder, fisheye camera encoder Fish_Encoder and short focal length camera encoder Short_Encoder are designed in the backbone feature extraction network to extract image features of different types of cameras. :

[0090] ;

[0091] ;

[0092] .

[0093] S22, adjusting the extracted image features to obtain new image features;

[0094] Specifically, considering that the surround camera field of view has an intersection, the present invention uses a global attention module Global to adjust the image features of each part of the camera to obtain a new image feature :

[0095] .

[0096] S23, normalize and concatenate the new image features to obtain image features.

[0097] Specifically, the new image feature extraction network is normalized and then concatenated to obtain the image feature F:

[0098] .

[0099] S3, establishing a water surface perspective conversion network for converting the camera space into a bird's-eye view space, inputting image features and attitude sensor data into the water surface perspective conversion network to obtain bird's-eye view image features;

[0100] In one embodiment, see Figures 5 to 7 As shown, the establishment of a water surface perspective conversion network for converting the camera space into a bird's-eye view space, inputting image features and gesture sensor data into the water surface perspective conversion network to obtain bird's-eye view image features, includes the following steps:

[0101] S31, constructing a query matrix for storing water surface perspective conversion parameters from camera space to surround bird's-eye view space;

[0102] Specifically, see Figure 6 The network structure of water surface perspective conversion shown in the figure first constructs a query matrix Q. The matrix Q is a high-dimensional matrix of 50*50*128. At the initial moment, all the parameters of the matrix are 0. It is used to save the water surface perspective conversion parameters from the camera space to the panoramic bird's-eye view space. Subsequently, with the training of the neural network, the parameters of the matrix Q are continuously optimized and iterated.

[0103] S32, obtaining a transformation matrix according to the camera intrinsic parameter matrix;

[0104] Specifically, the BEV conversion matrix is ​​obtained according to the camera internal parameter matrix T :

[0105] .

[0106] S33, performing feature encoding according to the attitude sensor data to obtain the attitude sensor data features;

[0107] Specifically, feature encoding is performed on the attitude sensor data P to obtain the attitude sensor data feature :

[0108] .

[0109] S34, obtaining a keyword matrix and a parameter matrix according to the image features;

[0110] Specifically, input the image feature F into the multi-layer perceptron MLP to obtain the keyword matrix and the parameter matrix :

[0111] .

[0112] S35, obtaining a view image feature according to the transformation matrix, the keyword matrix and the parameter matrix;

[0113] Specifically, the transformation matrix , keyword matrix and the parameter matrix Input to the CrossAttention conversion module to obtain the image features converted to the BEV perspective :

[0114] .

[0115] Further, see Figure 7 The calculation process of the cross-attention module CrossAttention is as follows: First, , , Perform transformation operations to obtain image features :

[0116] ;

[0117] in, is the activation function, is the dimension normalization factor, according to Next, the image features Input to Mapping layer and The mapping layer obtains the image features converted to the BEV perspective :

[0118] .

[0119] S36, fusing the gesture sensor data features and the view image features to obtain the bird's-eye view image features.

[0120] Specifically, considering that the conversion of the bird's-eye view of the water surface causes large conversion errors due to the pitch and roll of the camera, by fusing the IMU data features and image features Get the converted bird's-eye view image features :

[0121] .

[0122] S4, constructing a map head prediction network, inputting the bird's-eye view image features into the head prediction network to generate a water surface map prediction result, and calculating the loss function based on the water surface map prediction result and the true value of the training data to complete the iteration of the water surface map generation model;

[0123] In one embodiment, see Figures 8 to 9 As shown, the map head prediction network is constructed, the bird's-eye view image features are input into the head prediction network to generate a water surface map prediction result, and the loss function is calculated according to the water surface map prediction result and the true value of the training data to complete the water surface map generation model iteration, including the following steps:

[0124] S41, constructing a map head prediction network, inputting the bird's-eye view image features into the head prediction network to obtain prediction features;

[0125] Specifically, see Fig. 9 The head prediction network structure shown in the figure contains 4 layers of convolutional neural network, and the convolution kernel sizes are 3×3, 1×1, 3×3, and 1×1 respectively. Input into the head prediction network to get the prediction features , prediction features The size of is consistent with the size of the grid map constructed above, both are 100×100:

[0126] .

[0127] S42, performing threshold filtering on the predicted features to generate a water surface map prediction result;

[0128] Specifically, the predicted features Threshold filtering is performed through the Softmax activation function to generate water surface map prediction results :

[0129] ;

[0130] S43, calculating the loss function of the water surface map prediction result and the true value of the training data to obtain a water surface map generation model;

[0131] Specifically, the water surface map prediction results Real results with high-precision maps The loss function is used to calculate the loss value, and the neural network model is back-propagated according to the loss value to optimize the weights of the water surface high-precision map prediction model. When the model loss value reaches the iteration threshold, the iteration is stopped to obtain a trained water surface high-precision map prediction model.

[0132] Specifically, the water surface map prediction results Predicted target location and real results of high-precision map Calculate the position error loss value based on the mean square error function MSE :

[0133] .

[0134] The water surface map prediction results Confidence of predicted targets and real results of high-precision maps Calculate the confidence loss value according to the cross entropy function CrossEntropy :

[0135] .

[0136] The water surface map prediction results Prediction target classification and HD map real results The actual target classification is calculated according to the cross entropy function CrossEntropy classification loss value :

[0137] .

[0138] The position error loss value , confidence loss value And classification loss value Add together to get the total loss of the neural network model ,Will Perform model back propagation and update the weights of the water surface high-precision map prediction model. The calculation formula is:

[0139] ;

[0140] in, is a hyperparameter.

[0141] S44, the water surface map generation model performs back propagation, and the model training is completed after continuous iteration.

[0142] Specifically, the new water surface high-precision map prediction model weights are used for the next model training iteration. When it is less than the iteration threshold (the iteration threshold is a set hyperparameter, such as a fixed value of 0.1), the training is terminated and a trained water surface high-precision map prediction model is obtained.

[0143] S5, obtain the water surface map generation model trained by the unmanned ship, input the image data collected by multiple cameras and the attitude sensor data, and generate the water surface map in real time.

[0144] Specifically, the unmanned ship obtains the trained surface high-precision map prediction model from the cloud. After loading the model, it obtains image data and attitude sensor data in real time. By executing the above steps S1-S4, the real-time high-precision map prediction results can be output. .

[0145] Specifically, the unmanned ship obtains the trained high-precision water surface map prediction model from the deep learning cloud server and loads it to the unmanned ship. The unmanned ship obtains the raw data of the image data and the raw data of the attitude sensor data in real time, and then performs data preprocessing and analysis to obtain the attitude sensor data P and the image data I collected by multiple cameras. Furthermore, the preprocessed data is subjected to model inference to obtain the multi-sensor real-time fusion detection result, and the real-time high-precision map prediction result is output by executing the above steps S1-S4. .

[0146] The present invention can directly generate omnidirectional high-precision map data by collecting camera data and attitude sensor data of the unmanned ship, and then inputting them into the constructed vision-based water surface high-precision map generation network. The network based on surround vision input can reduce computing resources and has a higher price advantage compared to the lidar mapping solution. In addition, in order to achieve better prediction effects on the water surface, the water surface perspective conversion network using attitude sensor data can effectively improve the prediction robustness of the unmanned ship when moving on the water surface, and improve the prediction accuracy of the high-precision map.

[0147] See also Fig.10 As shown, the present invention also discloses a water surface map generating device for an unmanned ship, comprising: an acquisition and construction unit 10, a construction and extraction unit 20, an establishment input unit 30, a construction input calculation unit 40 and an acquisition and input generation unit 50;

[0148] The acquisition and construction unit 10 is used to acquire the attitude sensor data of the unmanned ship, the image data collected by multiple cameras and the laser radar data, and construct the water surface map prediction training data;

[0149] The construction and extraction unit 20 is used to construct multiple backbone feature extraction networks according to the number of cameras, and perform feature extraction on the image data to obtain image features;

[0150] The input unit 30 is used to establish a water surface perspective conversion network for converting the camera space into a surround bird's-eye view space, and input image features and gesture sensor data into the water surface perspective conversion network to obtain bird's-eye view image features;

[0151] The construction input calculation unit 40 is used to construct a map head prediction network, input the bird's-eye view image features into the head prediction network to generate a water surface map prediction result, and calculate the loss function according to the water surface map prediction result and the true value of the training data to complete the iteration of the water surface map generation model;

[0152] The acquisition input generation unit 50 is used to obtain the water surface map generation model trained by the unmanned ship, input the image data collected by multiple cameras and the attitude sensor data, and generate the water surface map in real time.

[0153] In one embodiment, see Fig.11 As shown, the acquisition construction unit 10 includes: an acquisition module 11, a building module 12 and an acquisition construction module 13;

[0154] The acquisition module 11 is used to obtain the camera intrinsic parameter matrix;

[0155] The establishment module 12 is used to establish a coordinate system with the position of the unmanned ship at the time of power-on as the origin and the front direction and right direction of the unmanned ship as the positive direction;

[0156] The acquisition and construction module 13 is used to acquire the attitude sensor data, image data and lidar data of the unmanned ship at a set sampling frequency, and construct water surface map prediction training data.

[0157] In one embodiment, see Fig.12 As shown, the construction and extraction unit 20 includes: a construction and extraction module 21, an adjustment module 22 and a normalization and splicing module 23;

[0158] The extraction module 21 is used to construct multiple backbone feature extraction networks to extract image features of the cameras from the image data including the image data of n cameras, so as to obtain extracted image features;

[0159] The adjustment module 22 is used to adjust the extracted image features to obtain new image features;

[0160] The normalization and splicing module 23 is used to normalize and splice new image features to obtain image features.

[0161] In one embodiment, see Fig.13 As shown, the establishing input unit 30 includes: a constructing and saving module 31, a first obtaining module 32, a second obtaining module 33, a third obtaining module 34, a fourth obtaining module 35 and a fusion module 36;

[0162] The construction and saving module 31 is used to construct a query matrix for saving the water surface perspective conversion parameters from the camera space to the surround bird's-eye view space;

[0163] The first obtaining module 32 is used to obtain a conversion matrix according to the camera intrinsic parameter matrix;

[0164] The second obtaining module 33 is used to perform feature encoding according to the posture sensor data to obtain the posture sensor data feature;

[0165] The third obtaining module 34 is used to obtain a keyword matrix and a parameter matrix according to image features;

[0166] The fourth obtaining module 35 is used to obtain the view image feature according to the conversion matrix, the keyword matrix and the parameter matrix;

[0167] The fusion module 36 is used to fuse the gesture sensor data features and the view image features to obtain the bird's-eye view image features.

[0168] In one embodiment, see Fig.14 As shown, the construction input calculation unit 40 includes: a construction input module 41, a filter generation module 42, a calculation module 43 and a propagation iteration module 44;

[0169] The construction input module 41 is used to construct a map head prediction network, and input the bird's-eye view image features into the head prediction network to obtain prediction features;

[0170] The filtering generation module 42 is used to perform threshold filtering on the prediction features to generate a water surface map prediction result;

[0171] The calculation module 43 is used to calculate the loss function of the water surface map prediction result and the true value of the training data to obtain a water surface map generation model;

[0172] The propagation iteration module 44 is used to calculate the loss function of the water surface map prediction result and the true value of the training data to obtain a water surface map generation model;

[0173] It should be noted that technicians in the relevant field can clearly understand that the specific implementation process of the water surface map generating device and each unit of the above-mentioned unmanned ship can refer to the corresponding description in the aforementioned method embodiment, and for the convenience and conciseness of the description, it will not be repeated here.

[0174] The water surface map generating device of the unmanned ship can be implemented in the form of a computer program. Fig.15 Runs on the computer device shown.

[0175] See also Fig.15 , Fig.15 is a schematic block diagram of a computer device provided in an embodiment of the present application; the computer device 500 may be a terminal or a server, wherein the terminal may be an electronic device with communication functions such as a smart phone, a tablet computer, a laptop computer, a desktop computer, a personal digital assistant, and a wearable device. The server may be an independent server or a server cluster composed of multiple servers.

[0176] See also Fig.15 The computer device 500 includes a processor 502 , a memory and a network interface 505 connected via a system bus 501 , wherein the memory may include a non-volatile storage medium 503 and an internal memory 504 .

[0177] The non-volatile storage medium 503 can store an operating system 5031 and a computer program 5032. The computer program 5032 includes program instructions, and when the program instructions are executed, the processor 502 can execute a method for generating a water surface map of an unmanned ship.

[0178] The processor 502 is used to provide computing and control capabilities to support the operation of the entire computer device 500 .

[0179] The internal memory 504 provides an environment for the operation of the computer program 5032 in the non-volatile storage medium 503. When the computer program 5032 is executed by the processor 502, the processor 502 can execute a method for generating a water surface map of an unmanned ship.

[0180] The network interface 505 is used to communicate with other devices over the network. Fig.15 The structure shown in the figure is merely a block diagram of a portion of the structure related to the solution of the present application, and does not constitute a limitation on the computer device 500 to which the solution of the present application is applied. The specific computer device 500 may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0181] The processor 502 is used to run the computer program 5032 stored in the memory to implement the following steps:

[0182] The attitude sensor data of the unmanned boat, the image data collected by multiple cameras and the lidar data are obtained, and the water surface map prediction training data is constructed; multiple backbone feature extraction networks are constructed according to the number of cameras to extract features from the image data to obtain image features; a water surface perspective conversion network is established to convert the camera space into a bird's-eye view space, and the image features and attitude sensor data are input into the water surface perspective conversion network to obtain the bird's-eye view image features; a map head prediction network is constructed, and the bird's-eye view image features are input into the head prediction network to generate a water surface map prediction result, and the loss function is calculated according to the water surface map prediction result and the true value of the training data to complete the iteration of the water surface map generation model; the water surface map generation model trained by the unmanned boat is obtained, and the image data collected by multiple cameras and attitude sensor data are input to generate a water surface map in real time.

[0183] It should be understood that in the embodiment of the present application, the processor 502 may be a central processing unit (CPU), and the processor 502 may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Among them, the general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0184] It can be understood by those skilled in the art that all or part of the processes in the method for implementing the above embodiment can be completed by instructing the relevant hardware through a computer program. The computer program includes program instructions, and the computer program can be stored in a storage medium, which is a computer-readable storage medium. The program instructions are executed by at least one processor in the computer system to implement the process steps of the embodiment of the above method.

[0185] Therefore, the present invention also provides a storage medium. The storage medium may be a computer-readable storage medium. The storage medium stores a computer program, wherein the computer program includes program instructions, and when the program instructions are executed by a processor, the above-mentioned method for generating a water surface map of an unmanned ship can be implemented. The storage medium stores a computer program, wherein the computer program includes program instructions, and when the program instructions are executed by a processor, the above-mentioned method can be implemented. The program instructions include the following steps:

[0186] The attitude sensor data of the unmanned boat, the image data collected by multiple cameras and the lidar data are obtained, and the water surface map prediction training data is constructed; multiple backbone feature extraction networks are constructed according to the number of cameras to extract features from the image data to obtain image features; a water surface perspective conversion network is established to convert the camera space into a bird's-eye view space, and the image features and attitude sensor data are input into the water surface perspective conversion network to obtain the bird's-eye view image features; a map head prediction network is constructed, and the bird's-eye view image features are input into the head prediction network to generate a water surface map prediction result, and the loss function is calculated according to the water surface map prediction result and the true value of the training data to complete the iteration of the water surface map generation model; the water surface map generation model trained by the unmanned boat is obtained, and the image data collected by multiple cameras and attitude sensor data are input to generate a water surface map in real time.

[0187] The storage medium may be a USB flash drive, a mobile hard disk, a read-only memory (ROM), a magnetic disk, or an optical disk, etc., which are computer-readable storage media that can store program codes.

[0188] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in terms of function in the above description. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0189] In the several embodiments provided by the present invention, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of each unit is only a logical function division, and there may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed.

[0190] The steps in the method of the embodiment of the present invention can be adjusted in order, combined and deleted according to actual needs. The units in the device of the embodiment of the present invention can be combined, divided and deleted according to actual needs. In addition, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0191] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a storage medium. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, terminal, or network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention.

[0192] The above embodiments are preferred implementation schemes of the present invention. In addition, the present invention may also be implemented in other ways. Any obvious replacement without departing from the concept of the present technical solution is within the protection scope of the present invention.

Claims

1. A method for generating a water surface map of an unmanned ship, characterized in that: The following steps are involved: Obtain the attitude sensor data of the unmanned boat, image data collected by multiple cameras, and lidar data, and construct water surface map prediction training data; Construct multiple backbone feature extraction networks according to the number of cameras, extract features from image data to obtain image features; Establish a water surface perspective conversion network for converting the camera space into a bird's-eye view space, input image features and attitude sensor data into the water surface perspective conversion network to obtain bird's-eye view image features; Construct a map head prediction network, input the bird's-eye view image features into the head prediction network to generate a water surface map prediction result, and calculate the loss function based on the water surface map prediction result and the true value of the training data to complete the iteration of the water surface map generation model; Obtain the water surface map generation model trained by the unmanned ship, input the image data collected by multiple cameras and the attitude sensor data, and generate the water surface map in real time; The method of obtaining the attitude sensor data of the unmanned boat, the image data collected by multiple cameras and the laser radar data, and constructing the water surface map prediction training data includes the following steps: Get the camera intrinsic parameter matrix; A coordinate system is established with the position of the unmanned ship when it is powered on as the origin and the front and right directions of the unmanned ship as the positive directions; Acquire the attitude sensor data, image data and lidar data of the unmanned ship at a set sampling frequency, and construct water surface map prediction training data; The method of constructing multiple backbone feature extraction networks according to the number of cameras and extracting features from image data to obtain image features includes the following steps: The image data includes image data of n cameras, and multiple backbone feature extraction networks are constructed to extract image features of the cameras to obtain extracted image features; Adjust the extracted image features to obtain new image features; Normalize and concatenate the new image features to obtain image features; The water surface perspective conversion network for converting the camera space into the bird's-eye view space is established, and image features and gesture sensor data are input into the water surface perspective conversion network to obtain bird's-eye view image features, including the following steps: Construct a query matrix to save the water surface perspective conversion parameters from the camera space to the bird's-eye view space; Get the transformation matrix according to the camera intrinsic parameter matrix; Perform feature encoding according to the attitude sensor data to obtain the attitude sensor data features; Obtain keyword matrix and parameter matrix according to image features; According to the transformation matrix, the keyword matrix and the parameter matrix, the view image features are obtained; The features of the attitude sensor data and the view image are fused to obtain the bird's-eye view image features.

2. The method for generating a water surface map of an unmanned ship according to claim 1, characterized in that: The map head prediction network is constructed, and the bird's-eye view image features are input into the head prediction network to generate a water surface map prediction result. The loss function is calculated according to the water surface map prediction result and the true value of the training data to complete the water surface map generation model iteration, including the following steps: Construct a map head prediction network and input the bird's-eye view image features into the head prediction network to obtain prediction features; Threshold filtering is performed on the predicted features to generate a water surface map prediction result; The loss function is calculated by combining the water surface map prediction result with the true value of the training data to obtain a water surface map generation model; The water surface map generation model performs back propagation and completes model training after continuous iteration.

3. A water surface map generating device for an unmanned ship, characterized in that: include: Acquire a construction unit, construct an extraction unit, establish an input unit, construct an input calculation unit, and acquire an input generation unit; The acquisition and construction unit is used to acquire the attitude sensor data of the unmanned ship, the image data collected by multiple cameras and the laser radar data, and construct the water surface map prediction training data; The construction and extraction unit is used to construct multiple backbone feature extraction networks according to the number of cameras, and perform feature extraction on the image data to obtain image features; The input unit is used to establish a water surface perspective conversion network for converting the camera space into a surround bird's-eye view space, and input image features and gesture sensor data into the water surface perspective conversion network to obtain bird's-eye view image features; The input calculation unit is used to construct a map head prediction network, input the bird's-eye view image features into the head prediction network to generate a water surface map prediction result, and calculate the loss function according to the water surface map prediction result and the true value of the training data to complete the iteration of the water surface map generation model; The acquisition input generation unit is used to obtain the water surface map generation model trained by the unmanned ship, input the image data collected by multiple cameras and the attitude sensor data, and generate the water surface map in real time; The acquisition construction unit includes: an acquisition module, a creation module and an acquisition construction module; The acquisition module is used to obtain the camera intrinsic parameter matrix; The establishment module is used to establish a coordinate system with the position of the unmanned ship at the time of power-on as the origin and the front direction and right direction of the unmanned ship as the positive direction; The acquisition and construction module is used to acquire the attitude sensor data, image data and lidar data of the unmanned ship at a set sampling frequency, and construct water surface map prediction training data; The construction and extraction unit includes: a construction and extraction module, an adjustment module and a normalization and splicing module; The extraction module is used to construct multiple backbone feature extraction networks to extract image features of the cameras from image data containing n cameras, so as to obtain extracted image features; The adjustment module is used to adjust the extracted image features to obtain new image features; The normalization and splicing module is used to normalize and splice new image features to obtain image features; The input establishment unit includes: a construction and preservation module, a first obtaining module, a second obtaining module, a third obtaining module, a fourth obtaining module and a fusion module; The construction and saving module is used to construct a query matrix for saving water surface perspective conversion parameters from camera space to surround bird's-eye view space; The first obtaining module is used to obtain a conversion matrix according to the camera intrinsic parameter matrix; The second obtaining module is used to perform feature encoding according to the posture sensor data to obtain the posture sensor data features; The third obtaining module is used to obtain a keyword matrix and a parameter matrix according to image features; The fourth obtaining module is used to obtain the view image feature according to the conversion matrix, the keyword matrix and the parameter matrix; The fusion module is used to fuse the gesture sensor data features and the view image features to obtain the bird's-eye view image features.

4. A computer device, characterized in that: The computer device includes a memory and a processor, the memory stores a computer program, and the processor implements the water surface map generation method of the unmanned ship according to any one of claims 1-2 when executing the computer program.

5. A storage medium, characterized in that: The storage medium stores a computer program, which includes program instructions. When the program instructions are executed by a processor, the method for generating a water surface map of an unmanned ship according to any one of claims 1 to 2 can be implemented.

Images