Ship scene depth estimation method, equipment and medium

Abstract

The invention relates to a ship scene depth estimation method and device and a medium, and the method comprises the steps: obtaining multi-source and multi-mode data, including point cloud data and image data, in a ship docking scene; a laser radar encoder is adopted to extract point cloud geometric features from the point cloud data, and an image encoder is adopted to extract image texture features from the image data; performing domain uniformization fusion on the point cloud geometric features and the image texture features along the paths of sparsification, fusion and diffusion to obtain a dense geometric-texture feature matrix with consistency among elements; wherein the domain uniformization fusion comprises fusion of sparse domain data and dense domain data and fusion of geometric domain data and texture domain data; and according to the dense geometric-texture feature matrix, decoding through a decoder to obtain a depth estimation result of the ship docking scene. Compared with the prior art, the method has the advantage of high scene depth estimation accuracy.

Description

Technical Field

[0001] The present invention relates to the field of ship depth estimation, and in particular to a ship scene depth estimation method, device and medium. Background Art

[0002] Currently, ships are developing towards intelligent and unmanned operations. During autonomous navigation, real-time monitoring of the distance between ship and shore is necessary to ensure safe nearshore navigation and berthing, while also providing data support for autonomous decision-making.

[0003] To achieve universal ship-shore distance estimation, the shoreline is selected as a shore landmark to address the application limitations of methods based on artificial landmarks. However, the selection of the shoreline makes it impossible to rely on specific shore landmarks to determine the ship-shore distance when performing depth estimation. Therefore, to achieve universal ship-shore distance estimation, it is necessary to estimate the overall depth of the current scene, including the shoreline.

[0004] When estimating scene depth, the depth estimation method based on which the traditional method of estimating the distance between ship and shore relies is easily affected by the weak texture of the water surface and the reflection of the water surface, resulting in poor accuracy of stereo matching between the left and right eye images, which in turn affects the accuracy of scene depth estimation; although the monocular depth estimation method can avoid the influence of factors such as water surface texture, the inverse projection from two-dimensional space to three-dimensional space is a typical ill-posed problem, which has a certain impact on the stability of the estimation results; lidar can provide accurate depth information corresponding to some pixel points, but it is too sparse to achieve dense estimation of scene depth. Summary of the Invention

[0005] The purpose of the present invention is to overcome the defects of the above-mentioned prior art and provide a ship scene depth estimation method, system, device and medium with higher scene depth estimation accuracy.

[0006] The purpose of the present invention can be achieved by the following technical solutions:

[0007] According to a first aspect of the present invention, a method for estimating depth of a ship scene is provided, comprising:

[0008] Acquire multi-source and multi-modal data in ship docking scenarios, including point cloud data and image data;

[0009] A lidar encoder is used to extract point cloud geometric features from point cloud data, and an image encoder is used to extract image texture features from image data;

[0010] Following the path of sparsification, fusion, and diffusion, the point cloud geometric features and image texture features are fused in a domain-consistent manner to obtain a dense geometry-texture feature matrix with consistency between elements. The domain-consistent fusion includes the fusion of sparse domain data with dense domain data and the fusion of geometry domain data with texture domain data.

[0011] According to the dense geometry-texture feature matrix, the depth estimation result of the ship docking scene is obtained through decoding by the decoder.

[0012] Preferably, after acquiring the point cloud data, the method further includes: performing downsampling preprocessing on the point cloud data using a farthest point sampling algorithm.

[0013] Preferably, the laser radar encoder includes a plurality of sequentially arranged dynamic graph convolution modules, each dynamic graph convolution module includes an adjacency matrix calculation unit and two sequentially arranged graph convolution neural networks;

[0014] The specific calculation expression of the dynamic graph convolution module is:

[0015] A=KNN(P,k) (1)

[0016]

[0017] Where: P is the input point cloud feature matrix, k is the number of neighbor nodes, A is the adjacency matrix of the corresponding point cloud; KNN represents the calculation process of the K nearest neighbor algorithm; is the extracted point cloud geometric features; GCNN is the graph convolutional neural network; GN is the group normalization function; ReLU is the activation function.

[0018] Preferably, during the point cloud geometric feature extraction process, the latent features output by the previous dynamic graph convolution module are used as the input of the next dynamic graph convolution module, and the features output by the last dynamic graph convolution module are used as the point cloud geometric features. At the same time, the adjacency matrix is ​​output for subsequent domain-consistent fusion between dense image texture features and sparse point cloud geometric features.

[0019] Preferably, along the path of sparsification, fusion and diffusion, the point cloud geometric features and the image texture features are subjected to domain uniform fusion to obtain a dense geometric-texture feature matrix with consistency between elements; wherein the uniform fusion includes the fusion of sparse domain data with dense domain data and the fusion of geometric domain data with texture domain data, specifically including:

[0020] Align the point cloud geometric features with the image texture features, and thin out the dense image texture features according to the correspondence between pixels to form a sparse texture domain feature point cloud. After compression and splicing with the original sparse point cloud geometric domain features, the original feature matrix is ​​obtained.

[0021] According to the adjacency matrix output by the laser radar encoder during the process of extracting geometric features of the point cloud, the original feature matrix is ​​fused to obtain a sparse geometry-texture feature matrix;

[0022] Based on the alignment results of the point cloud and image, the sparse geometry-texture feature matrix is ​​reconstructed into a sparse matrix with the same resolution as the original image data. The sparse matrix is ​​added to the dense texture domain feature matrix at the element level, and the transposed convolution block is used to diffuse and fuse the heterogeneous data after the element-level addition operation to generate a dense geometry-texture feature matrix with consistency between elements, thereby realizing cross-domain data fusion between camera and lidar point clouds.

[0023] Preferably, the adjacency matrix output from the process of extracting geometric features of point clouds by the laser radar encoder is fused with the original feature matrix through two graph convolutional neural networks arranged in sequence to obtain a sparse geometry-texture feature matrix.

[0024] Preferably, the image encoder is based on the ResNet-50 model, removes the average pooling layer and the fully connected layer in the original ResNet-50 model, uses void convolution to replace all convolution modules in Layer-3 and Layer-4 in the original ResNet-50 model, and adds an upsampling module composed of transposed convolution after the ResNet-50 model.

[0025] Preferably, the decoder as a whole comprises an encoder substructure and a decoder substructure;

[0026] The encoder substructure includes a channel aggregation module and three groups of convolution modules, which are used to map the dense geometry-texture feature matrix into the latent feature space;

[0027] The decoder submodule gradually restores the spatial resolution of the feature matrix based on the geometric-texture features obtained after encoder mapping and combines the multi-scale context information in the latent feature space to complete the scene depth estimation task.

[0028] According to a second aspect of the present invention, an electronic device is provided, comprising a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the program, any one of the methods described above is implemented.

[0029] According to a third aspect of the present invention, a computer-readable storage medium is provided, on which a computer program is stored, and when the program is executed by a processor, any one of the methods described above is implemented.

[0030] Compared with the prior art, the present invention has the following beneficial effects:

[0031] (1) The present invention uses cameras and lidar as sensing devices to perform joint scene depth estimation. By performing a domain-consistent fusion of dense texture features in images and sparse geometric features in lidar point clouds through "sparseness-fusion-diffusion", a deeper and more complete cross-domain fusion of multi-source and multi-modal features is achieved, thereby being able to comprehensively utilize the advantages of different modal features and using the sparse depth of the real world perceived by the lidar as an external constraint to alleviate the impact of the ill-posed problem of inverse projection from two-dimensional space to three-dimensional space.

[0032] (2) Based on the pixel alignment relationship between the point cloud and the image, the dense image features are sparsed, the image pixels that can correspond to the lidar point cloud are extracted, and a sparse texture domain feature point cloud is constructed. The alignment process ensures the consistency of different sensor data in the geometric space.

[0033] (3) The farthest point sampling algorithm is used to downsample the original point cloud data. The spatial geometric information in the original point cloud is maintained by maximizing the distance between all sampling points during the downsampling process.

[0034] (4) The adjacency matrix is ​​introduced, and the graph convolutional neural network is used to deeply fuse the sparse geometric domain feature point cloud and the sparse texture domain feature point cloud. The introduction of the adjacency matrix ensures the retention of the spatial structure information of the point cloud, while enhancing the collaborative expression ability between the two features.

[0035] (5) The heterogeneous data in the fusion feature matrix are diffused by the transposed convolution block to effectively solve the heterogeneity problem and generate a consistent dense geometry-texture feature matrix.

[0036] (6) The image encoder is based on the ResNet-50 model. The average pooling layer and fully connected layer in the original ResNet-50 model are removed, and all convolution modules in Layer-3 and Layer-4 in the original ResNet-50 model are replaced by dilated convolution to ensure that the model receptive field is expanded while maintaining the spatial resolution of the feature matrix unchanged, thereby obtaining deeper texture features. At the same time, an upsampling module composed of transposed convolution is added after the ResNet-50 model to restore the resolution of the feature matrix to the original spatial resolution of the input image. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 Flowchart of the ship scene depth estimation method;

[0038] Figure 2 This is a schematic diagram of the lidar encoder model;

[0039] Figure 3 Schematic diagram of the image encoder model;

[0040] Figure 4 This is a schematic diagram of the domain consistent fusion model;

[0041] Figure 5 Schematic diagram of the decoder architecture. DETAILED DESCRIPTION

[0042] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are part of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of the present invention.

[0043] Example 1

[0044] like Figure 1 As shown, this embodiment provides a method for estimating depth of a ship scene, which includes the following steps:

[0045] S1. Acquire multi-source and multi-modal data in a ship docking scenario, including point cloud data and image data. After acquiring the point cloud data, perform downsampling preprocessing on the point cloud data.

[0046] S2. Use the lidar encoder to extract point cloud geometric features from the point cloud data, and use the image encoder to extract image texture features from the image data.

[0047] In this embodiment, the laser radar encoder includes a plurality of sequentially arranged dynamic graph convolution modules, each of which includes an adjacency matrix calculation unit and two sequentially arranged graph convolution neural networks;

[0048] Specifically, the specific calculation expression of the dynamic graph convolution module is:

[0049] A=KNN(P,k) (1)

[0050]

[0051] Where: P is the input point cloud feature matrix, k is the number of neighbor nodes, A is the adjacency matrix of the corresponding point cloud; KNN represents the calculation process of the K nearest neighbor algorithm; is the extracted point cloud geometric features; GCNN is the graph convolutional neural network; GN is the group normalization function; ReLU is the activation function.

[0052] S3. Perform domain-consistent fusion of the point cloud geometric features and the image texture features along the path of sparsification, fusion, and diffusion to obtain a dense geometry-texture feature matrix with consistency between elements. The domain-consistent fusion includes the fusion of sparse domain data with dense domain data and the fusion of geometry domain data with texture domain data, specifically including:

[0053] S301, aligning the point cloud geometric features and the image texture features, thinning the dense image texture features according to the correspondence between pixels to form a sparse texture domain feature point cloud, and compressing and splicing it with the original sparse point cloud geometric domain features to obtain the original feature matrix;

[0054] S302, according to the adjacency matrix output by the laser radar encoder during the process of extracting geometric features of the point cloud, the original feature matrix is ​​fused to obtain a sparse geometry-texture feature matrix;

[0055] S303. Based on the alignment results of the point cloud and the image, the sparse geometry-texture feature matrix is ​​reconstructed into a sparse matrix with the same resolution as the original image data. The sparse matrix is ​​added to the dense texture domain feature matrix at the element level, and the transposed convolution block is used to diffuse and fuse the heterogeneous data after the element-level addition operation to generate a dense geometry-texture feature matrix with consistency between elements, thereby realizing cross-domain data fusion between camera-lidar point clouds.

[0056] S4. According to the dense geometry-texture feature matrix, the depth estimation result of the ship docking scene is obtained through decoding by the decoder.

[0057] According to the adjacency matrix output by the laser radar encoder during the process of extracting point cloud geometric features, the original feature matrix is ​​fused through two sequentially set graph convolutional neural networks to obtain a sparse geometry-texture feature matrix.

[0058] The image encoder is based on the ResNet-50 model, removing the average pooling layer and fully connected layer in the original ResNet-50 model, replacing all convolution modules in Layer-3 and Layer-4 in the original ResNet-50 model with void convolution, and adding an upsampling module consisting of transposed convolution after the ResNet-50 model.

[0059] The electronic device of the present invention includes a central processing unit (CPU), which can perform various appropriate actions and processes according to computer program instructions stored in a read-only memory (ROM) or loaded from a storage unit into a random access memory (RAM). In the RAM, various programs and data required for device operation can also be stored. The CPU, ROM, and RAM are connected to each other via a bus. An input / output (I / O) interface is also connected to the bus.

[0060] Many components in a device are connected to the I / O interface, including: input units, such as a keyboard and mouse; output units, such as various types of displays and speakers; storage units, such as magnetic disks and optical disks; and communication units, such as network cards, modems, and wireless communication transceivers. The communication unit allows the device to exchange information / data with other devices via computer networks such as the Internet and / or various telecommunication networks.

[0061] The processing unit performs the various methods and processes described above, such as methods S1 to S4. For example, in some embodiments, methods S1 to S4 can be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit. In some embodiments, part or all of the computer program can be loaded and / or installed on the device via a ROM and / or a communication unit. When the computer program is loaded into the RAM and executed by the CPU, one or more steps of methods S1 to S4 described above can be performed. Alternatively, in other embodiments, the CPU can be configured to execute methods S1 to S4 by any other appropriate means (for example, by means of firmware).

[0062] The functions described above herein may be performed, at least in part, by one or more hardware logic components. For example, and without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), and the like.

[0063] The program code for implementing the method of the present invention can be written in any combination of one or more programming languages. Such program code can be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device so that when the program code is executed by the processor or controller, the functions / operations specified in the flow chart and / or block diagram are implemented. The program code can be executed entirely on the machine, partially on the machine, as a stand-alone software package, partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0064] In the context of the present invention, machine-readable medium can be a tangible medium that can contain or store a program for use with an instruction execution system, device or equipment or used in combination with an instruction execution system, device or equipment. Machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. Machine-readable medium can include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared or semiconductor systems, devices or equipment, or any suitable combination of the foregoing. More specific examples of machine-readable storage media can include electrical connections based on one or more lines, portable computer disks, hard disks, random access memories (RAM), read-only memories (ROM), erasable programmable read-only memories (EPROM or flash memory), optical fibers, portable compact disk read-only memories (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0065] Example 2

[0066] This embodiment provides a method for estimating depth of a ship scene, which includes the following steps:

[0067] S1. Acquire multi-source and multi-modal data in the ship docking scenario, including point cloud data and image data.

[0068] In order to reduce the amount of 3D point cloud data and fully ensure that the sampling results retain the spatial geometric information contained in the original 3D point cloud as much as possible, the farthest point sampling algorithm is used to downsample the original point cloud.

[0069] Farthest point sampling: Iteratively select M points on the N-point ship scene point cloud. Each time, select the point with the largest minimum distance to all points in the currently selected point set S = {Po, P1..., Pi} and add it to the set S. The specific steps are as follows:

[0070] 1) From the input point cloud containing N points, select a point P0 as the starting point to obtain a sampling point set S = {P0}.

[0071] 2) Use array L to record the Euclidean distances of all N points to all points in the sampling point set S:

[0072]

[0073] Among them, P0 is the original three-dimensional point cloud, and PS is the three-dimensional point cloud after sampling; then the value assigned at this time is the distance from all points to P0, and the point P1 corresponding to the maximum value is selected and added to the sampling point set S = {P0, P1}.

[0074] 3) Calculate the distance from all points to P1. If the distance is less than L[i], assign it to L[i]. In this way, the array L always maintains the minimum distance between all N points and all points in the sampling point set S as mentioned in 2). Select the point P2 corresponding to the maximum value and add it to the sampling point set S = {P0, P1, P2}.

[0075] 4) Repeat steps 2-3 until the sampling of M points is completed.

[0076] By maximizing the distance between all sampling points during the downsampling process, the spatial geometric information in the original point cloud is maintained. After comprehensively considering the distribution range of the number of point clouds in the dataset, the final number of sampling points N is set to 2000.

[0077] S2. Use the lidar encoder to extract point cloud geometric features from the point cloud data, and use the image encoder to extract image texture features from the image data.

[0078] S201. Extracting geometric features of point clouds.

[0079] like Figure 2 As shown, in this embodiment, the lidar encoder includes five dynamic graph convolution modules (DGCNN-1 to DGCNN-5) arranged in sequence, and each dynamic graph convolution module includes an adjacency matrix calculation unit and two graph convolution neural networks.

[0080] Specifically, the calculation process of the dynamic graph convolution module is as follows:

[0081] A=KNN(P,k) (5)

[0082]

[0083] Where: P is the input point cloud feature matrix, k is the number of neighbor nodes, A is the adjacency matrix of the corresponding point cloud; KNN represents the calculation process of the K nearest neighbor algorithm; is the extracted point cloud geometric features; GCNN is the graph convolutional neural network; GN is the group normalization function; ReLU is the activation function.

[0084] The specific implementation process of the constructed lidar encoder is as follows:

[0085] 1) Point cloud data Mapping to latent feature space

[0086] Point cloud data The number of channels is expanded to S1, S2, S3, S4, S5, and the features of the feature matrix output in the previous stage are continuously extracted to obtain a set of latent features {F i|i=1,2,3,4,5}. Among them, S1, S2, S3, S4, S5 are usually but not limited to 32, 64, 128, 256, 512. F i Usually but not limited to

[0087]

[0088] 2) The last dynamic graph convolution module DGCNN-5 outputs the spatial geometric features extracted from the point cloud according to formulas (5) to (7), and outputs the adjacency matrix A for subsequent domain uniform fusion between dense texture features and sparse geometric features.

[0089] S202: Image texture feature extraction.

[0090] Figure 3 This is a schematic diagram of the image encoder structure of this embodiment. The specific implementation process includes:

[0091] 1) Ship image data L R Adjust the resolution to M×M, where M is usually but not limited to 256. R As input to the image encoder.

[0092] 2) Image data Mapping to the latent feature space, specifically L R Reduce K1, K2, K3, K4 times, and continuously extract the features of the feature matrix output in the previous stage to obtain a set of latent features {H i |i=1,2,3,4,}. K1, K2, K3, K4 are usually but not limited to 2, 4, 4, 8 times. F i Usually but not limited to

[0093] 3) In order to expand the model receptive field while maintaining the spatial resolution of the feature matrix unchanged, and thus obtain deeper texture features, the latent feature H4 is used as input, the number of channels is expanded to 1024 and 2048, and the features of the feature matrix output in the previous stage are continuously extracted to obtain the latent feature

[0094]

[0095] 4) In order to meet the requirement of subsequent modules to align point cloud features with image features, the constructed image encoder adds a set of upsampling modules composed of transposed convolution after ResNet-50, takes the latent feature H6 as input, and converts L R Sequentially expand by 2, 4, and 8 times (the expansion method is usually but not limited to convolution with a step size of 2), and continuously further extract the features of the feature matrix output in the previous stage to obtain a set of latent features Thus, the resolution of the feature matrix is ​​restored to the original spatial resolution of the input image.

[0096] S3, along the path of sparseness, fusion and diffusion, the point cloud geometric features and image texture features are fused in a domain-consistent manner to obtain a dense geometric-texture feature matrix with consistency between elements; Figure 4 As shown, domain uniform fusion includes the fusion of sparse domain data with dense domain data and the fusion of geometric domain data with texture domain data.

[0097] The entire domain consistency module performs cross-domain fusion of multimodal data along the path of "sparseness-fusion-diffusion". The specific implementation includes:

[0098] 1) First, after the image encoder and lidar encoder extract the features of the image and point cloud respectively, the point cloud and image are aligned. That is, the correspondence between the point cloud and each pixel in the image is determined. Based on this correspondence, the dense image features are thinned out, and the pixels in the image features that correspond to the lidar point cloud are extracted to form a sparse texture domain feature point cloud.

[0099] 2) Separately transform the sparse geometric domain feature point cloud Sparse texture domain feature point cloud Mapping to the latent feature space, specifically: reducing its number of channels to 128 and flattening it to obtain the latent features Then FMLP-2 and FMLP-4 are compressed and concatenated to form the original feature matrix.

[0100] 3) The obtained original feature matrix is ​​combined with the adjacency matrix output by the lidar encoder and fused using the graph convolutional neural network shown in formulas (6) and (7) to obtain the sparse geometry-texture feature matrix This achieves the fusion of spatial geometric features and spatial texture features in the sparse domain.

[0101] 4) According to the alignment results of the point cloud and the image, the obtained sparse geometry-texture feature matrix is ​​reconstructed into a sparse matrix with the same resolution as the original input image, and element-wise addition is performed with the dense texture domain feature matrix output by the image encoder. In order to solve the problem of heterogeneous data existing in the same feature matrix at the same time, a 3×3 transposed convolution is used to perform diffusion fusion between heterogeneous data to obtain This generates a dense geometry-texture feature matrix with consistency between elements, realizing cross-domain data fusion between camera-lidar point clouds.

[0102] S4. According to the dense geometry-texture feature matrix, the depth estimation result of the ship docking scene is obtained through decoding by the decoder.

[0103] In order to further promote the diffusion fusion between geometric and texture features and obtain deeper feature information of the fusion matrix, this embodiment designs the overall decoder architecture as a typical encoder-decoder architecture. The encoder submodule consists of a channel aggregation module AGG and three groups of convolution modules Enc-1, Enc-2 and Enc-3. Through a series of downsampling operations, it further performs feature diffusion fusion between pixels on the basis of domain uniform fusion, while extracting multi-scale context information and mapping it to the latent feature space. The decoder submodule gradually restores the spatial resolution of the feature matrix based on the geometric-texture features extracted by the encoder and the multi-scale context information in the latent feature space, and finally completes the scene depth estimation task based on this.

[0104] The specific implementation process is as follows:

[0105] 1) The encoder outputs the geometric and texture features after domain uniform fusion Mapping to latent feature space

[0106] First, the number of channels is reduced to 128 through the channel aggregation module AGG to obtain the latent features

[0107]

[0108] Then, U1 is sequentially reduced by R1, R2, and R3 times (the reduction method is usually but not limited to convolution with a step size of 2), and the features of the feature matrix output in the previous stage are continuously further extracted to obtain a set of latent features {U i |i=2,3,4,}. R1, R2, R3 are usually but not limited to 2, 4, 8 times. U i Usually but not limited to Therefore, on the basis of domain uniform fusion, further feature diffusion fusion between pixels is performed while extracting multi-scale context information and mapping it into the latent feature space.

[0109] 2) The decoder submodule takes the latent feature U4 as input and first expands U4 by 2 times to obtain Then, D3 and U3 are spliced ​​along the channel direction. Repeat the above operation on the spliced ​​feature map to obtain D2 and D1, usually but not limited to The obtained feature map D1 is processed by 1×1 transposed convolution and softmax function to obtain the final result Complete the scene depth estimation task.

[0110] In this example, the method of the present invention was experimentally compared with existing scene depth estimation models in terms of depth estimation performance. All compared models were retrained on the dataset constructed in this study to obtain optimal weights, and then tested on the test set. Among all compared methods, BTS-SAN's input data is RGB images and LiDAR point clouds, and the image texture features are fused with the point cloud geometric features using weighted summation. All other methods use RGB images as input.

[0111] In terms of model performance evaluation indicators, the selected evaluation indicators include error index (the smaller the value, the better) and accuracy index (the larger the value, the better).

[0112] In this embodiment, the selected error indicators include:

[0113] 1) Absolute Relative Error (Abs Rel)

[0114]

[0115] 2) Square Relative Error (Sq Rel)

[0116]

[0117] 3) Root Mean Square Error (RMSE)

[0118]

[0119] 4) and Logarithmic Root Mean Square Error (RMSE log)

[0120]

[0121] 5) The accuracy index is the ratio of the predicted depth to the actual depth in a certain range of pixels, specifically the predicted value y pred Falling into the true value y (0.8y,1.25y), (0.64y,1.25 2 y)、(0.512y,1.25 3 y) is the ratio of the number of pixels within the range to the total number of labeled pixels.

[0122]

[0123] The specific experimental comparison results are shown in Table 1 below.

[0124] Table 1

[0125]

[0126]

[0127] By comparing the experimental results in Table 1 above, it can be seen that the method of using multi-sensor data as input has significantly improved various performance indicators compared with the method of using only RGB images as input data. This fully demonstrates the impact of the ill-posed problem of inverse projection from two-dimensional space to three-dimensional space on the monocular depth estimation model based on RGB images. It also proves that imposing external real-world constraints has a significant alleviating effect on the ill-posed problem in inverse projection. In addition, compared with the BTS-SAN model, which is a multi-modal feature matrix fusion method based on weighted summation, the domain-consistent fusion depth estimation model implemented based on the domain-consistent fusion module achieves better depth estimation performance. This result shows that cross-domain fusion of multi-source and multi-modal data based on a unified data domain can better integrate multi-source sensor data, solve the interference caused by different sensor features during fusion, and achieve deeper and more complete cross-domain fusion of multi-source and multi-modal features.

[0128] The other configurations of this embodiment are the same as those of embodiment 1.

[0129] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in the present invention, and such modifications or substitutions are intended to be within the scope of protection of the present invention. Therefore, the scope of protection of the present invention shall be subject to the scope of protection of the claims.

Claims

1. A method for estimating the depth of a ship scene, characterized in that: include: Acquire multi-source and multi-modal data in ship docking scenarios, including point cloud data and image data; A lidar encoder is used to extract point cloud geometric features from point cloud data, and an image encoder is used to extract image texture features from image data; Following the path of sparsification, fusion, and diffusion, the point cloud geometric features and image texture features are fused in a domain-consistent manner to obtain a dense geometry-texture feature matrix with consistency between elements. The domain-consistent fusion includes the fusion of sparse domain data with dense domain data and the fusion of geometry domain data with texture domain data. According to the dense geometry-texture feature matrix, the depth estimation result of the ship docking scene is obtained through decoding by the decoder.

2. A ship scene depth estimation method according to claim 1, characterized in that: After obtaining the point cloud data, the method further includes: performing downsampling preprocessing on the point cloud data using the farthest point sampling algorithm.

3. A ship scene depth estimation method according to claim 1, characterized in that: The laser radar encoder includes a plurality of dynamic graph convolution modules arranged in sequence, each dynamic graph convolution module includes an adjacency matrix calculation unit and two graph convolution neural networks arranged in sequence; The specific calculation expression of the dynamic graph convolution module is: A=KNN(P,k) (1) Where: P is the input point cloud feature matrix, k is the number of neighbor nodes, A is the adjacency matrix of the corresponding point cloud; KNN represents the calculation process of the K nearest neighbor algorithm; is the extracted point cloud geometric features; GCNN is the graph convolutional neural network; GN is the group normalization function; ReLU is the activation function.

4. A method for estimating depth of a ship scene according to claim 3, characterized in that: During the point cloud geometric feature extraction process, the latent features output by the previous dynamic graph convolution module are used as the input of the next dynamic graph convolution module, and the features output by the last dynamic graph convolution module are used as the point cloud geometric features. At the same time, an adjacency matrix is ​​output for subsequent domain-consistent fusion between dense image texture features and sparse point cloud geometric features.

5. A method for estimating depth of a ship scene according to claim 1, characterized in that: Following the path of sparsification, fusion, and diffusion, the point cloud geometric features and image texture features are fused in a domain-consistent manner to obtain a dense geometry-texture feature matrix with consistency between elements. The consistent fusion includes the fusion of sparse domain data with dense domain data and the fusion of geometry domain data with texture domain data, specifically including: Align the point cloud geometric features with the image texture features, and thin out the dense image texture features according to the correspondence between pixels to form a sparse texture domain feature point cloud. After compression and splicing with the original sparse point cloud geometric domain features, the original feature matrix is ​​obtained. According to the adjacency matrix output by the laser radar encoder during the process of extracting geometric features of the point cloud, the original feature matrix is ​​fused to obtain a sparse geometry-texture feature matrix; Based on the alignment results of the point cloud and image, the sparse geometry-texture feature matrix is ​​reconstructed into a sparse matrix with the same resolution as the original image data. The sparse matrix is ​​added to the dense texture domain feature matrix at the element level, and the transposed convolution block is used to diffuse and fuse the heterogeneous data after the element-level addition operation to generate a dense geometry-texture feature matrix with consistency between elements, thereby realizing cross-domain data fusion between camera and lidar point clouds.

6. A method for estimating depth of a ship scene according to claim 5, characterized in that: The adjacency matrix output from the process of extracting geometric features of point clouds according to the laser radar encoder is fused with the original feature matrix through two graph convolutional neural networks set in sequence to obtain a sparse geometry-texture feature matrix.

7. A method for estimating depth of a ship scene according to claim 1, characterized in that: The image encoder is based on the ResNet-50 model, removing the average pooling layer and fully connected layer in the original ResNet-50 model, using void convolution to replace all convolution modules in Layer-3 and Layer-4 in the original ResNet-50 model, and adding an upsampling module consisting of transposed convolution after the ResNet-50 model.

8. A method for estimating depth of a ship scene according to claim 1, characterized in that: The decoder as a whole includes an encoder substructure and a decoder substructure; The encoder substructure includes a channel aggregation module and three groups of convolution modules, which are used to map the dense geometry-texture feature matrix into the latent feature space; The decoder submodule gradually restores the spatial resolution of the feature matrix based on the geometric-texture features obtained after encoder mapping and combines the multi-scale context information in the latent feature space to complete the scene depth estimation task.

9. An electronic device comprising a memory and a processor, wherein a computer program is stored in the memory, wherein: When the processor executes the program, the method according to any one of claims 1 to 8 is implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 8 is implemented.

Images