Generalizable, general monocular depth map inference methods, systems, media, and devices

Abstract

The application discloses a general monocular depth map inference method, system, medium and equipment, obtains a training data set containing input data and supervision data; constructs a strong generalization loss function for general monocular depth map inference; constructs a 7-layer U-shaped convolutional neural network stacked by improved convolution operation blocks; based on the training data set and the strong generalization loss function, the 7-layer U-shaped convolutional neural network is trained by using a gradient descent method, and a depth map of the same viewpoint is inferred by using the trained U-shaped convolutional neural network. The application can adapt to changes of sensors and application scenarios, can process unobserved application scenarios, and has good universality and strong generalization.

Description

Technical Field

[0001] This invention belongs to the field of image processing technology, specifically relating to a generalizable monocular depth map inference method, system, medium, and device. Background Technology

[0002] Monocular depth map reasoning is one of the fundamental research problems in robot perception of the real world. It involves reasoning from raw data acquired by physical sensors from a single viewpoint to obtain a depth map of the scene at that viewpoint, representing the distance information between objects and sensors. Depth maps can be widely used in various robot-related applications such as object detection, 3D reconstruction, and robot localization.

[0003] For different applications, customized robots equipped with specific sensors are typically used to perceive specific application scenarios. For example, existing physical sensors mainly include visual cameras, time-of-flight sensors, structured light sensors, radar sensors, and combinations of visual cameras with other sensors; common application scenarios include indoor scenes such as bedrooms, kitchens, and factories, and outdoor scenes such as roads, cities, and towns. Numerous monocular depth map inference methods have emerged to address these different physical sensors and application scenarios. However, these customized monocular depth map inference methods cannot adapt to changes in sensors and application scenarios, and are not universally applicable to various robotic applications.

[0004] In fact, the industry desires a universal monocular depth map inference method that can be applied to various robots, adapting well to multiple sensors or even combinations of sensors, while also meeting the industry's strong generalization requirements for diverse application scenarios. In contrast, existing monocular depth map inference methods primarily focus on improving the performance of specific applications. Therefore, there is an urgent need to develop a generalizable monocular depth map inference method to address the challenge of existing methods' inability to adapt to changes in sensors and application scenarios. This method would reduce the customization costs of monocular depth inference methods, improve the reusability of related application modules, and facilitate the rapid implementation and development of related industries. Summary of the Invention

[0005] The technical problem to be solved by the present invention is to provide a generalizable monocular depth map inference method, system, medium and device that addresses the shortcomings of the prior art, thereby solving the technical problem that the existing monocular depth map inference methods cannot adapt to changes in sensors and application scenarios.

[0006] The present invention adopts the following technical solution:

[0007] A generalizable monocular depth map inference method includes the following steps:

[0008] S1. Obtain the training dataset containing input data and supervision data;

[0009] S2. Construct a strong generalization loss function for general monocular depth map inference;

[0010] S3. Construct a 7-layer U-shaped convolutional neural network consisting of stacked improved convolutional operation blocks;

[0011] S4. Based on the training dataset obtained in step S1 and the strong generalization loss function determined in step S2, the 7-layer U-shaped convolutional neural network constructed in step S3 is trained using the gradient descent method, and the depth map of the same viewpoint is obtained by inference using the trained U-shaped convolutional neural network.

[0012] Specifically, in step S1, the input data includes a visual image and an optional raw depth map; the supervision data is a depth map from the same viewpoint as the input data.

[0013] Furthermore, an optional original depth map is obtained by manually degrading the supervised depth map, specifically as follows:

[0014] Simulated noise was added by changing the parameters of Gaussian noise and salt-and-pepper noise; simulated blur was added by changing the image resolution; simulated sparsity was added by random sampling of data; and simulated holes were added by creating a hole dataset and randomly sampling holes from the dataset. The supervised depth map was degraded into an optional original depth map.

[0015] Specifically, in step S2, the strong generalization loss function Specifically:

[0016]

[0017] in, For scale-adaptive loss, The loss is a scale-invariant multi-scale gradient loss, where λ is the scale-adaptive loss. With scale-invariant multiscale gradient loss The balance coefficient between them.

[0018] Furthermore, scale-adaptive loss Specifically:

[0019]

[0020] Where M is the number of valid pixels in the supervised depth map, M V To input the number of valid pixels in the depth map, d i Let z be the depth value of the inferred depth map at pixel i. i To monitor the depth value of the depth map at pixel i, The mean of the inferred depth map. To supervise the mean of the depth map, σ d σ is the average difference of the inferred depth map. z To supervise the average difference of the depth map, d v The depth value of the inferred depth map at the corresponding valid pixel v in the input depth map is z. v To monitor the depth value of the depth map corresponding to the valid pixel v in the input depth map, ∈ is used to prevent small real numbers with a denominator of 0.

[0021] Furthermore, scale-invariant multiscale gradient loss Specifically:

[0022]

[0023] Where, ρ k Let k be the nearest neighbor downsampling function, and k be the scaling factor. and For the downsampled image The Sobel gradients in the x and y directions are respectively, and M is the number of effective pixels in the supervised depth map.

[0024] Specifically, in step S3, each of the 7-layer U-shaped convolutional neural network, excluding the initial layer, contains three improved convolutional operation blocks. The improved convolutional operation blocks adopt residual convolutional operation blocks without normalization layers based on ReZero technology, which are composed of LReLU activation functions and convolution stacked together.

[0025] Secondly, embodiments of the present invention provide a generalizable monocular depth map inference system, comprising:

[0026] The data module obtains the training dataset containing input data and supervision data;

[0027] The function module constructs a strong generalization loss function for general monocular depth map inference;

[0028] The network module constructs a 7-layer U-shaped convolutional neural network consisting of stacked improved convolutional operation blocks;

[0029] The inference module, based on the training dataset obtained from the data module and the strong generalization loss function determined by the function module, trains a 7-layer U-shaped convolutional neural network constructed by the network module using the gradient descent method, and uses the trained U-shaped convolutional neural network to infer the depth map of the same viewpoint.

[0030] Thirdly, a computer device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the computer program, implements the steps of the generalizable monocular depth map inference method described above.

[0031] Fourthly, embodiments of the present invention provide a computer-readable storage medium including a computer program, which, when executed by a processor, implements the steps of the above-described generalizable monocular depth map inference method.

[0032] Compared with the prior art, the present invention has at least the following beneficial effects:

[0033] A generalizable monocular depth map inference method is proposed, which infers high-quality depth maps from visual images acquired by a single-viewpoint sensor and optional raw depth maps. It can adapt to changes in sensors and application scenarios, and has good versatility and strong generalization. It is implemented using diverse large-scale training data, a strong generalization loss function, and a neural network based on improved convolutional blocks.

[0034] Furthermore, the training data acquisition process provides convolutional neural networks with a massive and diverse dataset, enabling the trained convolutional neural networks to adapt to unseen application scenarios and unseen sensor data content.

[0035] Furthermore, using artificial degradation methods to obtain optional raw depth maps can solve the problem of the lack of raw depth maps in existing public datasets, expand the channels for obtaining training data, and provide a guarantee for obtaining ultra-large-scale training data.

[0036] Furthermore, the strong generalization loss function provides convolutional neural networks with a generalizable monocular depth map inference task objective, enabling convolutional neural networks to focus on learning how to adapt to various working conditions, including changes in sensors and application scenarios.

[0037] Furthermore, scale-adaptive loss The method provides scale constraints for the output depth map, enabling it to automatically adjust the scale of the output depth map based on different sensor data inputs, thereby ensuring that the method can adapt to changes in sensors and application scenarios.

[0038] Furthermore, scale-invariant multiscale gradient loss Smoothness constraints are provided for the output depth map, ensuring the smoothness and structural fineness of the depth map, thereby improving the quality of the inference depth map.

[0039] Furthermore, the improved convolutional neural network provides a better network structure and computational units for fitting the strong generalization loss function and the training data, thereby improving the convergence performance of the convolutional neural network under various working conditions.

[0040] It is understood that the beneficial effects of the second to fourth aspects mentioned above can be found in the relevant descriptions in the first aspect mentioned above, and will not be repeated here.

[0041] In summary, this invention can adapt to changes in sensors and application scenarios, and can handle unseen application scenarios, demonstrating good versatility and strong generalization.

[0042] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description

[0043] Figure 1 The test results diagram of the present invention are shown below, where (a) is the input visual image, (b) is the optional input original depth map, (c) is the depth map inferred by the present invention, and (d) is the supervised depth map.

[0044] Figure 2 The flowchart for obtaining the optional raw depth map is shown below.

[0045] Figure 3 This is a diagram of the architecture of a 7-layer U-shaped convolutional neural network;

[0046] Figure 4 A schematic diagram for improving the convolution operation block;

[0047] Figure 5 The test results diagrams for the present invention and other methods are shown below, where (a) represents the input sensor data, (b) represents the test results diagrams for other methods, and (c) represents the test results diagrams for the present invention. Detailed Implementation

[0048] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0049] In the description of this invention, it should be understood that the terms "comprising" and "including" indicate the presence of the described features, integrals, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or collections thereof.

[0050] It should also be understood that the terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to limit the invention. As used in this specification and the appended claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms unless the context clearly indicates otherwise.

[0051] It should also be further understood that the term "and / or" as used in this specification and the appended claims refers to any combination and all possible combinations of one or more of the associated listed items, and includes such combinations. For example, A and / or B can represent three cases: A alone, A and B simultaneously, and B alone. Additionally, the character " / " in this document generally indicates that the preceding and following objects have an "or" relationship.

[0052] It should be understood that although terms such as first, second, third, etc., may be used in the embodiments of the present invention to describe the preset range, these preset ranges should not be limited to these terms. These terms are only used to distinguish the preset ranges from one another. For example, without departing from the scope of the embodiments of the present invention, the first preset range may also be referred to as the second preset range, and similarly, the second preset range may also be referred to as the first preset range.

[0053] Depending on the context, the word "if" as used here can be interpreted as "when," "when," "in response to determination," or "in response to detection." Similarly, depending on the context, the phrase "if determination" or "if detection (of the stated condition or event)" can be interpreted as "when determination," "in response to determination," "when detection (of the stated condition or event)," or "in response to detection (of the stated condition or event)."

[0054] The accompanying drawings illustrate various structural schematic diagrams according to embodiments disclosed in this invention. These drawings are not to scale, and some details have been enlarged for clarity, and some details may have been omitted. The shapes of the various regions and layers shown in the drawings, as well as their relative sizes and positional relationships, are merely exemplary and may deviate from reality due to manufacturing tolerances or technical limitations. Furthermore, those skilled in the art can design regions / layers with different shapes, sizes, and relative positions as needed.

[0055] This invention provides a generalizable monocular depth map inference method. It establishes a training data acquisition process to obtain a large-scale, diverse dataset; establishes a strong generalization loss function for general monocular depth map inference; builds an improved convolutional neural network based on the strong generalization loss function; trains the convolutional neural network using the training data acquisition process, the strong generalization loss function, and the improved convolutional neural network; and uses the trained convolutional neural network to infer high-quality depth maps from the input sensor data into co-viewpoint depth maps.

[0056] This invention provides a generalizable monocular depth map inference method, comprising the following steps:

[0057] S1. Training Data Acquisition Process

[0058] The training dataset for monocular depth map inference includes input data and supervised data; the input data, i.e., sensor data, includes visual images such as... Figure 1 (a) with the optional original depth map as shown Figure 1 (b); Supervisory data refers to high-quality depth maps from the same viewpoint as the input data provided in existing public datasets, such as... Figure 1 (d)

[0059] S101. Acquisition of input visual image and supervised depth map

[0060] Based on the input visual images and supervised depth maps provided by public datasets, a large-scale and diverse training dataset is established by directly mixing multiple datasets.

[0061] S102, Optional acquisition of the original depth image

[0062] Please see Figure 2 Based on the large-scale and diverse training dataset in step S101, the original input depth map for simulation is obtained by artificially degrading the supervised depth map; the artificial degradation methods include:

[0063] Noise can be added to the simulation by changing the parameters of Gaussian noise and salt-and-pepper noise; blur can be added to the simulation by changing the image resolution; sparsity can be added to the simulation by random sampling of data; and holes can be added to the simulation by creating a hole dataset and randomly sampling holes from that dataset.

[0064] S2, Strong generalization loss function for general monocular depth map inference

[0065] The loss function is set as follows:

[0066]

[0067] Where λ is the scale-adaptive loss. With scale-invariant multiscale gradient loss The balance coefficient between the two.

[0068] S201, Scale-Adaptive Loss Setting

[0069] Scale-adaptive loss The settings are as follows:

[0070]

[0071] Where M is the number of valid pixels in the supervised depth map, M V To input the number of valid pixels in the depth map, d i Let z be the depth value of the inferred depth map at pixel i. iTo monitor the depth value of the depth map at pixel i, The mean of the inferred depth map. To supervise the mean of the depth map, σ d σ is the mean deviation of the inferred depth map. z To supervise the average difference of the depth map, d v The depth value of the inferred depth map at the corresponding valid pixel v in the input depth map is z. v To monitor the depth value of the depth map corresponding to the valid pixel v in the input depth map, ∈ is used to prevent small real numbers with a denominator of 0.

[0072] S202, Scale-invariant multi-scale gradient loss setting

[0073] Scale-invariant multiscale gradient loss The settings are as follows:

[0074]

[0075]

[0076] Where, ρ k Let k be the nearest neighbor downsampling function, and k be the scaling factor. and For the downsampled image Sobel gradients in the x and y directions, respectively.

[0077] S3, Improved Convolutional Neural Network

[0078] This invention employs a 7-layer U-shaped convolutional neural network composed of improved convolutional operation blocks stacked together.

[0079] S301, 7-layer U-shaped convolutional neural network

[0080] like Figure 3 Apart from the initial layer, each layer of the 7-layer U-shaped convolutional neural network contains three improved convolutional operation blocks. Furthermore, the convolutional dimensions of the improved convolutional operation blocks differ between layers.

[0081] S302, Improved Convolution Operation Block

[0082] Please see Figure 4 The network employs a residual convolutional block with no normalized layers based on the ReZero technique. This block consists of stacked LReLU activation functions and convolutions, where α is the scaling factor of the ReZero technique.

[0083] S4. Based on the above training data acquisition process and strong generalization loss function, the improved convolutional neural network is trained using the gradient descent algorithm in deep learning. The trained convolutional neural network can directly infer high-quality depth maps from the same viewpoint from the input sensor data.

[0084] In another embodiment of the present invention, a generalizable monocular depth map inference system is provided. This system can be used to implement the above-mentioned generalizable monocular depth map inference method. Specifically, the generalizable monocular depth map inference system includes a data module, a function module, a network module, and an inference module.

[0085] The data module acquires the training dataset, which includes input data and supervision data.

[0086] The function module constructs a strong generalization loss function for general monocular depth map inference;

[0087] The network module constructs a 7-layer U-shaped convolutional neural network consisting of stacked improved convolutional operation blocks;

[0088] The inference module, based on the training dataset obtained from the data module and the strong generalization loss function determined by the function module, trains a 7-layer U-shaped convolutional neural network constructed by the network module using the gradient descent method, and uses the trained U-shaped convolutional neural network to infer the depth map of the same viewpoint.

[0089] In another embodiment of the present invention, a terminal device is provided, comprising a processor and a memory. The memory stores a computer program, which includes program instructions. The processor executes the program instructions stored in the computer storage medium. The processor may be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. It is the computing and control core of the terminal, suitable for implementing one or more instructions, specifically suitable for loading and executing one or more instructions to implement a corresponding method flow or corresponding function. The processor described in this embodiment can be used for the operation of a generalizable monocular depth map inference method, including:

[0090] Obtain a training dataset containing input and supervision data; construct a strong generalization loss function for general monocular depth map inference; construct a 7-layer U-shaped convolutional neural network consisting of stacked improved convolutional operation blocks; train the 7-layer U-shaped convolutional neural network using gradient descent based on the training dataset and the strong generalization loss function, and use the trained U-shaped convolutional neural network to infer the depth map of the same viewpoint.

[0091] In another embodiment of the present invention, a storage medium is also provided, specifically a computer-readable storage medium (memory). This computer-readable storage medium is a memory device in a terminal device used to store programs and data. It is understood that the computer-readable storage medium here can include both the built-in storage medium in the terminal device and extended storage media supported by the terminal device. The computer-readable storage medium provides storage space that stores the terminal's operating system. Furthermore, this storage space also stores one or more instructions suitable for loading and execution by a processor. These instructions can be one or more computer programs (including program code). It should be noted that the computer-readable storage medium here can be high-speed RAM or non-volatile memory, such as at least one disk storage device.

[0092] One or more instructions stored in a computer-readable storage medium can be loaded and executed by a processor to implement the corresponding steps of the generalizable monocular depth map inference method in the above embodiments; one or more instructions in the computer-readable storage medium are loaded and executed by the processor in the following steps:

[0093] Obtain a training dataset containing input and supervision data; construct a strong generalization loss function for general monocular depth map inference; construct a 7-layer U-shaped convolutional neural network consisting of stacked improved convolutional operation blocks; train the 7-layer U-shaped convolutional neural network using gradient descent based on the training dataset and the strong generalization loss function, and use the trained U-shaped convolutional neural network to infer the depth map of the same viewpoint.

[0094] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. The components of the embodiments of the present invention described and shown in the accompanying drawings can generally be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely to illustrate selected embodiments of the invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without inventive effort are within the scope of protection of the present invention.

[0095] Please see Figure 1 The test results on the test set are as follows Figure 1 (c) is the depth map deduced by this invention. Figure 1 As can be seen, this invention can complete monocular depth reasoning tasks under different sensors and application scenarios, demonstrating its good versatility and strong generalization ability. Meanwhile, Figure 1 (c) shows that the inferred depth map has accurate depth and clear boundaries, and is close to... Figure 1 (d) shows the supervision depth map.

[0096] Please see Figure 5 Existing monocular depth map inference methods cannot adapt to changes in sensors and application scenarios. Figure 5 In (b), 'MiDas', 'NLSPN', and 'JIFF' are representative methods for existing monocular depth map inference. Figure 5 This is for application scenarios that have not been seen by any monocular depth map inference method. 'MiDas' is applicable only to sensor 1; 'NLSPN' is applicable only to sensors 2-4, but performs poorly in unseen application scenarios; 'JIFF' is applicable only to sensor 5, but performs only moderately in unseen application scenarios. Figure 5(c) represents the inference result of the method described in this invention. This inference result is accurate and well-structured, and adapts well to unseen application scenarios and various different sensors. Here, 'MiDas' refers to R. Ranftl's paper "Towards Robust Monocular Depth Estimation: Mixing Datasets for Zero-Shot Cross-Dataset Transfer". 'NLSPN' refers to J. Park's paper "Non-local Spatial Propagation Network for Depth Completion". 'JIFF' refers to X. Yu's paper "Joint Implicit Image Function for Guided Depth Super-Resolution".

[0097] In summary, this invention provides a generalizable monocular depth map inference method, system, medium, and device, aiming to infer high-quality depth maps from data from a single-viewpoint sensor. This method can adapt to changes in sensors and application scenarios, and can handle unseen application scenarios, exhibiting good versatility and strong generalization capabilities.

[0098] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this application. The specific working process of the units and modules in the above system can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0099] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0100] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0101] In the embodiments provided by this invention, it should be understood that the disclosed devices / terminals and methods can be implemented in other ways. For example, the device / terminal embodiments described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0102] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0103] Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0104] If the integrated module / unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include: any entity or device capable of carrying the computer program code, recording media, USB flash drives, portable hard drives, magnetic disks, optical disks, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the content included in the computer-readable medium can be appropriately added or removed according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media do not include electrical carrier signals and telecommunication signals.

[0105] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0106] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0107] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0108] The above content is only for illustrating the technical concept of the present invention and should not be construed as limiting the scope of protection of the present invention. Any modifications made to the technical solution based on the technical concept proposed in this invention shall fall within the scope of protection of the claims of this invention.

Claims

1. A generalizable monocular depth map reasoning method, characterized in that, Includes the following steps: S1. Obtain a training dataset containing input data and supervision data. The input data includes visual images or visual images and original depth maps; the supervision data is a depth map from the same viewpoint as the input data. The original depth map is obtained by manually degrading the supervision depth map. Specifically: Simulated noise is added to the supervised depth map by changing the parameters of Gaussian noise and salt-and-pepper noise; simulated blur is added by changing the image resolution; simulated sparsity is added by randomly sampling the number of effective pixels; and simulated holes are added by randomly sampling and enhancing the hole image from a pre-established hole dataset. The supervised depth image is progressively degraded into the original depth image required for input by four types of simulated distortion. The hole dataset is created from the training set, and holes are extracted from depth maps with more than 60% effective pixels and saved as a hole dataset. Simulated holes are added by randomly sampling holes from the pre-established hole dataset, and the supervised depth map is degraded into the original depth map. S2. Construct a strong generalization loss function for general monocular depth map inference. Specifically: in, For scale-adaptive loss, For scale-invariant multiscale gradient loss, Scale-adaptive loss With scale-invariant multiscale gradient loss The balance coefficient between them, scale-adaptive loss Specifically: in, To monitor the number of valid pixels in the depth map, To input the number of valid pixels in the depth map, For the inferred depth map at pixels The depth value, To supervise the depth map at pixels The depth value, The mean of the inferred depth map. To monitor the mean of the depth map, The average difference of the inferred depth map, To monitor the average difference in depth maps, The inferred depth map is based on the effective pixels of the input depth map. The corresponding depth value, To supervise the depth map at the valid pixels of the input depth map The corresponding depth value, To prevent real numbers with a denominator of 0; S3. Construct a 7-layer U-shaped convolutional neural network consisting of stacked improved convolutional operation blocks. Except for the initial layer, each of the 7-layer U-shaped convolutional neural network contains three improved convolutional operation blocks. The improved convolutional operation blocks adopt residual convolutional operation blocks without normalization layers based on ReZero technology, which are formed by stacking LReLU activation function and convolution. S4. Based on the training dataset obtained in step S1 and the strong generalization loss function determined in step S2, the 7-layer U-shaped convolutional neural network constructed in step S3 is trained using the gradient descent method, and the depth map of the same viewpoint is obtained by inference using the trained U-shaped convolutional neural network.

2. The generalizable monocular depth map inference method according to claim 1, characterized in that, Scale-invariant multiscale gradient loss Specifically: in, For nearest neighbor downsampling function, As a scale factor, and For the downsampled image Sobel gradients in the x and y directions, respectively. To monitor the number of valid pixels in the depth map.

3. A generalizable monocular depth map inference system, characterized in that, include: The data module acquires a training dataset containing input and supervision data. Input data includes visual images or visual images combined with the original depth map. Supervision data consists of depth maps from the same viewpoint as the input data. The original depth map is obtained using a method of manually degrading the supervision depth map. Simulated noise is added to the supervised depth map by changing the parameters of Gaussian noise and salt-and-pepper noise; simulated blur is added by changing the image resolution; simulated sparsity is added by randomly sampling the number of effective pixels; and simulated holes are added by randomly sampling and enhancing the hole image from a pre-established hole dataset. The supervised depth image is progressively degraded into the original depth image required for input by four types of simulated distortion. The hole dataset is created from the training set, and holes are extracted from depth maps with more than 60% effective pixels and saved as a hole dataset. Simulated holes are added by randomly sampling holes from the pre-established hole dataset, and the supervised depth map is degraded into the original depth map. The function module constructs a strong generalization loss function for general monocular depth map inference. Specifically: in, For scale-adaptive loss, For scale-invariant multiscale gradient loss, Scale-adaptive loss With scale-invariant multiscale gradient loss The balance coefficient between them, scale-adaptive loss Specifically: in, To monitor the number of valid pixels in the depth map, To input the number of valid pixels in the depth map, For the inferred depth map at pixels The depth value, To supervise the depth map at pixels The depth value, The mean of the inferred depth map. To monitor the mean of the depth map, The average difference of the inferred depth map, To monitor the average difference in depth maps, The inferred depth map is based on the effective pixels of the input depth map. The corresponding depth value, To supervise the depth map at the valid pixels of the input depth map The corresponding depth value, To prevent real numbers with a denominator of 0; The network module constructs a 7-layer U-shaped convolutional neural network consisting of stacked improved convolutional operation blocks; The inference module, based on the training dataset obtained from the data module and the strong generalization loss function determined by the function module, trains a 7-layer U-shaped convolutional neural network constructed by the network module using the gradient descent method, and uses the trained U-shaped convolutional neural network to infer the depth map of the same viewpoint.

4. A computer-readable storage medium for storing one or more programs, characterized in that, The one or more programs include instructions that, when executed by a computing device, cause the computing device to perform the method of claim 1 or 2.

5. A computing device, characterized in that, include: One or more processors, a memory, and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, the one or more programs including steps for performing the method of claim 1 or 2.

Images