Depth estimation method based on infrared image, electronic device, and storage medium

Abstract

Embodiments of the present application provide a depth estimation method based on an infrared image, an electronic device and a storage medium. The method comprises: obtaining a to-be-processed infrared image; inputting the to-be-processed infrared image into a target depth estimation model to obtain a corresponding first depth estimation result, the first depth estimation result being information related to the depth of each pixel in the to-be-processed infrared image; and determining first depth information corresponding to the to-be-processed infrared image based on the first depth estimation result. The target depth estimation model is trained using synthetic infrared training data, the synthetic infrared training data comprising a synthetic infrared image and labeled depth information, the synthetic infrared image being generated based on a labeled RGB image, and the labeled depth information being depth information corresponding to the labeled RGB image. Since the target depth estimation model is trained using the synthetic infrared image, the available training data is sufficient, and the performance of the obtained target depth estimation model is also relatively robust.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and more specifically to a depth estimation method based on infrared images, electronic devices, storage media, and computer program products. Background Technology

[0002] In many technological fields, such as autonomous driving, it is necessary to acquire 3D information about the application scene. This typically requires sensors to collect scene information. Currently, the most commonly used and economical sensor for acquiring 3D information about application scenes is the monocular camera. However, the 2D images acquired by a monocular camera are insufficient to directly and effectively obtain the 3D structural information of the application scene. Therefore, depth estimation of the 2D images is necessary to obtain the 3D information. Currently, depth estimation is mainly performed using visible light images (i.e., RGB images) to obtain the 3D information of the application scene. However, in low-light scenes such as at night, the scene information in images acquired by a typical RGB camera is significantly lacking. Infrared cameras, on the other hand, can capture rich scene information in nighttime scenes. Therefore, depth estimation based on infrared images is of great value for acquiring 3D information in low-light scenarios.

[0003] In the field of monocular depth estimation, deep neural networks, due to their powerful feature learning capabilities, can be trained on large amounts of data to make good predictions of image depth even when 3D information is lacking. However, in the infrared depth domain, depth-annotated datasets are very scarce, which limits the application of deep neural networks, which require large-scale data training, in infrared depth estimation. Summary of the Invention

[0004] This application is made in view of the above-mentioned problems. This application provides a depth estimation method based on infrared images, an electronic device, a storage medium, and a computer program product.

[0005] According to one aspect of this application, a depth estimation method based on infrared images is provided, comprising: acquiring an infrared image to be processed; inputting the infrared image to be processed into a target depth estimation model to obtain a corresponding first depth estimation result, wherein the first depth estimation result is information related to the depth of each pixel in the infrared image to be processed; determining the first depth estimation result as the first depth information corresponding to the infrared image to be processed, or determining the first depth information based on the first depth estimation result; wherein the target depth estimation model is obtained by training using synthetic infrared training data, wherein the synthetic infrared training data includes a synthetic infrared image and labeled depth information, wherein the synthetic infrared image is generated based on a labeled RGB image, and the labeled depth information is the depth information corresponding to the labeled RGB image.

[0006] For example, a synthetic infrared image is generated by: obtaining a labeled RGB image; inputting the labeled RGB image into a style transfer network to obtain a synthetic infrared image.

[0007] For example, the style transfer network is trained as follows: A sample RGB image and a first sample infrared image are input into the style transfer network to obtain a first predicted RGB image and a first predicted infrared image corresponding to the sample RGB image, and a second predicted RGB image and a second predicted infrared image corresponding to the first sample infrared image; the first predicted RGB image is input as a positive sample into a first discriminant network to obtain a first discriminant result; the second predicted RGB image is input as a negative sample into the first discriminant network to obtain a second discriminant result; the first predicted infrared image is input as a negative sample into the first discriminant network to obtain a third discriminant result; the second predicted infrared image is input as a positive sample into the first discriminant network to obtain a fourth discriminant result; a first prediction loss is calculated based on the first, second, third, and fourth discriminant results; and adversarial training is performed on the style transfer network and the first discriminant network based on the first prediction loss.

[0008] For example, inputting a sample RGB image and a first sample infrared image into a style transfer network to obtain a first predicted RGB image and a first predicted infrared image corresponding to the sample RGB image, and a second predicted RGB image and a second predicted infrared image corresponding to the first sample infrared image, includes: performing the following operations through the style transfer network: performing a first encoding operation on the sample RGB image to obtain a first encoded feature; performing a second encoding operation on the first sample infrared image to obtain a second encoded feature; merging the first encoded feature and the second encoded feature together to obtain a merged feature; performing a first decoding operation on the merged feature to obtain the first predicted RGB image and the second predicted RGB image; and performing a second decoding operation on the merged feature to obtain the first predicted infrared image and the second predicted infrared image.

[0009] For example, the target depth estimation model is trained through the following first training operation: acquiring a synthetic infrared image and labeled depth information; inputting the synthetic infrared image into the target depth estimation model to obtain a corresponding first prediction estimation result, wherein the first prediction estimation result is information related to the depth of each pixel in the synthetic infrared image; determining the first predicted depth information corresponding to the synthetic infrared image based on the first prediction estimation result; calculating a second prediction loss based on the labeled depth information and the first predicted depth information; and optimizing the parameters in the target depth estimation model based on the second prediction loss; or, the target depth estimation model is trained through the following second training operation: acquiring a second sample infrared image and a third sample infrared image; initializing the weights of the target depth estimation model using the weights of the depth estimation model to be transferred, wherein the network structure of the depth estimation model to be transferred and the target depth estimation model are the same, and the depth estimation model to be transferred is trained based on synthetic infrared training data; The second sample infrared image is input into the depth estimation model to be transferred to obtain the corresponding second prediction estimation result. The second prediction estimation result is the depth-related information of each pixel in the second sample infrared image. Based on the second prediction estimation result, the second predicted depth information corresponding to the second sample infrared image is determined. The third sample infrared image is input into the target depth estimation model to obtain the corresponding third prediction estimation result. The third prediction estimation result is the depth-related information of each pixel in the third sample infrared image. Based on the third prediction estimation result, the third predicted depth information corresponding to the third sample infrared image is determined. The second predicted depth information is used as a positive sample and input into the second discriminant network to obtain the fifth discriminant result. The third predicted depth information is used as a negative sample and input into the second discriminant network to obtain the sixth discriminant result. The third prediction loss is calculated based on at least the fifth and sixth discriminant results. The target depth estimation model and the second discriminant network are then subjected to adversarial training based on the third prediction loss.

[0010] For example, the depth estimation model to be transferred and the target depth estimation model each include a feature extraction module and a depth prediction module connected in sequence. Inputting a second sample infrared image into the depth estimation model to be transferred to obtain the corresponding second prediction result includes: inputting the second sample infrared image into the depth estimation model to be transferred to obtain the first infrared feature output by the feature extraction module of the depth estimation model to be transferred and the second prediction result output by the depth prediction module of the depth estimation model to be transferred; inputting a third sample infrared image into the target depth estimation model to obtain the corresponding third prediction result includes: inputting the third sample infrared image into the target depth estimation model to obtain the second infrared feature output by the feature extraction module of the target depth estimation model and the third prediction result output by the depth prediction module of the target depth estimation model; at least based on Before calculating the third prediction loss based on the fifth and sixth discrimination results, the second training operation further includes: inputting the first infrared feature as a positive sample into the second discrimination network to obtain the seventh discrimination result; inputting the second infrared feature as a negative sample into the second discrimination network to obtain the eighth discrimination result; calculating the third prediction loss based at least on the fifth and sixth discrimination results includes: calculating the third prediction loss based on the fifth, sixth, seventh, and eighth discrimination results; wherein, during the adversarial training of the target depth estimation model and the second discrimination network based on the third prediction loss, the target parameters in the target depth estimation model are optimized, and the remaining parameters in the target depth estimation model other than the target parameters are fixed, and the target parameters are at least some of the parameters in the feature extraction module of the target depth estimation model.

[0011] For example, before calculating the second prediction loss based on the labeled depth information and the first predicted depth information, the first training operation further includes: inputting the synthesized infrared image into the image segmentation network to obtain image segmentation results; determining effective regions and invalid regions based on the image segmentation results; calculating the second prediction loss based on the labeled depth information and the first predicted depth information includes: for any pixel in the effective region, using the depth value corresponding to that pixel in the labeled depth information as the target value, and using the depth value corresponding to that pixel in the first predicted depth information as the predicted value, and calculating the first loss; for any pixel in the invalid region, using a specific depth value as the target value, and using the depth value corresponding to that pixel in the first predicted depth information as the predicted value, and calculating the second loss, wherein the disparity value corresponding to the specific depth value is 0; and calculating the second prediction loss based on the first loss and the second loss.

[0012] For example, the method further includes: acquiring an RGB image to be processed, wherein the RGB image to be processed and an infrared image to be processed are acquired for the same target scene; spatially aligning the RGB image to be processed and the infrared image to be processed; converting the aligned RGB image to be processed into an infrared image to obtain a converted infrared image; inputting the converted infrared image or a new infrared image into a target depth estimation model to obtain a corresponding second depth estimation result, wherein the new infrared image is generated based on the converted infrared image, and the second depth estimation result is information related to the depth of each pixel in the RGB image to be processed; determining the second depth information corresponding to the RGB image to be processed based on the second depth estimation result; and fusing the first depth information and the second depth information pixel by pixel to obtain first comprehensive depth information.

[0013] For example, the method further includes: extracting one or more image patches from the infrared image to be processed, wherein the one or more image patches correspond one-to-one with one or more different scales, and each image patch contains the center point of the infrared image to be processed; for each of the one or more image patches, inputting the image patch into a target depth estimation model to obtain a corresponding sub-depth estimation result, wherein the sub-depth estimation result is information related to the depth of each pixel in the image patch; determining the sub-depth information corresponding to the sub-patch based on the sub-depth estimation result; and fusing the first depth information with the sub-depth information corresponding to the one or more image patches on a pixel-by-pixel basis to obtain second comprehensive depth information.

[0014] According to another aspect of this application, an electronic device is provided, including a processor and a memory, wherein the memory stores computer program instructions, which are executed by the processor to perform the aforementioned depth estimation method based on infrared images.

[0015] According to another aspect of this application, a storage medium is provided on which program instructions are stored, wherein the program instructions are used to execute the above-described depth estimation method based on infrared images when executed.

[0016] According to another aspect of this application, a computer program product is provided, the computer program product comprising a computer program, wherein the computer program, when running, is used to execute the above-described depth estimation method based on infrared images.

[0017] According to the infrared image-based depth estimation method, electronic device, storage medium, and computer program product of this application, the target depth estimation model used can be trained using synthetic infrared images, which are generated based on labeled RGB images. Since a large number of publicly available datasets with depth annotations are available for monocular RGB images, the scheme of generating synthetic infrared images based on RGB images helps to quickly generate large-scale labeled synthetic infrared datasets, which has significant application value for training infrared depth estimation models. Because the target depth estimation model used in the infrared image-based depth estimation method of this application is trained using the aforementioned synthetic infrared images, there is sufficient training data available, and the performance of the target depth estimation model is robust. Therefore, the accuracy of depth estimation based on this target depth estimation model is also high. Attached Figure Description

[0018] The above and other objects, features, and advantages of this application will become more apparent from the more detailed description of the embodiments of this application in conjunction with the accompanying drawings. The accompanying drawings are used to provide a further understanding of the embodiments of this application and form part of the specification. They are used together with the embodiments of this application to explain this application and do not constitute a limitation thereof. In the accompanying drawings, the same reference numerals generally represent the same components or steps.

[0019] Figure 1 A schematic block diagram of an example electronic device for implementing the infrared image-based depth estimation method and apparatus according to embodiments of this application is shown.

[0020] Figure 2 A schematic flowchart of a depth estimation method based on infrared images according to an embodiment of this application is shown;

[0021] Figure 3 A schematic block diagram of a style transfer network according to an embodiment of this application is shown;

[0022] Figure 4 A schematic diagram illustrating the training of a target depth estimation model according to an embodiment of this application is shown;

[0023] Figure 5 A schematic block diagram of an infrared image-based depth estimation apparatus according to an embodiment of this application is shown; and

[0024] Figure 6 A schematic block diagram of an electronic device according to an embodiment of this application is shown. Detailed Implementation

[0025] In recent years, significant progress has been made in research on technologies based on artificial intelligence, such as computer vision, deep learning, machine learning, image processing, and image recognition. Artificial intelligence (AI) is an emerging science and technology that studies and develops theories, methods, technologies, and testing systems to simulate and extend human intelligence. AI is a comprehensive discipline involving numerous technologies, including chips, big data, cloud computing, the Internet of Things, distributed storage, deep learning, machine learning, and neural networks. Computer vision, as an important branch of AI, specifically enables machines to recognize the world. Computer vision technologies typically include facial recognition, image processing, fingerprint recognition and anti-counterfeiting verification, biometric recognition, face detection, pedestrian detection, object detection, image processing, image recognition, image semantic understanding, image retrieval, text recognition, video processing, video content recognition, 3D reconstruction, virtual reality, augmented reality, simultaneous localization and mapping (SLAM), computational photography, and robot navigation and localization. With the research and advancement of artificial intelligence technology, this technology has been tested in numerous fields, such as urban management, traffic management, building management, park management, facial recognition access control, facial recognition attendance, logistics management, warehouse management, robotics, intelligent marketing, computational photography, mobile imaging, cloud services, smart homes, wearable devices, autonomous driving, autonomous driving, smart healthcare, facial recognition payment, facial recognition unlocking, fingerprint unlocking, identity verification, smart screens, smart TVs, cameras, mobile internet, live streaming, beauty filters, cosmetics, medical aesthetics, and intelligent temperature measurement.

[0026] To make the objectives, technical solutions, and advantages of this application more apparent, exemplary embodiments according to this application will be described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are merely some embodiments of this application, and not all embodiments of this application. It should be understood that this application is not limited to the exemplary embodiments described herein. Based on the embodiments of this application described herein, all other embodiments obtained by those skilled in the art without inventive effort should fall within the protection scope of this application.

[0027] This application provides a depth estimation method based on infrared images, an electronic device, a storage medium, and a computer program product. According to the infrared image-based depth estimation method of this application, the target depth estimation model can be obtained by training using synthetic infrared images, which are generated based on labeled RGB images. The infrared image-based depth estimation method of this application can be applied to any technical field requiring depth estimation, including but not limited to: robot navigation and localization, autonomous driving, SLAM, etc.

[0028] First, refer to Figure 1This describes an example electronic device 100 for implementing the infrared image-based depth estimation method and apparatus according to embodiments of this application.

[0029] like Figure 1 As shown, the electronic device 100 includes one or more processors 102 and one or more storage devices 104. Optionally, the electronic device 100 may also include an input device 106, an output device 108, and an image capturing device 110, these components being interconnected via a bus system 112 and / or other forms of connection mechanisms (not shown). It should be noted that... Figure 1 The components and structure of the electronic device 100 shown are merely exemplary and not limiting; the electronic device may also have other components and structures as needed.

[0030] The processor 102 may be implemented in at least one of the following hardware forms: digital signal processor (DSP), field-programmable gate array (FPGA), programmable logic array (PLA), and microprocessor. The processor 102 may be one or a combination of several of the following: central processing unit (CPU), graphics processing unit (GPU), application-specific integrated circuit (ASIC), or other processing units with data processing capabilities and / or instruction execution capabilities. It may also control other components in the electronic device 100 to perform the desired functions.

[0031] The storage device 104 may include one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may include, for example, random access memory (RAM) and / or cache memory. The non-volatile memory may include, for example, read-only memory (ROM), hard disk, flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage medium, and the processor 102 may execute the program instructions to implement the client functions (implemented by the processor) in the embodiments of this application described below, and / or other desired functions. Various test programs and various data may also be stored in the computer-readable storage medium, such as various data used and / or generated by the test programs.

[0032] The input device 106 may be a device used by a user to input commands, and may include one or more of the following: keyboard, mouse, microphone, and touch screen.

[0033] The output device 108 can output various information (e.g., images and / or sound) to the outside (e.g., a user), and may include one or more of a display, speaker, etc. Optionally, the input device 106 and the output device 108 can be integrated together and implemented using the same interactive device (e.g., a touch screen).

[0034] The image capturing device 110 can capture images and store the captured images in the storage device 104 for use by other components. The image capturing device 110 can be a standalone camera or a camera in a mobile terminal, etc. It should be understood that the image capturing device 110 is only an example, and the electronic device 100 may not include the image capturing device 110. In this case, other devices with image capturing capabilities can be used to capture images and send the captured images to the electronic device 100.

[0035] For example, an example electronic device for implementing the infrared image-based depth estimation method and apparatus according to embodiments of this application can be implemented on devices such as personal computers, terminal devices, time and attendance machines, panel displays, cameras, or remote servers. Terminal devices include, but are not limited to, tablet computers, mobile phones, PDAs (Personal Digital Assistants), touchscreen all-in-one machines, wearable devices, etc.

[0036] Below, we will refer to Figure 2 This application describes a depth estimation method based on infrared images according to embodiments of the present application. Figure 2 A schematic flowchart of an infrared image-based depth estimation method 200 according to an embodiment of this application is shown. Figure 2 As shown, the depth estimation method 200 based on infrared images includes steps S210, S220 and S230.

[0037] Step S210: Obtain the infrared image to be processed.

[0038] The infrared image to be processed is an image acquired using an infrared image acquisition device. For example, the infrared image to be processed can be any type of image containing a target scene. The target scene can contain any object, including but not limited to: pedestrians, animals, vehicles, scenery, buildings, etc. The infrared image to be processed can be a still image or any video frame from a dynamic video. The infrared image to be processed can be a raw image acquired by an image acquisition device (such as a standalone camera or a camera in a mobile terminal), or an image obtained after preprocessing the raw image (such as digitization, normalization, smoothing, etc.).

[0039] The infrared image to be processed can be sent to the electronic device 100 by an external device (e.g., a cloud server) for processing by the processor 102 of the electronic device 100, or it can be acquired by an image acquisition device 110 (e.g., a camera) included in the electronic device 100 and transmitted to the processor 102 for processing, or it can be stored by a storage device 104 included in the electronic device 100 and transmitted to the processor 102 for processing.

[0040] Step S220: Input the infrared image to be processed into the target depth estimation model to obtain the corresponding first depth estimation result. The first depth estimation result is the depth-related information of each pixel in the infrared image to be processed. The target depth estimation model is trained using synthetic infrared training data. The synthetic infrared training data includes synthetic infrared images and labeled depth information. The synthetic infrared images are generated based on labeled RGB images, and the labeled depth information is the depth information corresponding to the labeled RGB images.

[0041] It should be noted that the terms "first," "second," and "third" used in this article are for distinguishing purposes only and do not indicate order or other special meanings.

[0042] For example, the target depth estimation model can be implemented using any algorithm capable of depth estimation, such as a neural network model. In one embodiment, the target depth estimation model can be a U-Net network model or any network model based on a Transformer structure, such as a NeuralWindow Fully-connected Conditional Random Field (NeW CRFs) network model. Inputting the acquired infrared image to be processed into the target depth estimation model yields the corresponding first depth estimation result.

[0043] The first depth estimation result is information related to the depth of each pixel in the infrared image to be processed. For example, the first depth estimation result can be depth information or disparity information corresponding to the infrared image to be processed. Depth information can include the depth values ​​corresponding to each pixel in the infrared image to be processed, and disparity information can include the disparity values ​​corresponding to each pixel in the infrared image to be processed. That is, the depth information can be a depth image of the same size as the image to be processed. The pixel value of any pixel in the depth image is used to represent the depth value corresponding to a pixel at the same location in the image to be processed. Similarly, the disparity information can be a disparity image of the same size as the image to be processed. The pixel value of any pixel in the disparity image is used to represent the disparity value corresponding to a pixel at the same location in the image to be processed.

[0044] The target depth estimation model can be trained directly or indirectly using synthetic infrared training data. In one example, the synthetic infrared training data can be directly used as the training data for the target depth estimation model, trained using, for example, conventional model training methods (such as the first training operation described below). In another example, the synthetic infrared training data can be used as the training data for another depth estimation model (which may be referred to as the depth estimation model to be transferred), trained using, for example, conventional model training methods (such as the first training operation described below). Subsequently, the target depth estimation model can be further trained using the depth estimation model to be transferred through the domain transfer algorithm described below.

[0045] Step S230: Determine the first depth estimation result as the first depth information corresponding to the infrared image to be processed, or determine the first depth information based on the first depth estimation result.

[0046] For example, if the obtained first depth estimation result is depth information, it can be directly determined as the first depth information corresponding to the infrared image to be processed. If the obtained first depth estimation result is disparity information, the value obtained by taking the reciprocal of the disparity value corresponding to each pixel in the infrared image to be processed can be used as the depth value corresponding to each pixel to obtain the first depth information.

[0047] According to the aforementioned infrared image-based depth estimation method, the target depth estimation model can be trained using synthetic infrared images generated from labeled RGB images. Since a large number of publicly available datasets with depth annotations are available for monocular RGB images, the scheme of generating synthetic infrared images from RGB images helps to quickly generate large-scale labeled synthetic infrared datasets, which has significant application value for training infrared depth estimation models. Because the target depth estimation model used in the infrared image-based depth estimation method according to the embodiments of this application is trained using the aforementioned synthetic infrared images, there is sufficient training data available, and the performance of the target depth estimation model is robust. Therefore, the accuracy of depth estimation based on this target depth estimation model is also high.

[0048] For example, the infrared image-based depth estimation method according to the embodiments of this application can be implemented in a device, apparatus or system having a memory and a processor.

[0049] The infrared image-based depth estimation method according to the embodiments of this application can be deployed at the image acquisition end, for example, at a personal terminal or server with image acquisition function.

[0050] Alternatively, the infrared image-based depth estimation method according to embodiments of this application can also be deployed distributedly at the server (or cloud) and client. For example, an infrared image to be processed can be acquired at the client, and the client can transmit the infrared image to be processed to the server (or cloud), whereby the server (or cloud) estimates the depth information of the infrared image to be processed.

[0051] For example, a synthetic infrared image is generated by: obtaining a labeled RGB image; inputting the labeled RGB image into a style transfer network to obtain a synthetic infrared image.

[0052] In one embodiment, the style transfer network can have any network structure capable of performing style transfer. For example, the style transfer network can be an unsupervised image-to-image translation network (UNIT). Exemplarily, and not limitingly, labeled RGB images can be obtained from the Driving Stereo or the large-scale autonomous driving domain dataset (KITTI). Both Driving Stereo and KITI datasets contain depth-annotated data (referred to herein as labeled depth information). Of course, any other suitable RGB image can also be used as a labeled RGB image to generate an infrared image. Inputting the labeled RGB image into the style transfer network yields the transferred infrared image, i.e., the synthesized infrared image.

[0053] According to the above technical solution, a style transfer network is used to automatically generate synthetic infrared images based on RGB images for training a target depth estimation model. This solution can generate a large number of infrared datasets very conveniently.

[0054] For example, the style transfer network is trained as follows: A sample RGB image and a first sample infrared image are input into the style transfer network to obtain a first predicted RGB image and a first predicted infrared image corresponding to the sample RGB image, and a second predicted RGB image and a second predicted infrared image corresponding to the first sample infrared image; the first predicted RGB image is input as a positive sample into a first discriminant network to obtain a first discriminant result; the second predicted RGB image is input as a negative sample into the first discriminant network to obtain a second discriminant result; the first predicted infrared image is input as a negative sample into the first discriminant network to obtain a third discriminant result; the second predicted infrared image is input as a positive sample into the first discriminant network to obtain a fourth discriminant result; a first prediction loss is calculated based on the first, second, third, and fourth discriminant results; and adversarial training is performed on the style transfer network and the first discriminant network based on the first prediction loss.

[0055] The sample RGB image and the first sample infrared image may contain the same target scene or different target scenes. In other words, the image content contained in the sample RGB image and the first sample infrared image may be the same or different.

[0056] In one embodiment, the acquisition method of the sample RGB image is similar to step S210, and will not be repeated here for simplicity. Inputting the sample RGB image into the style transfer network yields a first predicted RGB image and a first predicted infrared image. Similarly, inputting the first sample infrared image into the style transfer network yields a second predicted RGB image and a second predicted infrared image. Subsequently, a discriminant network can determine the true / false nature of each image generated by the style transfer network. This training method is a Generative Adversarial Network (GAN) training method. The style transfer network can be considered as the generator G in a GAN, and the first discriminant network can be considered as the discriminator D in a GAN. For positive samples, the discriminant network needs to classify them as true as possible, while for negative samples, it needs to classify them as false as possible. The style transfer network, as the generator G, needs to classify its generated negative samples as true as possible, that is, to classify the first predicted infrared image generated based on the sample RGB image as true as possible, and to classify the second predicted RGB image generated based on the sample infrared image as true as possible.

[0057] The first discriminant network can be implemented using a single discriminator or multiple discriminators. In one example, the first discriminant network may include a first discriminator and a second discriminator. The first and second discriminators can be used to determine the authenticity of RGB and infrared images, respectively. For example, the first predicted RGB image and the second predicted RGB image can be input into the first discriminator to obtain their respective first and second discrimination results. The first and second discrimination results can be represented by values ​​between 0 and 1. Furthermore, the first predicted infrared image and the second predicted infrared image can be input into the second discriminator to obtain their respective third and fourth discrimination results. The third and fourth discrimination results can also be represented by values ​​between 0 and 1.

[0058] Subsequently, the first prediction loss can be calculated based on the above four discrimination results. For example, the first, second, third, and fourth discrimination results can be substituted into the GAN's maximum-minimum loss function to obtain the first prediction loss. Based on the first prediction loss, adversarial training can be performed on the style transfer network and the first discrimination network. During adversarial training, the parameters in the style transfer network and the first discrimination network can be optimized using backpropagation and gradient descent algorithms. It can be understood that during adversarial training, the parameters in the first discrimination network and the style transfer network can be optimized alternately. Those skilled in the art will understand this training method of GANs, and it will not be elaborated upon in this paper.

[0059] For example, the Driving Stereo and KITTI datasets can be used to train a style transfer network. For instance, at least a portion of the RGB images from Driving Stereo and / or at least a portion of the RGB images from the KITTI dataset can be used as sample RGB images for training the style transfer network. More preferably, at least a portion of the RGB images from both Driving Stereo and KITTI datasets can be used as sample RGB images for training. Driving Stereo contains RGB images of natural scenes, which can better represent real-world scenes, thus enabling the trained style transfer network to have high generalization ability. However, the depth images corresponding to Driving Stereo are relatively sparse, making it difficult to effectively train the generation of dense depth images. KITTI has dense depth images, but its RGB images are virtual images, which differ significantly from real-world scenes, resulting in poor generalization. Therefore, jointly training these two datasets can ensure that the style transfer network has high generalization ability while also possessing dense depth images.

[0060] According to the above technical solution, a style transfer network is trained using sample RGB images and a first sample infrared image. The style transfer network can predict the RGB and infrared images corresponding to the sample RGB images, as well as the RGB and infrared images corresponding to the first sample infrared image. Through training, the performance of the predicted infrared image generated based on the sample RGB image is made close to that of the sample infrared image, and vice versa. This training method allows the style transfer network to learn a common implicit representation for RGB and infrared images, enabling it to predict the desired image regardless of whether the input is an RGB or infrared image.

[0061] For example, inputting a sample RGB image and a first sample infrared image into a style transfer network to obtain a first predicted RGB image and a first predicted infrared image corresponding to the sample RGB image, and a second predicted RGB image and a second predicted infrared image corresponding to the first sample infrared image, includes: performing the following operations through the style transfer network: performing a first encoding operation on the sample RGB image to obtain a first encoded feature; performing a second encoding operation on the first sample infrared image to obtain a second encoded feature; merging the first encoded feature and the second encoded feature together to obtain a merged feature; performing a first decoding operation on the merged feature to obtain the first predicted RGB image and the second predicted RGB image; and performing a second decoding operation on the merged feature to obtain the first predicted infrared image and the second predicted infrared image.

[0062] Figure 3 A schematic diagram of a style transfer network according to an embodiment of this application is shown. Figure 3 As shown, a style transfer network can include an encoding module, a feature merging module, and a decoding module.

[0063] First, the sample RGB image x1 can be input into encoding module E1. The first encoding operation of encoding module E1 yields the first encoded feature. Similarly, the first sample infrared image x2 can be input into encoding module E2. The second encoding operation of encoding module E2 yields the second encoded feature. The first and second encoded features are then input into the feature merging module, where the two features are merged to obtain the merged feature z. Feature merging can combine the channels of the two features together. The merged feature is then input into decoding module G1, where the first decoding operation yields the first predicted RGB image. Second predicted RGB image The merged features are input into the decoding module G2, and after a second decoding operation, the first predicted infrared image can be obtained. Second predicted infrared image

[0064] Subsequently, the first predicted RGB image and the second predicted RGB image can be input into the first discriminator D1, and the first predicted infrared image and the second predicted infrared image can be input into the second discriminator D2, respectively, to obtain the corresponding first discrimination result R1, second discrimination result R2, third discrimination result R3 and fourth discrimination result R4, and the loss can be calculated for optimization.

[0065] According to the above technical solution, after the sample RGB image and the first sample infrared image are input into the style transfer network, they can be encoded and merged to obtain a shared feature space that can be used to simultaneously represent RGB images and infrared images, which facilitates style transfer between RGB images and infrared images.

[0066] As mentioned above, the target depth estimation model can be trained directly or indirectly using synthetic infrared training data. Before training, the target depth estimation model has initial parameters (including weights and biases, etc.). By training the target depth estimation model, the parameters in the model can be optimized.

[0067] In one example, the target depth estimation model can be trained using a traditional model training method (referred to as the first training operation). The trained depth estimation model can then be directly used for depth estimation, for example, by treating it as the target depth estimation model and performing step S220 as described above. The target depth estimation model can be trained using a synthetic infrared dataset containing the aforementioned synthetic infrared images and applied to the depth estimation of any infrared image to be processed. This training method is simple, convenient, and relatively efficient.

[0068] In one example, the target depth estimation model can be trained using an unsupervised domain transfer algorithm. A domain can be understood as the acquisition environment. The acquisition environment can include one or more of the following environmental information: the image acquisition device used for acquisition; the scene targeted for acquisition; the acquisition time, etc. One or more predefined environmental information can be defined as needed; when these predefined environmental information differ, the acquisition environment is considered different. Different domains correspond to different acquisition environments and also to different datasets. After training a depth estimation model based on a dataset in a certain acquisition environment (e.g., infrared images acquired by camera A), directly applying it to depth estimation of an infrared image to be processed in another acquisition environment (e.g., infrared images acquired by camera B) can easily lead to performance degradation. A synthetic infrared dataset can be considered as one domain, and an infrared image in a certain real acquisition environment (the target acquisition environment) can be considered as another domain. Thus, after training a depth estimation model (referred to as the depth estimation model to be transferred in this paper) based on a synthetic infrared dataset, directly using it as the target depth estimation model for depth estimation in a real acquisition scene will also lead to performance degradation. To address this issue, this application provides an adaptive domain transfer algorithm for updating model parameters, which can achieve accurate infrared depth estimation under different acquisition environments. The depth estimation model trained using the unsupervised domain transfer algorithm is the target depth estimation model. Specifically, any infrared image (such as the synthetic infrared image mentioned above) can be used as the second sample infrared image, and a real infrared image under the target acquisition environment can be used as the third sample image. The following second training operation is then performed to improve the prediction performance of the target depth estimation model on real infrared images. When using a synthetic infrared image as the second sample infrared image, the second sample infrared image can be the same as or different from the synthetic infrared image used in the first training operation. Furthermore, the second sample infrared image and the third sample infrared image can contain the same target scene or different target scenes. That is, the image content contained in the second sample infrared image and the third sample infrared image can be the same or different. The above training method can adaptively update the parameters of the depth estimation model, enabling it to achieve good depth estimation results for infrared images under different acquisition environments.

[0069] The following describes exemplary implementations of the first and second training operations of the target depth estimation model.

[0070] The first training operation may include: acquiring a synthetic infrared image and labeled depth information; inputting the synthetic infrared image into a target depth estimation model to obtain a corresponding first prediction estimation result, wherein the first prediction estimation result is information related to the depth of each pixel in the synthetic infrared image; determining the first predicted depth information corresponding to the synthetic infrared image based on the first prediction estimation result; calculating a second prediction loss based on the labeled depth information and the first predicted depth information; and optimizing the parameters in the target depth estimation model based on the second prediction loss.

[0071] For example, the first prediction result obtained by inputting a synthetic infrared image into a target depth estimation model can be the depth information or disparity information corresponding to the synthetic infrared image. The meanings of depth information and disparity information have been described above and will not be repeated here.

[0072] For example, if the first prediction estimation result is the depth information corresponding to the synthesized infrared image, this first prediction estimation result can be directly used as the first predicted depth information corresponding to the synthesized infrared image. If the first prediction estimation result is the disparity information corresponding to the synthesized infrared image, the reciprocal of each disparity value can be taken. The result obtained after taking the reciprocal can be used as the first predicted depth information corresponding to the synthesized infrared image.

[0073] A first prediction loss can be calculated based on the labeled depth information and the first predicted depth information. For example, the labeled depth information can be used as the target value, the first predicted depth information as the predicted value, and both can be substituted into the target loss function to calculate the loss value, which is then used as the first prediction loss. The target loss function can be set to any suitable type of loss function as needed, such as the loss function used in conventional monocular depth estimation algorithms. For example, based on the first prediction loss, all or some parameters in the target depth estimation model can be optimized through backpropagation and gradient descent algorithms. The above optimization can be performed iteratively until the loss of the target depth estimation model converges. Those skilled in the art will understand the meaning of loss convergence, which will not be elaborated upon here.

[0074] In one example, the target depth estimation model is U-Net. After acquiring a dataset containing one or more synthetic infrared images, U-Net can be trained on this dataset. The U-Net encoder takes the synthetic infrared images as input, extracts features to generate a low-dimensional image encoding, and then the decoder continuously upsamples the image encoding to generate a predicted disparity image (i.e., the first predicted estimation result). The last layer of the decoder uses a Sigmoid function, ensuring that the output disparity value is between 0 and 1 and cannot be 0. For each pixel in the predicted disparity image, the reciprocal of the pixel value is taken to obtain the predicted depth value, thus obtaining the predicted depth image (i.e., the first predicted depth information), which is used to optimize U-Net. Since the input to the target depth estimation model is a monocular image and lacks scale information, the SILog function can be used as a scale-independent loss function to calculate the error between the labeled depth information and the first predicted depth information. The SILog function is calculated as follows: in, This is the depth value corresponding to the i-th pixel in the first predicted depth information. To label the depth value corresponding to the i-th pixel in the depth information, n is the total number of pixels in the synthesized infrared image.

[0075] The second training operation may include: acquiring a second sample infrared image and a third sample infrared image; initializing the weights of the target depth estimation model using the weights of the depth estimation model to be transferred, wherein the network structures of the depth estimation model to be transferred and the target depth estimation model are the same, and the depth estimation model to be transferred is trained based on synthetic infrared training data; inputting the second sample infrared image into the depth estimation model to be transferred to obtain the corresponding second prediction estimation result, wherein the second prediction estimation result is information related to the depth of each pixel in the second sample infrared image; determining the second predicted depth information corresponding to the second sample infrared image based on the second prediction estimation result; inputting the third sample infrared image into the target depth estimation model to obtain the corresponding third prediction estimation result, wherein the third prediction estimation result is information related to the depth of each pixel in the third sample infrared image; determining the third predicted depth information corresponding to the third sample infrared image based on the third prediction estimation result; inputting the second predicted depth information as a positive sample into the second discriminant network to obtain a fifth discriminant result; inputting the third predicted depth information as a negative sample into the second discriminant network to obtain a sixth discriminant result; calculating the third prediction loss based at least on the fifth and sixth discriminant results; and performing adversarial training on the target depth estimation model and the second discriminant network based on the third prediction loss.

[0076] In one example, the method for obtaining the second sample infrared image and / or the third sample infrared image is similar to step S210, which has been described in detail above and will not be repeated here for the sake of brevity. In another example, the second sample infrared image and / or the third sample infrared image can be a synthetic infrared image generated based on an RGB image, and the generation method can be referred to the description above.

[0077] For example, the weights of the depth estimation model to be transferred can be assigned to the corresponding positions of the target depth estimation model to initialize the weights of the target depth estimation model.

[0078] The second and third prediction estimation results are similar in meaning, form, and acquisition method to the first prediction estimation results. The second and third prediction depth information are similar in meaning, form, and acquisition method to the first prediction depth information. These have been described in detail above and will not be repeated here for the sake of brevity.

[0079] The second training operation employs a GAN training method. The target depth estimation model can be viewed as the generator G in a GAN, and the second discriminator network as the discriminator D. The second predicted depth information obtained through the depth estimation model to be transferred is used as positive samples, while the third predicted depth information obtained through the target depth estimation model is used as negative samples. For positive samples, the second discriminator network needs to classify them as true as possible, and for negative samples, it needs to classify them as false as possible. The target depth estimation model, acting as the generator G, needs to classify its corresponding negative samples as true as possible, thus ensuring that the third predicted depth information is classified as true as much as possible. Through adversarial training between the target depth estimation model and the second discriminator network, the prediction performance of the target depth estimation model in the new domain (e.g., real infrared images) can be made close to the performance of the depth estimation model to be transferred in the initial domain (e.g., synthetic infrared images).

[0080] The second discriminant network can be implemented using a single discriminator or multiple discriminators. For example, the second discriminant network may include a third discriminator. The third discriminator can be used to determine the authenticity of the second and third predicted depth information. For example, the second discriminant network may also include a fourth discriminator. The fourth discriminator can be used to determine the authenticity of the first and second infrared features. For example, the fifth and sixth discrimination results can be represented by values ​​between 0 and 1.

[0081] In one example, the third prediction loss can be calculated based solely on the fifth and sixth discrimination results. For instance, the fifth and sixth discrimination results can be substituted into the GAN's max-min loss function to obtain the third prediction loss. In another example, the third prediction loss can be further determined by combining other information with the fifth and sixth discrimination results. An exemplary scheme for this combination is described below.

[0082] Based on the third prediction loss, adversarial training can be performed on the target depth estimation model and the second discriminant network. During adversarial training, backpropagation and gradient descent algorithms can be used to optimize the parameters in the target depth estimation model and the second discriminant network. This optimization can target all or some of the parameters in the target depth estimation model.

[0083] For example, the depth estimation model to be transferred and the target depth estimation model each include a feature extraction module and a depth prediction module connected in sequence. Inputting a second sample infrared image into the depth estimation model to obtain a corresponding second prediction result may include: inputting the second sample infrared image into the depth estimation model to obtain a first infrared feature output by the feature extraction module of the depth estimation model to be transferred and a second prediction result output by the depth prediction module of the depth estimation model to be transferred; inputting a third sample infrared image into the target depth estimation model to obtain a corresponding third prediction result may include: inputting the third sample infrared image into the target depth estimation model to obtain a second infrared feature output by the feature extraction module of the target depth estimation model and a third prediction result output by the depth prediction module of the target depth estimation model; in at least Before calculating the third prediction loss based on the fifth and sixth discrimination results, the second training operation may further include: inputting the first infrared feature as a positive sample into the second discrimination network to obtain the seventh discrimination result; inputting the second infrared feature as a negative sample into the second discrimination network to obtain the eighth discrimination result; calculating the third prediction loss based at least on the fifth and sixth discrimination results includes: calculating the third prediction loss based on the fifth, sixth, seventh, and eighth discrimination results; wherein, during the adversarial training of the target depth estimation model and the second discrimination network based on the third prediction loss, the target parameters in the target depth estimation model are optimized, and the remaining parameters in the target depth estimation model other than the target parameters are fixed, and the target parameters are at least some of the parameters in the feature extraction module of the target depth estimation model.

[0084] Figure 4 A schematic diagram illustrating the training of a target depth estimation model according to an embodiment of this application is shown. Figure 4 As shown, the depth estimation model to be transferred may include a feature extraction module M. sThe target depth estimation model may include a feature extraction module M, along with a depth prediction module T. t And depth prediction module T. Feature extraction module M of the depth estimation model to be transferred. s Feature extraction module M of the target depth estimation model t Each module can include convolutional layers (Conv), pooling layers (Pool), and multiple residual modules (Res-2 block, Res-3 block, Res-4 block, and Res-5 block). The feature extraction module M of the depth estimation model to be transferred... s Feature extraction module M of the target depth estimation model t The network structure is the same, only the parameter sizes may differ. Before training the target depth estimation model, the feature extraction module M can be... s The parameters are assigned to the feature extraction module M. t Optionally, the depth prediction module of the depth estimation model to be transferred and the depth prediction module of the target depth estimation model can be exactly the same depth prediction module T, that is, they share parameters. For example, the depth prediction module T may include convolutional layers (Conv), up-projection layers, etc. It should be noted that... Figure 4 The specific network structures of the depth estimation model to be transferred and the target depth estimation model shown are examples and not limitations of this application. For example, the number of convolutional layers, pooling layers and residual modules contained in the depth estimation model to be transferred and the target depth estimation model can be arbitrary, the order of the layers and modules can also be arbitrary, and other network layers can be optionally included.

[0085] In one embodiment, the second sample infrared image X can be used. s Input the depth estimation model M to be transferred s The feature extraction module M of the depth estimation model to be transferred is obtained. s The first infrared feature output L s The second prediction depth information can be determined based on the intermediate features (i.e., the intermediate features) and the second prediction estimation result output by the depth prediction module T. The first infrared feature L... s It can be any type of feature, which depends on the design of the depth estimation model. For example, if the second sample infrared image is a building image, the first infrared feature L... s It can be the edge contour features of a building.

[0086] Similarly, the third sample infrared image X t By inputting the target depth estimation model, we can obtain the feature extraction module M of the target depth estimation model. t The output second infrared feature L tThe third prediction estimate, derived from the intermediate features (i.e., the depth prediction module T) and the third prediction estimate, can be used to determine the third predicted depth information. The second infrared feature L... t It can be related to the first infrared feature L s Features of the same type.

[0087] Based on the second and third prediction information, the fifth discrimination result R5 and the sixth discrimination result R6 can be calculated one-to-one, as described above. Furthermore, training the target depth estimation model may also include the following steps: The first infrared feature L... s Second infrared feature L t The results are input into the fourth discriminator to obtain the seventh and eighth discrimination results R7 and R8, respectively. These results can also be represented by values ​​between 0 and 1. The fifth, sixth, seventh, and eighth discrimination results can be substituted into the GAN's maximum-minimum loss function to calculate the third prediction loss. Based on this third prediction loss, adversarial training can be performed. During adversarial training, the feature extraction module M of the target depth estimation model can be improved using backpropagation and gradient descent algorithms. t Optimize all or some of the parameters in the model. If it is the feature extraction module M of the target depth estimation model... t Some parameters in the model can be optimized while the rest remain unchanged. In one example, the parameters of the depth prediction module T can be fixed, while the parameters of the feature extraction module M in the target depth estimation model can be optimized. t All parameters in the module are optimized. In another example, the depth prediction module T and the feature extraction module M can be optimized. t The parameters in some modules are fixed, and the remaining parameters in the target depth estimation model are optimized. The aforementioned feature extraction module M... t Some modules in it can be feature extraction modules M t The deepest module in a system, such as the Res-5 block. This can be understood as the feature extraction module M... t In this model, along the data transmission direction, modules that appear earlier are shallower, and modules that appear later are deeper. Considering that deeper features are generally task-specific and have lower transferability, the deepest module can be used to align the outputs of the Ms and Mt branches.

[0088] According to the above technical solution, the third prediction loss is calculated using the fifth, sixth, seventh, and eighth discrimination results, and then the target parameters in the target depth estimation model are updated. This allows for the calculation of the prediction loss by combining multiple discrimination results, resulting in a more accurate loss calculation. Optimizing the target depth estimation model based on this prediction loss can improve its performance.

[0089] For example, before calculating the second prediction loss based on the labeled depth information and the first predicted depth information, the first training operation further includes: inputting the synthesized infrared image into the image segmentation network to obtain image segmentation results; determining effective regions and invalid regions based on the image segmentation results; calculating the second prediction loss based on the labeled depth information and the first predicted depth information includes: for any pixel in the effective region, using the depth value corresponding to that pixel in the labeled depth information as the target value, and using the depth value corresponding to that pixel in the first predicted depth information as the predicted value, and calculating the first loss; for any pixel in the invalid region, using a specific depth value as the target value, and using the depth value corresponding to that pixel in the first predicted depth information as the predicted value, and calculating the second loss, wherein the disparity value corresponding to the specific depth value is 0; and calculating the second prediction loss based on the first loss and the second loss.

[0090] In one example, when calculating the second prediction loss, the same loss function can be applied uniformly to all pixels in the image. In another example, when calculating the second prediction loss, the image can be divided into valid and invalid regions, and different loss functions can be applied to the two regions respectively.

[0091] For example, a synthesized infrared image can be input into an image segmentation network to obtain the image segmentation result. The image segmentation network can be any network capable of image segmentation, such as the U-Net network or the encoder-decoder-based polyp segmentation network (HarDNet-MSEG). For the synthesized infrared image, valid and invalid regions can be determined according to requirements. The region division can be arbitrary. For example, in a landscape image, the sky region can be identified as an invalid region, and the regions outside the sky region can be identified as valid regions. Based on the image segmentation result, the valid and invalid regions of the synthesized infrared image can be determined.

[0092] Based on the labeled depth information obtained above, the depth values ​​of the pixels corresponding to the effective region in the labeled depth information are determined as the target values. Based on the first predicted depth information obtained above, the depth values ​​of the pixels corresponding to the effective region in the first predicted depth information are determined as the predicted values. Exemplarily, but not limitingly, based on the target value and the predicted value, the above SILog function can be used: Calculate the first loss. Wherein, This is the depth value corresponding to the i-th pixel in the effective region from the first predicted depth information. The depth value corresponding to the i-th pixel in the effective region is used to annotate the depth information, where n is the total number of pixels in the effective region.

[0093] For any pixel in the invalid region, a specific depth value can be determined as the target value. For example, the depth value corresponding to a disparity value of 0 can be used as the target value. The depth value corresponding to the pixel in the invalid region in the first predicted depth information is used as the predicted value. Exemplarily, but not limitingly, based on the predicted value and the target value corresponding to the invalid region, the L1Loss loss function can be used: Calculate the second loss. Where y i y represents the depth value corresponding to the i-th pixel in the invalid region. i Equal to a specific depth value, Let represent the depth value corresponding to the i-th pixel in the invalid region in the first predicted depth information, where n is the total number of pixels in the invalid region. For example, the second predicted loss can be calculated by weighted summation of the first and second losses.

[0094] According to the above technical solution, image segmentation can be performed on the synthetic infrared image to obtain the effective and ineffective regions. The target depth estimation model is primarily trained on the effective regions, while ineffective regions are kept out of the training process by setting specific depth values ​​to their corresponding disparity values ​​of 0. This reduces the training load of the target depth estimation model, improves its training efficiency, and enhances its predictive performance.

[0095] For example, method 200 may further include: acquiring an RGB image to be processed, wherein the RGB image to be processed and an infrared image to be processed are acquired for the same target scene; spatially aligning the RGB image to be processed and the infrared image to be processed; converting the aligned RGB image to be processed into an infrared image to obtain a converted infrared image; inputting the converted infrared image or a new infrared image into a target depth estimation model to obtain a corresponding second depth estimation result, wherein the new infrared image is generated based on the converted infrared image, and the second depth estimation result is information related to the depth of each pixel in the RGB image to be processed; determining the second depth information corresponding to the RGB image to be processed based on the second depth estimation result; and fusing the first depth information and the second depth information pixel by pixel to obtain first comprehensive depth information.

[0096] For example, the method for acquiring the RGB image to be processed is similar to the method for acquiring the infrared image to be processed in step S210, which has been described in detail above and will not be repeated here for the sake of brevity. Furthermore, the RGB image and the infrared image to be processed are acquired in the same target scene. Also, the RGB image and the infrared image to be processed are the same size. If the target scene is a static scene, the two images can be acquired simultaneously or at certain time intervals. If the target scene is a dynamic scene, it is best to acquire the two images simultaneously.

[0097] After acquiring the RGB image and infrared image to be processed, they can optionally be spatially aligned. Because the RGB and infrared cameras are positioned differently, and because the RGB and infrared images may not be acquired completely simultaneously, the image information contained in the RGB and infrared images may not be spatially aligned. Therefore, the acquired RGB and infrared images can be spatially aligned. That is, the content corresponding to pixels at the same location in the RGB and infrared images should be identical.

[0098] Subsequently, the aligned RGB image to be processed can be converted into an infrared image to obtain a converted infrared image. For example, the style transfer network described above can be used to convert the aligned RGB image to be processed into an infrared image, thereby obtaining the converted infrared image.

[0099] In one example, the transformed infrared image can be directly input into the target depth estimation model to obtain a second depth estimate. In another example, the transformed infrared image can be processed to obtain a new infrared image, which can then be input into the target depth estimation model to obtain a second depth estimate. For example, the transformed infrared image can be horizontally flipped to obtain the new infrared image. It should be noted that if the transformed infrared image is horizontally flipped, the depth information must be flipped back accordingly. Flipping the transformed infrared image helps improve the robustness of the depth estimation method.

[0100] Similar to the first depth estimation result, the second depth estimation result can be either depth information or disparity information. The method for determining the first depth information based on the first depth estimation result is similar to the method for determining the second depth information based on the second depth estimation result described above, and will not be repeated here.

[0101] For example, the first depth information and the second depth information can be fused pixel by pixel, such as by taking a weighted average of the depth values ​​at the same pixel corresponding to the first depth information and the second depth information to obtain the first comprehensive depth information. The first comprehensive depth information can be determined as the depth information of the target scene.

[0102] According to the above technical solution, the RGB image to be processed, acquired in the same scene as the infrared image to be processed, is converted into an infrared image. The converted infrared image or the new infrared image is then processed using a target depth estimation model to obtain a second depth estimation result. Subsequently, the second depth information obtained from the second depth estimation result is fused with the first depth information to obtain first comprehensive depth information. Compared with single-modal prediction, this multi-modal fusion scheme provides more accurate depth information. That is, this multi-modal fusion scheme can improve the robustness of the depth estimation method.

[0103] For example, the method further includes: extracting one or more image patches from the infrared image to be processed, wherein the one or more image patches correspond one-to-one with one or more different scales, and each image patch contains the center point of the infrared image to be processed; for each of the one or more image patches, inputting the image patch into a target depth estimation model to obtain a corresponding sub-depth estimation result, wherein the sub-depth estimation result is information related to the depth of each pixel in the image patch; determining the sub-depth information corresponding to the sub-patch based on the sub-depth estimation result; and fusing the first depth information with the sub-depth information corresponding to the one or more image patches on a pixel-by-pixel basis to obtain second comprehensive depth information.

[0104] In one embodiment, one or more image blocks can be extracted from the acquired infrared image to be processed. The shape and size of each image block can be arbitrary. Preferably, the shape of the image block can be rectangular. In the infrared image to be processed, different image blocks can correspond to different scales, and each image block can include the center point of the infrared image to be processed. For example, each image block can be a rectangular image region centered on the center point of the infrared image to be processed. Any two different image blocks can be image regions with different widths and / or different heights.

[0105] For each of one or more image patches, the image patch can be input into the target depth estimation model described above to obtain the sub-depth estimation result corresponding to that image patch. Based on the sub-depth estimation result, the sub-depth information corresponding to that image patch can be determined in a manner similar to step S230. Subsequently, the sub-depth information corresponding to one or more image patches can be fused with the first depth information pixel by pixel, for example, by taking a weighted average of the sub-depth information corresponding to one or more image patches and the depth values ​​at the same pixel in the first depth information. The result obtained after weighted averaging can be used as the second comprehensive depth information. The second comprehensive depth information can be used as the depth information of the target scene.

[0106] For tasks such as autonomous driving, the accuracy of depth information in the central region of an image is more important than in other areas. According to the above technical solution, one or more image patches are extracted from the central region of the infrared image to be processed, and their sub-depth estimation results are calculated. Based on the sub-depth estimation results, the corresponding sub-depth information is obtained and fused with first depth information to obtain second comprehensive depth information. This scheme belongs to a multi-scale fusion scheme, which can improve the prediction accuracy of depth information in the central region of the image to be processed and is better suited for fields such as autonomous driving.

[0107] The aforementioned multi-scale fusion and multi-modal fusion can be implemented individually or in the same embodiment. For example, the sub-depth information, first depth information, and second depth information corresponding to one or more image blocks can be fused together to obtain third comprehensive depth information as the depth information of the target scene.

[0108] The preceding text primarily used the U-Net network model as an example to describe the depth estimation model presented in this paper. However, as mentioned above, the depth estimation model can also be implemented as NeW CRFs. The Transformer structure has significant advantages in global feature extraction. Using NeW CRFs networks can improve the feature extraction capability and fine-grained depth prediction of the depth estimation model. Among them, Conditional Random Fields (CRFs) can enhance the detail richness of the depth image by post-processing the coarse depth image predicted by the network.

[0109] For example, considering the instability in training target depth estimation models, especially Transformer models, when the batch size is small, gradient explosion can be added to improve the stability of target depth estimation model training. Furthermore, exponential moving average can be used to perform a moving average operation on the model weights, thereby avoiding the jitter caused by mini-batch gradient descent updating the target depth estimation model parameters and improving the generalization ability of the trained target depth estimation model in application scenarios.

[0110] For example, the target depth estimation model can be trained using a Quantization Aware Training (QAT) method based on a low-bit quantization model with floating-point self-distillation. For instance, a low-bit model with 4-bit weights and 4-bit features can be trained, and the floating-point target depth estimation model can be used for distillation training to improve the accuracy of the target depth estimation model. During the testing phase, the low-bit target depth estimation model can be deployed to optimize the model's running efficiency.

[0111] According to another aspect of this application, a depth estimation device based on infrared images is provided. Figure 5 A schematic block diagram of an infrared image-based depth estimation device 500 according to an embodiment of this application is shown.

[0112] like Figure 5 As shown, the infrared image-based depth estimation device 500 according to an embodiment of this application includes an acquisition module 510, an input module 520, and a determination module 530. Each module can respectively perform the functions described above. Figure 2 The following describes the individual steps of the infrared image-based depth estimation method. Only the main functions of each component of the infrared image-based depth estimation device 500 are described below, omitting the details already described above.

[0113] The acquisition module 510 is used to acquire the infrared image to be processed. The acquisition module 510 can be composed of... Figure 1 The processor 102 in the illustrated electronic device executes program instructions stored in the storage device 104 to achieve this.

[0114] The input module 520 is used to input the infrared image to be processed into the target depth estimation model to obtain the corresponding first depth estimation result. The first depth estimation result is information related to the depth of each pixel in the infrared image to be processed. The target depth estimation model is trained using synthetic infrared training data, which includes a synthetic infrared image and labeled depth information. The synthetic infrared image is generated based on a labeled RGB image, and the labeled depth information is the depth information corresponding to the labeled RGB image. The input module 520 can be... Figure 1 The processor 102 in the illustrated electronic device executes program instructions stored in the storage device 104 to achieve this.

[0115] The determining module 530 is used to determine the first depth estimation result as the first depth information corresponding to the infrared image to be processed, or, based on the first depth estimation result, to determine the first depth information. The determining module 530 may be composed of... Figure 1 The processor 102 in the illustrated electronic device executes program instructions stored in the storage device 104 to achieve this.

[0116] Figure 6 A schematic block diagram of an electronic device 600 according to an embodiment of this application is shown. The electronic device 600 includes a memory 610 and a processor 620.

[0117] The memory 610 stores computer program instructions for implementing the corresponding steps in the infrared image-based depth estimation method 200 according to embodiments of the present application.

[0118] The processor 620 is used to run computer program instructions stored in the memory 610 to perform corresponding steps of the infrared image-based depth estimation method 200 according to embodiments of the present application.

[0119] For example, the electronic device 600 may also include an image acquisition device 630. The image acquisition device 630 is used to acquire an infrared image to be processed. The image acquisition device 630 is optional, and the electronic device 600 may also exclude the image acquisition device 630. In this case, the processor 620 may acquire the infrared image to be processed by other means, such as from an external device or from the memory 610.

[0120] Furthermore, according to embodiments of this application, a storage medium is also provided, on which program instructions are stored. When the program instructions are run by a computer or processor, they are used to execute corresponding steps of the infrared image-based depth estimation method 200 of this application embodiment, and to implement corresponding modules in the infrared image-based depth estimation device 500 of this application embodiment. The storage medium may include, for example, a memory card of a smartphone, a storage component of a tablet computer, a hard disk of a personal computer, a read-only memory (ROM), an erasable programmable read-only memory (EPROM), a portable compact disc read-only memory (CD-ROM), a USB memory, or any combination of the above storage media.

[0121] In one embodiment, when the program instructions are executed by a computer or processor, the computer or processor may implement the various functional modules of the infrared image-based depth estimation apparatus according to the embodiments of this application, and / or may execute the infrared image-based depth estimation method according to the embodiments of this application.

[0122] Furthermore, according to an embodiment of this application, a computer program product is also provided, which includes a computer program that, when running, performs the aforementioned depth estimation method 200 based on infrared images.

[0123] Each module in the electronic device according to the embodiments of this application can be implemented by a processor that implements depth estimation based on infrared images according to the embodiments of this application running computer program instructions stored in memory, or by computer instructions stored in a computer-readable storage medium of a computer program product according to the embodiments of this application being implemented by a computer running.

[0124] Furthermore, according to an embodiment of this application, a computer program is also provided, which, when running, is used to execute the aforementioned depth estimation method 200 based on infrared images.

[0125] Although exemplary embodiments have been described herein with reference to the accompanying drawings, it should be understood that the above exemplary embodiments are merely illustrative and are not intended to limit the scope of this application. Various changes and modifications can be made therein by those skilled in the art without departing from the scope and spirit of this application. All such changes and modifications are intended to be included within the scope of this application as claimed in the appended claims.

[0126] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific testing and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific test, but such implementation should not be considered beyond the scope of this application.

[0127] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another device, or some features may be ignored or not executed.

[0128] Numerous specific details are set forth in the specification provided herein. However, it will be understood that embodiments of this application may be practiced without these specific details. In some instances, well-known methods, structures, and techniques have not been shown in detail so as not to obscure the understanding of this specification.

[0129] Similarly, it should be understood that, in order to simplify this application and aid in understanding one or more aspects of the various applications, features of this application are sometimes grouped together in a single embodiment, figure, or description thereof in the description of exemplary embodiments of this application. However, this approach should not be construed as reflecting an intention that the claimed application requires more features than are expressly recited in each claim. Rather, as reflected in the corresponding claims, the inventive point lies in solving the corresponding technical problem with fewer features than all features of a single disclosed embodiment. Therefore, the claims following the detailed description are hereby expressly incorporated into that detailed description, wherein each claim itself is a separate embodiment of this application.

[0130] Those skilled in the art will understand that, apart from the mutual exclusion of features, all features disclosed in this specification (including the accompanying claims, abstract, and drawings) and all processes or units of any method or apparatus so disclosed can be combined in any combination. Unless otherwise expressly stated, each feature disclosed in this specification (including the accompanying claims, abstract, and drawings) may be replaced by an alternative feature that serves the same, equivalent, or similar purpose.

[0131] Furthermore, those skilled in the art will understand that although some embodiments herein include certain features included in other embodiments but not others, combinations of features from different embodiments are intended to be within the scope of this application and form different embodiments. For example, in the claims, any of the claimed embodiments can be used in any combination.

[0132] The various component embodiments of this application can be implemented in hardware, or as software modules running on one or more processors, or a combination thereof. Those skilled in the art will understand that microprocessors or digital signal processors (DSPs) can be used in practice to implement some or all of the functions of some modules in the infrared image-based depth estimation apparatus according to the embodiments of this application. This application can also be implemented as an apparatus program (e.g., a computer program and computer program product) for performing part or all of the methods described herein. Such an implementation of this application can be stored on a computer-readable medium, or can be in the form of one or more signals. Such signals can be downloaded from an Internet website, provided on a carrier signal, or provided in any other form.

[0133] It should be noted that the above embodiments are illustrative of this application and not restrictive, and that those skilled in the art can devise alternative embodiments without departing from the scope of the appended claims. In the claims, any reference signs placed between parentheses should not be construed as limiting the claims. The word "comprising" does not exclude the presence of elements or steps not listed in the claims. The word "a" or "an" preceding an element does not exclude the presence of a plurality of such elements. This application can be implemented by means of hardware comprising several different elements and by means of a suitably programmed computer. In the unit claims enumerating several means, several of these means may be embodied by the same item of hardware. The use of the words first, second, and third, etc., does not indicate any order. These words can be interpreted as names.

[0134] The above are merely specific embodiments or descriptions of specific embodiments of this application. The scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. The scope of protection of this application shall be determined by the scope of the claims.

Claims

1. A depth estimation method based on infrared images, comprising: Acquire the infrared image to be processed; The infrared image to be processed is input into the target depth estimation model to obtain the corresponding first depth estimation result. The first depth estimation result is information related to the depth of each pixel in the infrared image to be processed. The first depth estimation result is determined as the first depth information corresponding to the infrared image to be processed, or the first depth information is determined based on the first depth estimation result; The target depth estimation model is obtained by training synthetic infrared training data, which includes synthetic infrared images and labeled depth information. The synthetic infrared images are generated based on labeled RGB images, and the labeled depth information is the depth information corresponding to the labeled RGB images. The synthesized infrared image is generated in the following manner: Obtain the labeled RGB image; The labeled RGB image is input into a style transfer network to obtain the synthesized infrared image; The style transfer network is trained in the following way: The sample RGB image and the first sample infrared image are input into the style transfer network to obtain a first predicted RGB image and a first predicted infrared image corresponding to the sample RGB image, as well as a second predicted RGB image and a second predicted infrared image corresponding to the first sample infrared image. The first predicted RGB image is used as a positive sample and input into the first discrimination network to obtain the first discrimination result; The second predicted RGB image is used as a negative sample and input into the first discrimination network to obtain the second discrimination result; The first predicted infrared image is used as a negative sample and input into the first discrimination network to obtain a third discrimination result. The second predicted infrared image is input as a positive sample into the first discrimination network to obtain the fourth discrimination result; Based on the first discrimination result, the second discrimination result, the third discrimination result, and the fourth discrimination result, the first prediction loss is calculated; The style transfer network and the first discriminant network are trained adversarially based on the first prediction loss.

2. The method as described in claim 1, wherein, The step of inputting the sample RGB image and the first sample infrared image into the style transfer network to obtain a first predicted RGB image and a first predicted infrared image corresponding to the sample RGB image, and a second predicted RGB image and a second predicted infrared image corresponding to the first sample infrared image, includes: Perform the following operations through the style transfer network: Perform a first encoding operation on the sample RGB image to obtain a first encoded feature; Perform a second encoding operation on the first sample infrared image to obtain a second encoded feature; The first encoded feature and the second encoded feature are combined to obtain the merged feature; Perform a first decoding operation on the merged features to obtain the first predicted RGB image and the second predicted RGB image; A second decoding operation is performed on the merged features to obtain the first predicted infrared image and the second predicted infrared image.

3. The method as described in claim 1 or 2, wherein, The target depth estimation model is obtained through the following first training operation: Obtain the synthesized infrared image and the labeled depth information; The synthetic infrared image is input into the target depth estimation model to obtain the corresponding first prediction estimation result, wherein the first prediction estimation result is information related to the depth of each pixel in the synthetic infrared image; Based on the first prediction estimation result, the first predicted depth information corresponding to the synthesized infrared image is determined; Calculate the second prediction loss based on the labeled depth information and the first predicted depth information; The parameters in the target depth estimation model are optimized based on the second prediction loss; or, The target depth estimation model is obtained through the following second training operation: Acquire the second and third sample infrared images; The weights of the target depth estimation model are initialized using the weights of the depth estimation model to be transferred, wherein the network structures of the depth estimation model to be transferred and the target depth estimation model are the same, and the depth estimation model to be transferred is obtained by training based on the synthetic infrared training data; The second sample infrared image is input into the depth estimation model to be transferred to obtain the corresponding second prediction estimation result, which is information related to the depth of each pixel in the second sample infrared image. Based on the second prediction estimation result, the second predicted depth information corresponding to the second sample infrared image is determined; The third sample infrared image is input into the target depth estimation model to obtain the corresponding third prediction estimation result, which is information related to the depth of each pixel in the third sample infrared image. Based on the third prediction estimation result, the third predicted depth information corresponding to the third sample infrared image is determined; The second predicted depth information is used as a positive sample and input into the second discriminant network to obtain the fifth discriminant result; The third predicted depth information is input as a negative sample into the second discrimination network to obtain the sixth discrimination result; The third prediction loss is calculated based at least on the fifth and sixth discrimination results; The target depth estimation model and the second discriminant network are trained adversarially based on the third prediction loss.

4. The method of claim 3, wherein, The depth estimation model to be transferred and the target depth estimation model each include a feature extraction module and a depth prediction module connected in sequence. The step of inputting the second sample infrared image into the depth estimation model to be migrated to obtain the corresponding second prediction estimation result includes: The second sample infrared image is input into the depth estimation model to be transferred, and the first infrared feature output by the feature extraction module of the depth estimation model to be transferred and the second prediction estimation result output by the depth prediction module of the depth estimation model to be transferred are obtained. The step of inputting the third sample infrared image into the target depth estimation model to obtain the corresponding third prediction estimation result includes: The third sample infrared image is input into the target depth estimation model to obtain the second infrared feature output by the feature extraction module of the target depth estimation model and the third prediction estimation result output by the depth prediction module of the target depth estimation model. Before calculating the third prediction loss based at least on the fifth and sixth discrimination results, the second training operation further includes: The first infrared feature is input as a positive sample into the second discrimination network to obtain the seventh discrimination result; The second infrared feature is used as a negative sample and input into the second discrimination network to obtain the eighth discrimination result; The calculation of the third prediction loss based at least on the fifth and sixth discrimination results includes: The third prediction loss is calculated based on the fifth, sixth, seventh, and eighth discrimination results; In the process of adversarial training of the target depth estimation model and the second discriminant network based on the third prediction loss, the target parameters in the target depth estimation model are optimized, and the remaining parameters in the target depth estimation model other than the target parameters are fixed. The target parameters are at least some of the parameters in the feature extraction module of the target depth estimation model.

5. The method of claim 3, wherein, Before calculating the second prediction loss based on the labeled depth information and the first prediction depth information, the first training operation further includes: The synthesized infrared image is input into an image segmentation network to obtain the image segmentation result; Based on the image segmentation results, valid and invalid regions are determined; The calculation of the second prediction loss based on the labeled depth information and the first predicted depth information includes: For any pixel in the effective region, the depth value corresponding to that pixel in the labeled depth information is taken as the target value, and the depth value corresponding to that pixel in the first predicted depth information is taken as the predicted value, and the first loss is calculated. For any pixel in the invalid region, a specific depth value is used as the target value, and the depth value corresponding to that pixel in the first predicted depth information is used as the predicted value to calculate the second loss, wherein the disparity value corresponding to the specific depth value is 0. The second predicted loss is calculated based on the first loss and the second loss.

6. The method as described in claim 1 or 2, wherein, The method further includes: Acquire an RGB image to be processed, wherein the RGB image to be processed and the infrared image to be processed are acquired for the same target scene; Spatially align the RGB image to be processed with the infrared image to be processed; The aligned RGB image to be processed is converted into an infrared image to obtain the converted infrared image; The transformed infrared image or the new infrared image is input into the target depth estimation model to obtain the corresponding second depth estimation result, wherein the new infrared image is generated based on the transformed infrared image, and the second depth estimation result is information related to the depth of each pixel in the RGB image to be processed; Based on the second depth estimation result, the second depth information corresponding to the RGB image to be processed is determined; The first depth information and the second depth information are fused pixel by pixel to obtain the first comprehensive depth information.

7. The method as described in claim 1 or 2, wherein, The method further includes: One or more image patches are extracted from the infrared image to be processed. The one or more image patches correspond one-to-one with one or more different scales. Each image patch contains the center point of the infrared image to be processed. For each of the one or more image blocks The image patch is input into the target depth estimation model to obtain the corresponding sub-depth estimation result, which is information related to the depth of each pixel in the image patch; The sub-depth information corresponding to the image patch is determined based on the sub-depth estimation result; The first depth information is fused with the sub-depth information corresponding to the one or more image blocks on a pixel-by-pixel basis to obtain the second comprehensive depth information.

8. An electronic device comprising a processor and a memory, wherein, The memory stores computer program instructions, which, when executed by the processor, are used to perform the depth estimation method based on infrared images as described in any one of claims 1 to 7.

9. A storage medium on which program instructions are stored, wherein, The program instructions are used to execute the depth estimation method based on infrared images as described in any one of claims 1 to 7 when the program is run.

10. A computer program product, the computer program product comprising a computer program, wherein, The computer program, when running, is used to execute the depth estimation method based on infrared images as described in any one of claims 1 to 7.

Images