Depth image generation method and device, equipment, computer readable medium and product

Abstract

The embodiment of the invention discloses a depth image generation method, device and equipment, a computer readable medium and a product. A specific embodiment of the method comprises the following steps: carrying out conversion operation on an image sequence to obtain an image information sequence; for each piece of image information, the image information is input into an encoder included in a pre-trained depth image generation model, a feature map is obtained, and the depth image generation model comprises the encoder, a memory module and a decoder; inputting the feature map into a memory module to perform attention operation on the feature map and the target memory feature to obtain an updated feature map; and inputting the updated feature map into a decoder to obtain a depth image. The embodiment is related to depth estimation, and the accuracy of each depth image generated for the time sequence image sequence is improved.

Description

Technical Field

[0001] Embodiments of the present disclosure relate to the field of computer technology, and in particular to a depth image generation method, apparatus, device, computer-readable medium, and product. Background Art

[0002] Depth Estimation is a computer vision technology that aims to infer the depth information in a scene from a single or multiple images. This process can help us understand the three-dimensional world and plays an important role in many applications, such as augmented reality (AR) and virtual reality (VR). At present, with the development of deep learning, convolutional neural networks (CNNs) or other deep learning models can be used to learn the mapping relationship between an image and its corresponding depth map.

[0003] However, when the above method is adopted, the following technical problems often occur: it mainly relies on a single image data as input, lacks temporal sequence, and the model does not consider the temporal dependency and spatial consistency between adjacent frames. As a result, when depth prediction is applied to a temporal image sequence (such as video), the predicted depth values ​​of the same pixel jump greatly, resulting in poor accuracy of the generated depth images.

[0004] The above information disclosed in this Background section is only for enhancement of understanding of the background of the inventive concept and therefore it may contain information that does not form the prior art that is already known to a person of ordinary skill in the art. Summary of the invention

[0005] The content of this disclosure is used to introduce concepts in a brief form, which will be described in detail in the detailed implementation section below. The content of this disclosure is not intended to identify the key features or essential features of the technical solution claimed for protection, nor is it intended to limit the scope of the technical solution claimed for protection.

[0006] Some embodiments of the present disclosure propose a depth image generation method, a depth image generation device, an electronic device, a computer-readable medium, and a computer program product to solve one or more of the technical problems mentioned in the above background technology section.

[0007] In a first aspect, some embodiments of the present disclosure provide a method for generating a depth image, the method comprising: performing a conversion operation on each image in a time-series image sequence to obtain an image information sequence; for each image information in the above image information sequence, executing the following loop steps: inputting the above image information into an encoder included in a pre-trained deep image generation model to obtain a feature map, wherein the above deep image generation model includes an encoder, a memory module and a decoder; inputting the above feature map into the above memory module to perform an attention operation on the above feature map and the target memory feature to obtain an updated feature map; inputting the above updated feature map into the above decoder to obtain a depth image.

[0008] Optionally, the method further includes: combining the obtained depth images into a depth image sequence corresponding to the above-mentioned image sequence.

[0009] Optionally, the memory module includes an attention module; and the feature map is input into the memory module to perform an attention operation on the feature map and the target memory feature to obtain an updated feature map, including: inputting the feature map into the attention module to perform an attention operation on the feature map and the target memory feature to obtain an updated feature map.

[0010] Optionally, the above-mentioned memory module also includes a memory encoding module and a memory storage module; and the above-mentioned loop step also includes: inputting the obtained depth image and the above-mentioned feature map into the above-mentioned memory encoding module to obtain memory features; and storing the above-mentioned memory features into the above-mentioned memory storage module.

[0011] Optionally, the above-mentioned loop step also includes: in response to determining that the above-mentioned loop step does not meet the preset loop number condition, skipping the above-mentioned attention module, and inputting the above-mentioned feature map into the above-mentioned decoder to obtain a depth image.

[0012] Optionally, the encoder includes sequential feature extraction layers; and the image information is input into an encoder included in a pre-trained deep image generation model to obtain a feature map, including: determining the first feature extraction layer among the feature extraction layers as the current feature extraction layer; determining the image information as the current image information; based on the current feature extraction layer and the current image information, performing the following feature extraction steps: inputting the current image information into the current feature extraction layer to obtain the current feature map; in response to determining that the current feature extraction layer is the last feature extraction layer, determining the current feature map as the feature map; in response to determining that the current feature extraction layer is not the last feature extraction layer, taking the next feature extraction layer of the current feature extraction layer as the current feature extraction layer, taking the current feature map as the current image information, and performing the feature extraction steps again.

[0013] Optionally, the decoder includes feature decoding layers corresponding to the feature extraction layers; and the updated feature map is input into the decoder to obtain a depth image, including: for each of the feature extraction layers, the current feature map or updated feature map corresponding to the feature extraction layer is input into the feature decoding layer corresponding to the feature extraction layer to obtain a depth feature map; and a depth image is generated based on the obtained depth feature maps.

[0014] Optionally, generating a depth image based on the obtained depth feature maps includes: fusing the above-mentioned depth feature maps to obtain a fused depth feature map; and normalizing the above-mentioned fused depth feature map to obtain a depth image.

[0015] Optionally, the attention module includes various attention layers; and the feature map is input into the attention module to perform an attention operation on the feature map and the target memory feature to obtain an updated feature map, including: inputting the feature map into the first attention layer of the attention layers to obtain an output feature as the current output feature; determining the second attention layer of the attention layers as the current attention layer; based on the current attention layer and the current output feature, performing the following attention operations: based on the current output feature and the position encoding corresponding to the feature map, performing a self-attention mechanism on the current output feature in the current attention layer to obtain a first output feature; based on the first output feature , the target memory feature, the position code corresponding to the feature map, and the position code corresponding to the target memory feature, perform cross-attention mechanism processing on the current output feature and the target memory feature in the current attention layer to obtain a second output feature; based on the second output feature, generate the output feature corresponding to the current attention layer as the target output feature; in response to determining that the current attention layer is the last attention layer, normalize the target output feature to obtain an updated feature map; in response to determining that the current attention layer is not the last attention layer, take the next attention layer of the current attention layer as the current attention layer, take the target output feature as the current output feature, and perform the above attention operation again.

[0016] Optionally, the above-mentioned generating an output feature corresponding to the current attention layer as a target output feature based on the above-mentioned second output feature includes: generating an output feature of the current output feature at the current attention layer as a third output feature based on the above-mentioned second output feature; and determining the sum of the above-mentioned second output feature and the above-mentioned third output feature as the output feature corresponding to the current attention layer.

[0017] Optionally, the memory encoding module includes building blocks of each layer; and the depth image and the feature map are input into the memory encoding module to obtain memory features, including: performing resolution preprocessing on the depth image to obtain a processed depth image; fusing the processed depth image and the feature map to obtain fused features; inputting the fused features into the first layer building blocks of each layer of the building blocks to obtain a coding result as a current coding result; determining the second layer building block of each layer of the building blocks as a current building block; based on the current coding result and the current building block, performing the following coding steps: inputting the current coding result into the current building block to obtain a coding result as a target coding result; in response to determining that the current building block is the last building block, determining the target coding result as a memory feature; in response to determining that the current building block is not the last building block, taking the next building block of the current building block as the current building block, taking the target coding result as the current coding result, and performing the coding steps again.

[0018] Optionally, the above-mentioned depth image generation model is obtained through the following training steps: obtaining each group of sample image sequences and sample depth image sequences with a time sequence, wherein the sample depth image sequence in each group corresponds to the sample image sequence; performing image preprocessing on each group of sample image sequences and sample depth image sequences; for each group of sample image sequences and sample depth image sequences after image preprocessing, generating a sample group, wherein each sample in the sample group includes sample image information and sample depth image information corresponding to the sample image information; based on each generated sample group, taking the sample image information in each sample group as input and the sample depth image information corresponding to the sample image as the expected output, training the initial depth image generation model to obtain a trained depth image generation model.

[0019] In a second aspect, some embodiments of the present disclosure provide a depth image generation device, comprising: a conversion unit, configured to perform a conversion operation on each image in a time-series image sequence to obtain an image information sequence; an execution unit, configured to execute the following loop steps for each image information in the above-mentioned image information sequence: input the above-mentioned image information into an encoder included in a pre-trained deep image generation model to obtain a feature map, wherein the above-mentioned deep image generation model includes an encoder, a memory module and a decoder; input the above-mentioned feature map into the above-mentioned memory module to perform an attention operation on the above-mentioned feature map and the target memory feature to obtain an updated feature map; input the above-mentioned updated feature map into the above-mentioned decoder to obtain a depth image.

[0020] Optionally, the depth image generating apparatus further includes: a combining unit configured to combine the obtained depth images into a depth image sequence corresponding to the above-mentioned image sequence.

[0021] Optionally, the above-mentioned memory module includes an attention module.

[0022] Optionally, the execution unit is further configured to: input the above-mentioned feature map into the above-mentioned attention module to perform attention operation on the above-mentioned feature map and the target memory feature to obtain an updated feature map.

[0023] Optionally, the above-mentioned memory module also includes a memory encoding module and a memory storage module.

[0024] Optionally, the loop step further includes: an input unit and a storage unit. The input unit is configured to input the obtained depth image and the feature map into the memory encoding module to obtain the memory feature. The storage unit is configured to store the memory feature into the memory storage module.

[0025] Optionally, the above-mentioned loop step may also include: a skipping unit, configured to skip the above-mentioned attention module in response to determining that the above-mentioned loop step does not meet the preset loop number condition, and input the above-mentioned feature map into the above-mentioned decoder to obtain a depth image.

[0026] Optionally, the encoder comprises sequential feature extraction layers.

[0027] Optionally, the execution unit is further configured to: determine the first feature extraction layer among the above-mentioned feature extraction layers as the current feature extraction layer; determine the above-mentioned image information as the current image information; based on the current feature extraction layer and the current image information, perform the following feature extraction steps: input the current image information into the current feature extraction layer to obtain the current feature map; in response to determining that the current feature extraction layer is the last feature extraction layer, determine the current feature map as the feature map; in response to determining that the current feature extraction layer is not the last feature extraction layer, use the next feature extraction layer of the current feature extraction layer as the current feature extraction layer, use the current feature map as the current image information, and perform the above-mentioned feature extraction steps again.

[0028] Optionally, the decoder includes feature decoding layers corresponding to the feature extraction layers.

[0029] Optionally, the execution unit is further configured to: for each of the above-mentioned feature extraction layers, input the current feature map or updated feature map corresponding to the above-mentioned feature extraction layer into the feature decoding layer corresponding to the above-mentioned feature extraction layer to obtain a depth feature map; and generate a depth image based on the obtained depth feature maps.

[0030] Optionally, the execution unit is further configured to: perform fusion processing on the above-mentioned depth feature maps to obtain a fused depth feature map; and perform normalization processing on the above-mentioned fused depth feature map to obtain a depth image.

[0031] Optionally, the above-mentioned attention module includes various attention layers.

[0032] Optionally, the execution unit is further configured to: input the above-mentioned feature map to the first attention layer among the above-mentioned attention layers, and obtain the output feature as the current output feature; determine the second attention layer among the above-mentioned attention layers as the current attention layer; based on the current attention layer and the current output feature, perform the following attention operations: based on the current output feature and the position code corresponding to the above-mentioned feature map, perform self-attention mechanism processing on the current output feature at the current attention layer to obtain the first output feature; based on the above-mentioned first output feature, the above-mentioned target memory feature, the position code corresponding to the above-mentioned feature map, and the position code corresponding to the above-mentioned target memory feature, perform cross-attention mechanism processing on the current output feature and the target memory feature at the current attention layer to obtain the second output feature; based on the above-mentioned second output feature, generate the output feature corresponding to the current attention layer as the target output feature; in response to determining that the current attention layer is the last attention layer, normalize the above-mentioned target output feature to obtain an updated feature map; in response to determining that the current attention layer is not the last attention layer, take the next attention layer of the current attention layer as the current attention layer, take the target output feature as the current output feature, and perform the above-mentioned attention operation again.

[0033] Optionally, the execution unit is further configured to: based on the above-mentioned second output feature, generate an output feature of the current output feature at the current attention layer as a third output feature; and determine the sum of the above-mentioned second output feature and the above-mentioned third output feature as the output feature corresponding to the current attention layer.

[0034] Optionally, the memory encoding module includes various layers of building blocks.

[0035] Optionally, the execution unit is further configured to: perform resolution preprocessing on the depth image to obtain a processed depth image; perform fusion processing on the processed depth image and the feature map to obtain a fusion feature; input the fusion feature into the first layer building block in each layer of the building blocks to obtain a coding result as the current coding result; determine the second layer building block in each layer of the building blocks as the current building block; based on the current coding result and the current building block, perform the following coding steps: input the current coding result into the current building block to obtain a coding result as a target coding result; in response to determining that the current building block is the last building block, determine the target coding result as a memory feature; in response to determining that the current building block is not the last building block, use the next building block of the current building block as the current building block, use the target coding result as the current coding result, and perform the above coding steps again.

[0036] Optionally, the above-mentioned depth image generation model is obtained through the following training steps: obtaining each group of sample image sequences and sample depth image sequences with a time sequence, wherein the sample depth image sequence in each group corresponds to the sample image sequence; performing image preprocessing on each group of sample image sequences and sample depth image sequences; for each group of sample image sequences and sample depth image sequences after image preprocessing, generating a sample group, wherein each sample in the sample group includes sample image information and sample depth image information corresponding to the sample image information; based on each generated sample group, taking the sample image information in each sample group as input and the sample depth image information corresponding to the sample image as the expected output, training the initial depth image generation model to obtain a trained depth image generation model.

[0037] In a third aspect, some embodiments of the present disclosure provide an electronic device comprising: one or more processors; a storage device on which one or more programs are stored, and when the one or more programs are executed by the one or more processors, the one or more processors implement the method described in any implementation manner of the above-mentioned first aspect.

[0038] In a fourth aspect, some embodiments of the present disclosure provide a computer-readable medium having a computer program stored thereon, wherein when the program is executed by a processor, the method described in any implementation manner of the above-mentioned first aspect is implemented.

[0039] In a fifth aspect, some embodiments of the present disclosure provide a computer program product, including a computer program, which implements the method described in any implementation manner of the above-mentioned first aspect when executed by a processor.

[0040] The above-mentioned embodiments of the present disclosure have the following beneficial effects: through the depth image generation method of some embodiments of the present disclosure, a depth estimation algorithm framework with a memory mechanism capable of maintaining the temporal consistency of image sequence reasoning is provided, thereby improving the accuracy of each depth image generated for a temporal image sequence. Specifically, the reason for the poor accuracy of each depth image generated is that it mainly relies on a single image data as input, lacks temporal sequence, and the model does not consider the temporal dependence and spatial consistency between adjacent frames, resulting in a large jump in the predicted depth values ​​of the same pixel when the depth prediction is applied to a temporal image sequence (such as a video), resulting in poor accuracy of each depth image generated. Based on this, the depth image generation method of some embodiments of the present disclosure, first, performs a conversion operation on each image in a temporal image sequence to obtain an image information sequence. Thus, each temporal image can be converted into an information form that is convenient for the model to read. Then, for each image information in the above image information sequence, the following loop steps are performed: First, the above image information is input into the encoder included in the pre-trained deep image generation model to obtain a feature map, wherein the above deep image generation model includes an encoder, a memory module and a decoder. Thus, the encoder can be used to extract features from the current image. Second, the feature map is input into the memory module to perform attention operations on the feature map and the target memory features to obtain an updated feature map. Thus, the historical information can be fused with the feature map of the current frame to enhance the feature map of the current frame. Third, the updated feature map is input into the decoder to obtain a depth image. Thus, when the decoder generates a depth prediction for the current frame, it can inherit the feature information of the historical frame and maintain a smooth depth transition with the historical frame. Also because the attention module of the deep image generation model can fuse the historical information with the feature map of the current frame to enhance the feature map of the current frame, when the decoder generates a depth prediction for the current frame, it can inherit the feature information of the historical frame and maintain a smooth depth transition with the historical frame, thereby providing a depth estimation algorithm framework with a memory mechanism that can maintain the temporal consistency of image sequence reasoning, thereby improving the accuracy of each depth image generated for a temporal image sequence. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. Throughout the accompanying drawings, the same or similar reference numerals represent the same or similar elements. It should be understood that the drawings are schematic and that components and elements are not necessarily drawn to scale.

[0042] Figure 1 is an architectural diagram of an exemplary system in which some embodiments of the present disclosure may be applied;

[0043] Figure 2 is a flow chart of some embodiments of the depth image generation method according to the present disclosure;

[0044] Figure 3 is a schematic diagram of the structure of a deep image generation model in some embodiments of the present disclosure;

[0045] Figure 4 is a training flow chart of a depth image generation model according to the depth image generation method disclosed herein;

[0046] Figure 5 is a schematic structural diagram of some embodiments of a depth image generating device according to the present disclosure;

[0047] Figure 6 It is a schematic diagram of the structure of an electronic device suitable for implementing some embodiments of the present disclosure. DETAILED DESCRIPTION

[0048] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments set forth herein. On the contrary, these embodiments are provided to provide a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes and are not intended to limit the scope of protection of the present disclosure.

[0049] It should also be noted that, for ease of description, only the parts related to the invention are shown in the drawings. In the absence of conflict, the embodiments and features in the embodiments of the present disclosure can be combined with each other.

[0050] It should be noted that the concepts such as "first" and "second" mentioned in the present disclosure are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.

[0051] It should be noted that the modifications of "one" and "plurality" mentioned in the present disclosure are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise clearly indicated in the context, it should be understood as "one or more".

[0052] The names of the messages or information exchanged between multiple devices in the embodiments of the present disclosure are only used for illustrative purposes and are not used to limit the scope of these messages or information.

[0053] With regard to the collection, storage, and use of user personal information (e.g., image sequences) involved in this disclosure, before performing the corresponding operations, the relevant organizations or individuals shall fulfill obligations including conducting personal information security impact assessments, fulfilling the obligation to inform the personal information subject, and obtaining the authorization and consent of the personal information subject in advance.

[0054] The present disclosure will be described in detail below with reference to the accompanying drawings and in conjunction with embodiments.

[0055] Figure 1 An exemplary system architecture 100 is shown to which a depth image generation method or a depth image generation apparatus according to some embodiments of the present disclosure may be applied.

[0056] like Figure 1 As shown, the system architecture 100 may include terminal devices 101, 102, 103, a network 104 and a server 105. The network 104 is used to provide a medium for communication links between the terminal devices 101, 102, 103 and the server 105. The network 104 may include various connection types, such as wired, wireless communication links or optical fiber cables, etc.

[0057] The user can use the terminal devices 101, 102, 103 to interact with the server 105 through the network 104 to receive or send messages, etc. Various communication client applications can be installed on the terminal devices 101, 102, 103, such as web browser applications, depth estimation applications, search applications, instant messaging tools, email clients, social platform software, etc.

[0058] Terminal devices 101, 102, 103 can be hardware or software. When terminal devices 101, 102, 103 are hardware, they can be various electronic devices with display screens and supporting information display, including but not limited to smart phones, tablet computers, e-book readers, laptop portable computers, desktop computers, etc. When terminal devices 101, 102, 103 are software, they can be installed in the electronic devices listed above. They can be implemented as multiple software or software modules for providing distributed services, for example, or they can be implemented as a single software or software module. No specific limitation is made here.

[0059] The server 105 may be a server that provides various services, such as a background server that provides support for information displayed on the terminal devices 101, 102, and 103. The background server may analyze and process the received request and other data, and feed back the processing results (such as a depth image sequence) to the terminal device.

[0060] It should be noted that the depth image generation method provided in the embodiment of the present disclosure can be executed by the terminal device 101, 102, 103, or by the server 105. Accordingly, the depth image generation device can be set in the terminal device 101, 102, 103, or in the server 105. No specific limitation is made here.

[0061] It should be noted that the server can be hardware or software. When the server is hardware, it can be implemented as a distributed server cluster consisting of multiple servers, or it can be implemented as a single server. When the server is software, it can be implemented as multiple software or software modules for providing distributed services, or it can be implemented as a single software or software module. No specific limitation is made here.

[0062] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to implementation requirements.

[0063] Continue to refer Figure 2 , shows a process 200 of some embodiments of the depth image generation method according to the present disclosure. The depth image generation method comprises the following steps:

[0064] Step 201 : performing a conversion operation on each image in a time-sequential image sequence to obtain an image information sequence.

[0065] In some embodiments, the execution subject of the depth image generation method (eg Figure 1 The server or terminal device shown in the figure can perform conversion operations on each image in the image sequence with a time sequence to obtain an image information sequence. Among them, the above-mentioned image sequence with a time sequence can be a video, or a collection of each image with a time sequence. In practice, for each image in the above-mentioned image sequence, the above-mentioned execution subject can convert the above-mentioned image into a tensor as image information. In practice, for each image in the above-mentioned image sequence, first, the above-mentioned execution subject can perform image size conversion on the above-mentioned image, then the image can be standardized, and then the image can be converted into a tensor. The converted tensor can be used as image information. The image size conversion can be a conversion to a preset image size. The tensor can be a PyTorch tensor.

[0066] Step 202: for each image information in the image information sequence, perform the following loop steps:

[0067] Step 2021, input the image information into the encoder included in the pre-trained deep image generation model to obtain a feature map.

[0068] In some embodiments, the execution subject may input the image information into an encoder included in a pre-trained deep image generation model to obtain a feature map. The deep image generation model may be a neural network model that takes at least one image with a time sequence as input and corresponding depth images as output. The deep image generation model may include an encoder, a memory module, and a decoder. The encoder may be used for feature extraction. For example, the encoder may be an encoder based on Vision Transformer. The memory module may be used to save the features and depth information of historical frames, and may also fuse the historical information with the feature map of the current frame to enhance the feature map of the current frame. The decoder may be used to decode the enhanced feature map to obtain a depth image. For example, the decoder may be a decoder based on DPT (Dense Prediction Transformer) Head. DPT Head may be a network structure for dense prediction tasks.

[0069] Optionally, the encoder may include feature extraction layers in order. For example, the feature extraction layer may be a DINOv2 feature extraction layer (DINOv2 layer). The encoder may include 4 DINOv2 layers.

[0070] In some optional implementations of some embodiments, the above execution subject may perform the following steps:

[0071] In the first step, the first feature extraction layer among the above feature extraction layers is determined as the current feature extraction layer.

[0072] The second step is to determine the above image information as the current image information.

[0073] The third step is to perform the following feature extraction steps based on the current feature extraction layer and the current image information:

[0074] The first sub-step is to input the current image information into the current feature extraction layer to obtain the current feature map.

[0075] In the second sub-step, in response to determining that the current feature extraction layer is the last feature extraction layer, the current feature map is determined as the feature map. Thus, the output of the last feature extraction layer can be used as the feature map output by the encoder.

[0076] In the third sub-step, in response to determining that the current feature extraction layer is not the last feature extraction layer, the next feature extraction layer of the current feature extraction layer is used as the current feature extraction layer, and the current feature map is used as the current image information, and the above feature extraction step is performed again. Thus, the output of the previous feature extraction layer can be used as the input of the next feature extraction layer, so that the feature extraction of the image information can be gradually realized.

[0077] Step 2022, input the feature map into the memory module to perform an attention operation on the feature map and the target memory feature to obtain an updated feature map.

[0078] In some embodiments, the execution subject may input the feature map into the memory module to perform an attention operation on the feature map and the target memory feature to obtain an updated feature map. When the loop step is executed for the first time, the target memory feature may be empty, and the updated feature map may still be the feature map. When the loop step is executed later, the target memory feature may be the re-encoded feature of the depth image and the feature map obtained when the loop step was executed last time. The re-encoded feature may be a fused feature. In practice, the execution subject may perform an attention mechanism on the feature map and the target memory feature to obtain a weighted feature representation as an updated feature map.

[0079] Optionally, the memory module may include an attention module. The attention module may be used to perform an attention operation.

[0080] In some optional implementations of some embodiments, the above-mentioned execution entity may input the above-mentioned feature map into the above-mentioned attention module to perform attention operations on the above-mentioned feature map and the target memory feature to obtain an updated feature map.

[0081] Optionally, when the above loop step is executed for the first time, the target memory feature may be empty, and the updated feature map may still be the above feature map. When the above loop step is executed later, the target memory feature may be the memory feature stored in the memory storage module when the above loop step was executed last time.

[0082] Optionally, the above-mentioned attention module may include various attention layers. In each attention layer, the current feature map is first processed by the self-attention mechanism to consider the relationship between internal elements, and then processed with the target memory feature by the cross-attention mechanism to capture the dependency between the current feature map and the target memory feature. Each attention layer may also include a feedforward network for further processing of the data, and normalization and regularization through LayerNorm and Dropout. After all attention layers are processed, the final output may be normalized by LayerNorm.

[0083] In some optional implementations of some embodiments, the above execution subject may perform the following steps:

[0084] In the first step, the feature map is input into the first attention layer of each attention layer to obtain the output feature as the current output feature.

[0085] In the second step, the second attention layer among the above attention layers is determined as the current attention layer.

[0086] The third step is to perform the following attention operations based on the current attention layer and the current output features:

[0087] In the first sub-step, based on the current output feature and the position code corresponding to the feature map, the current output feature is processed by the self-attention mechanism at the current attention layer to obtain the first output feature. The first output feature can be the result of the output feature of the previous attention layer being processed by the self-attention mechanism at the current attention layer. The first output feature can be expressed as:

[0088] A l =Attention(Q=K=V=X l-1 +P X ).

[0089] Among them, A l represents the first output feature. l represents the current attention layer. l-1 represents the previous attention layer. P X Represents the position encoding corresponding to the above feature map. l-1 Represents the current output characteristics mentioned above.

[0090] In the second sub-step, based on the first output feature, the target memory feature, the position code corresponding to the feature map, and the position code corresponding to the target memory feature, the current output feature and the target memory feature are processed by the cross attention mechanism at the current attention layer to obtain the second output feature. The second output feature can be the result of the output feature of the previous attention layer being processed by the cross attention mechanism at the current attention layer and the target memory feature. The second output feature can be expressed as:

[0091] A′ l =CrrssAttention(Q=A l +P X ,K=M+P M , V=M).

[0092] Among them, A′ l represents the second output feature. M represents the target memory feature. P M Represents the positional encoding corresponding to the target memory feature.

[0093] The third sub-step is to generate an output feature corresponding to the current attention layer as a target output feature based on the second output feature. The target output feature may be an output feature obtained at the current attention layer.

[0094] In a fourth sub-step, in response to determining that the current attention layer is the last attention layer, the target output feature is normalized to obtain an updated feature map. In practice, the target output feature can be input into a normalization layer to obtain an updated feature map.

[0095] The fifth sub-step, in response to determining that the current attention layer is not the last attention layer, takes the next attention layer of the current attention layer as the current attention layer, takes the target output feature as the current output feature, and performs the above attention operation again.

[0096] In some optional implementations of some embodiments, the execution subject may generate an output feature corresponding to the current attention layer as a target output feature based on the second output feature through the following steps:

[0097] The first step is to generate the output feature of the current output feature in the current attention layer as the third output feature based on the second output feature. The third output feature can be the result of the output feature of the previous attention layer being processed in the current attention layer. The third output feature can be expressed as:

[0098] X′ l-1 =Linear2(dropout(Activation(Linear1(A′ l ))).

[0099] Among them, X′ l-1 Represents the third output feature. Linear1 represents a fully connected layer. Linear2 represents another fully connected layer. Activation represents the activation function. Dropout represents the dropout layer.

[0100] In the second step, the sum of the second output feature and the third output feature is determined as the output feature corresponding to the current attention layer.

[0101] Step 2023, input the updated feature map to the decoder to obtain a depth image.

[0102] In some embodiments, the execution entity may input the updated feature map into the decoder to obtain a depth image.

[0103] Optionally, the decoder may include feature decoding layers corresponding to the feature extraction layers. The number of the feature decoding layers may be the same as the number of the feature extraction layers. Each feature decoding layer corresponds to one feature extraction layer. The feature decoding layers correspond to the feature extraction layers in reverse order. For example, the feature decoding layer may be a DPT head network. The decoder may include 4 DPT head networks. The first feature extraction layer corresponds to the last feature decoding layer, and so on. Except for the first feature decoding layer, the input of each feature decoding layer is the output of the corresponding feature extraction layer. When the attention module is not skipped, the input of the first feature decoding layer may be the output of the feature map output by the last feature extraction layer after being processed by the attention module, that is, the updated feature map. When the attention module is skipped, the input of the first feature decoding layer may be the feature map output by the last feature extraction layer.

[0104] In some optional implementations of some embodiments, the above execution subject may perform the following steps:

[0105] In the first step, for each of the above-mentioned feature extraction layers, the current feature map or the updated feature map corresponding to the above-mentioned feature extraction layer is input into the feature decoding layer corresponding to the above-mentioned feature extraction layer to obtain a depth feature map. When the feature extraction layer is not the last layer, the current feature map corresponding to the above-mentioned feature extraction layer may be input into the feature decoding layer. When the feature extraction layer is the last layer, the updated feature map corresponding to the above-mentioned feature extraction layer may be input into the feature decoding layer.

[0106] The second step is to generate a depth image based on the obtained depth feature maps.

[0107] In some optional implementations of some embodiments, the above-mentioned execution subject may generate a depth image based on each obtained depth feature map through the following steps:

[0108] The first step is to fuse the above-mentioned depth feature maps to obtain a fused depth feature map. In practice, the above-mentioned execution subject can splice the above-mentioned depth feature maps to obtain a fused depth feature map.

[0109] The second step is to normalize the fused depth feature map to obtain a depth image. Thus, multiple decoded depth feature maps can be fused to generate a depth image.

[0110] Optionally, the memory module may further include a memory encoding module and a memory storage module. The memory encoding module may be used to re-encode the depth information and feature map of the historical frame. The memory storage module may be used to store the feature and depth information of the historical frame.

[0111] Optionally, the above-mentioned loop step may further include: inputting the obtained depth image and feature map into a memory encoding module to obtain a memory feature. In practice, the above-mentioned execution subject may fuse the pixel features of the above-mentioned depth image and the above-mentioned feature map to obtain the memory feature. Here, the fusion may be addition. Thus, the depth information and features may be re-encoded.

[0112] Optionally, the memory encoding module includes layers of building blocks. Each layer of building blocks may include a depthwise convolution, a pointwise convolution, and a layer normalization layer (LayerNorm).

[0113] In some optional implementations of some embodiments, the execution subject may input the obtained depth image and the feature map into the memory encoding module to obtain the memory feature through the following steps:

[0114] The first step is to perform resolution preprocessing on the depth image to obtain a processed depth image. In practice, the execution subject may reduce the resolution of the depth image to a preset resolution to obtain a processed depth image. For example, the preset resolution may be 518*518. Thus, the amount of calculation may be reduced by reducing the resolution of the depth image.

[0115] The second step is to fuse the processed depth image and the feature map to obtain a fused feature. In practice, the execution subject may add the pixel features of the processed depth image and the feature map to obtain a fused feature.

[0116] The third step is to input the above fusion features into the first layer of the building blocks in each layer of the building blocks to obtain the encoding result as the current encoding result. In practice, each layer of the building blocks can be encoded by the following formula:

[0117] Block(X)=X+PwConv(LayerNorm(DwConv(X))).

[0118] Among them, Block(X) represents the encoding result. X represents the fused feature. DwConv represents the depth convolution. LayerNorm represents the layer normalization layer. PwConv represents the point convolution.

[0119] In the fourth step, the second layer building block among the above-mentioned layers of building blocks is determined as the current building block.

[0120] Step 5: Based on the current encoding result and the current building block, perform the following encoding steps:

[0121] In the first sub-step, the current encoding result is input into the current building block to obtain the encoding result as the target encoding result. In practice, the current encoding result can be input into the encoding formula in the third step as a fusion feature to obtain the target encoding result.

[0122] In the second sub-step, in response to determining that the current building block is the last building block, the target encoding result is determined as a memory feature. Thus, the encoding result of the last layer can be used as the final memory feature.

[0123] The third sub-step is, in response to determining that the current building block is not the last building block, taking the next building block of the current building block as the current building block, taking the above target encoding result as the current encoding result, and performing the above encoding step again.

[0124] Optionally, the above-mentioned looping step may further include: storing the memory features in a memory storage module, thereby saving the features and depth information of the current frame to be used as the memory features of the historical frame in the next loop.

[0125] As an example, the model structure of the deep image generation model can be referenced Figure 3 . Figure 3 In , the deep image generation model includes an encoder, an attention module, a memory encoding module, a memory storage module and a decoder. The encoder includes 4 feature extraction layers. The decoder includes 4 feature decoding layers corresponding to the 4 feature extraction layers.

[0126] Optionally, the execution subject may further combine the obtained depth images into a depth image sequence corresponding to the image sequence. In practice, the execution subject may combine the obtained depth images into a depth image sequence corresponding to the image sequence in the order of the corresponding time ascending order. Thus, a depth image sequence with a time sequence may be obtained.

[0127] Optionally, the execution subject may also skip the attention module and input the feature map to the decoder to obtain a depth image in response to determining that the loop step does not meet the preset loop number condition. The loop number condition may be that the loop number of the loop step is less than or equal to the preset loop number. Thus, the attention module may be skipped to directly output the depth map when there is no historical memory or less historical memory in the first n frames. The attention module is used to process the historical frame features and the current feature frame only after the preset loop number condition is met, so as to obtain a feature map with time consistency and thus obtain a more stable and accurate depth estimation.

[0128] The above-mentioned embodiments of the present disclosure have the following beneficial effects: through the depth image generation method of some embodiments of the present disclosure, a depth estimation algorithm framework with a memory mechanism capable of maintaining the temporal consistency of image sequence reasoning is provided, thereby improving the accuracy of each depth image generated for a temporal image sequence. Specifically, the reason for the poor accuracy of each depth image generated is that it mainly relies on a single image data as input, lacks temporal sequence, and the model does not consider the temporal dependence and spatial consistency between adjacent frames, resulting in a large jump in the predicted depth values ​​of the same pixel when the depth prediction is applied to a temporal image sequence (such as a video), resulting in poor accuracy of each depth image generated. Based on this, the depth image generation method of some embodiments of the present disclosure, first, performs a conversion operation on each image in a temporal image sequence to obtain an image information sequence. Thus, each temporal image can be converted into an information form that is convenient for the model to read. Then, for each image information in the above image information sequence, the following loop steps are performed: First, the above image information is input into the encoder included in the pre-trained deep image generation model to obtain a feature map, wherein the above deep image generation model includes an encoder, a memory module and a decoder. Thus, the encoder can be used to extract features from the current image. Second, the feature map is input into the memory module to perform attention operations on the feature map and the target memory feature to obtain an updated feature map. Thus, the historical information can be fused with the feature map of the current frame to enhance the feature map of the current frame. Third, the updated feature map is input into the decoder to obtain a depth image. Thus, when the decoder generates a depth prediction of the current frame, it can inherit the feature information of the historical frame and maintain a smooth depth transition with the historical frame. Thus, the features and depth information of the current frame can be saved to serve as memory features of the historical frame in the next cycle. Also because the attention module of the deep image generation model can fuse the historical information with the feature map of the current frame to enhance the feature map of the current frame, when the decoder generates a depth prediction of the current frame, it can inherit the feature information of the historical frame and maintain a smooth depth transition with the historical frame, thereby providing a depth estimation algorithm framework with a memory mechanism that can maintain the temporal consistency of image sequence reasoning, thereby improving the accuracy of each depth image generated for a temporal image sequence.

[0129] Further references Figure 4 , which shows a training process 400 of a depth image generation model of a depth image generation method. The training process 400 of the depth image generation model includes the following steps:

[0130] Step 401: Acquire each group of sample image sequences and sample depth image sequences with a time sequence.

[0131] In some embodiments, the execution entity (e.g., Figure 1 The server shown in the figure can obtain each group of sample image sequences and sample depth image sequences with time sequence from the database through a wired connection or a wireless connection. Among them, the sample depth image sequence in each group corresponds to the sample image sequence. Specifically, each sample depth image in the sample depth image sequence can correspond to each sample image in the sample image sequence one by one. That is, the sample depth image is a depth image of the corresponding sample image.

[0132] Step 402: perform image preprocessing on each group of sample image sequences and sample depth image sequences.

[0133] In some embodiments, the execution entity may perform image preprocessing on each group of sample image sequences and sample depth image sequences. In practice, for each group of sample image sequences and sample depth image sequences, the execution entity may perform image size transformation on each image in the sample image sequence and the sample depth image sequence, and then may perform image standardization, and then may convert the image into a tensor. The image size transformation may be a transformation into a preset image size. The tensor may be a PyTorch tensor. The tensor corresponding to the sample image may be used as sample image information. The tensor corresponding to the sample depth image may be used as sample depth image information.

[0134] Optionally, for each sample depth image, in response to determining that the type of the sample depth image is black-front and white-back, the sample depth image may be subjected to value inversion processing to adapt to the camera coordinate system. Black-front and white-back may be understood as the foreground being black and the background being white. For example, the value inversion processing may be a processing of inverting a pixel value of 0 to 1 and a pixel value of 1 to 0.

[0135] Step 403: Generate a sample group based on the sample image sequence and the sample depth image sequence after the preprocessing of each group of images.

[0136] In some embodiments, the execution entity may generate a sample group based on the sample image sequence and sample depth image sequence after preprocessing each group of images. Each sample in the sample group includes sample image information and sample depth image information corresponding to the sample image. In practice, for each sample image in the sample image sequence, the execution entity may combine the sample image information corresponding to the sample image and the sample depth image information of the sample depth image corresponding to the sample image into one sample. Thus, a sample group corresponding to each group of sample image sequence and sample depth image sequence may be obtained.

[0137] Step 404 , based on the generated sample groups, the initial depth image generation model is trained with the sample image information in each sample group as input and the sample depth image information corresponding to the sample image as the desired output to obtain a trained depth image generation model.

[0138] In some embodiments, the above-mentioned execution subject can train the initial depth image generation model based on the generated sample groups, with the sample image information in each sample group as input and the sample depth image information corresponding to the sample image as the expected output, to obtain a trained depth image generation model. In practice, the above-mentioned execution subject can train by estimating the entire time series image in sequence, calculating the loss function and regressing. Each batch contains an image sequence with a time series, that is, a sample group. The depth of the image is estimated and the loss function is calculated in time sequence. The loss function of the entire sequence is averaged and the model is gradient regressed. The model structure and processing process of the initial depth image generation model can refer to Figure 2 The model structure and processing process of the deep image generation model in the corresponding embodiments are not described in detail here. Thus, image data of a time series can be imported in a batch, and a single image can be estimated during the training process.

[0139] Further references Figure 5 As an implementation of the methods shown in the above figures, the present disclosure provides some embodiments of a depth image generation device, and these device embodiments are Figure 2 Corresponding to the method embodiments shown, the device can be specifically applied to various electronic devices.

[0140] like Figure 5 As shown, the depth image generation device 500 of some embodiments includes: a conversion unit 501 and an execution unit 502. The conversion unit 501 is configured to perform a conversion operation on each image in a time-sequential image sequence to obtain an image information sequence; the execution unit 502 is configured to perform the following loop steps for each image information in the above image information sequence: input the above image information into an encoder included in a pre-trained depth image generation model to obtain a feature map, wherein the above depth image generation model includes an encoder, a memory module and a decoder; input the above feature map into the above memory module to perform an attention operation on the above feature map and the target memory feature to obtain an updated feature map; input the above updated feature map into the above decoder to obtain a depth image.

[0141] Optionally, the depth image generating device 500 may further include: a combining unit (not shown in the figure), configured to combine the obtained depth images into a depth image sequence corresponding to the above-mentioned image sequence.

[0142] Optionally, the above-mentioned memory module may include an attention module.

[0143] Optionally, the execution unit 502 can be further configured to: input the above-mentioned feature map into the above-mentioned attention module to perform an attention operation on the above-mentioned feature map and the target memory feature to obtain an updated feature map.

[0144] Optionally, the above-mentioned memory module may also include a memory encoding module and a memory storage module.

[0145] Optionally, the above-mentioned loop step may further include: an input unit and a storage unit (not shown in the figure). The input unit is configured to input the obtained depth image and the above-mentioned feature map into the above-mentioned memory encoding module to obtain the memory feature. The storage unit is configured to store the above-mentioned memory feature into the above-mentioned memory storage module.

[0146] Optionally, the above-mentioned loop step may also include: a skipping unit (not shown in the figure), which is configured to skip the above-mentioned attention module in response to determining that the above-mentioned loop step does not meet the preset number of loop conditions, and input the above-mentioned feature map into the above-mentioned decoder to obtain a depth image.

[0147] Optionally, the encoder comprises sequential feature extraction layers.

[0148] Optionally, the execution unit 502 can be further configured to: determine the first feature extraction layer among the above-mentioned feature extraction layers as the current feature extraction layer; determine the above-mentioned image information as the current image information; based on the current feature extraction layer and the current image information, perform the following feature extraction steps: input the current image information into the current feature extraction layer to obtain the current feature map; in response to determining that the current feature extraction layer is the last feature extraction layer, determine the current feature map as the feature map; in response to determining that the current feature extraction layer is not the last feature extraction layer, use the next feature extraction layer of the current feature extraction layer as the current feature extraction layer, use the current feature map as the current image information, and perform the above-mentioned feature extraction steps again.

[0149] Optionally, the decoder includes feature decoding layers corresponding to the feature extraction layers.

[0150] Optionally, the execution unit 502 can be further configured to: for each of the above-mentioned feature extraction layers, input the current feature map or the updated feature map corresponding to the above-mentioned feature extraction layer into the feature decoding layer corresponding to the above-mentioned feature extraction layer to obtain a depth feature map; and generate a depth image based on the obtained depth feature maps.

[0151] Optionally, the execution unit 502 may be further configured to: perform fusion processing on the above-mentioned depth feature maps to obtain a fused depth feature map; and perform normalization processing on the above-mentioned fused depth feature map to obtain a depth image.

[0152] Optionally, the above-mentioned attention module includes various attention layers.

[0153] Optionally, the execution unit 502 can be further configured to: input the above-mentioned feature map to the first attention layer among the above-mentioned attention layers, and obtain the output feature as the current output feature; determine the second attention layer among the above-mentioned attention layers as the current attention layer; based on the current attention layer and the current output feature, perform the following attention operations: based on the current output feature and the position code corresponding to the above-mentioned feature map, perform self-attention mechanism processing on the current output feature at the current attention layer to obtain the first output feature; based on the above-mentioned first output feature, the above-mentioned target memory feature, the position code corresponding to the above-mentioned feature map, and the position code corresponding to the above-mentioned target memory feature, perform cross-attention mechanism processing on the current output feature and the target memory feature at the current attention layer to obtain the second output feature; based on the above-mentioned second output feature, generate the output feature corresponding to the current attention layer as the target output feature; in response to determining that the current attention layer is the last attention layer, normalize the above-mentioned target output feature to obtain an updated feature map; in response to determining that the current attention layer is not the last attention layer, take the next attention layer of the current attention layer as the current attention layer, take the target output feature as the current output feature, and perform the above-mentioned attention operation again.

[0154] Optionally, the execution unit 502 can be further configured to: based on the above-mentioned second output feature, generate the output feature of the current output feature at the current attention layer as the third output feature; determine the sum of the above-mentioned second output feature and the above-mentioned third output feature as the output feature corresponding to the current attention layer.

[0155] Optionally, the memory encoding module includes various layers of building blocks.

[0156] Optionally, the execution unit 502 can be further configured to: perform resolution preprocessing on the above-mentioned depth image to obtain a processed depth image; perform fusion processing on the above-mentioned processed depth image and the above-mentioned feature map to obtain a fusion feature; input the above-mentioned fusion feature into the first layer building block of the above-mentioned layers of building blocks to obtain a coding result as the current coding result; determine the second layer building block of the above-mentioned layers of building blocks as the current building block; based on the current coding result and the current building block, perform the following coding steps: input the current coding result into the current building block to obtain a coding result as a target coding result; in response to determining that the current building block is the last building block, determine the above-mentioned target coding result as a memory feature; in response to determining that the current building block is not the last building block, use the next building block of the current building block as the current building block, use the above-mentioned target coding result as the current coding result, and perform the above-mentioned coding steps again.

[0157] Optionally, the above-mentioned depth image generation model can be obtained through the following training steps: obtaining each group of sample image sequences and sample depth image sequences with a time sequence, wherein the sample depth image sequence in each group corresponds to the sample image sequence; performing image preprocessing on each group of sample image sequences and sample depth image sequences; for each group of sample image sequences and sample depth image sequences after image preprocessing, generating a sample group, wherein each sample in the sample group includes sample image information and sample depth image information corresponding to the sample image information; based on the generated sample groups, taking the sample image information in each sample group as input and the sample depth image information corresponding to the sample image as the expected output, training the initial depth image generation model to obtain a trained depth image generation model.

[0158] It is understood that the units described in the device 500 are similar to those described in the reference Figure 2 Therefore, the operations, features and beneficial effects described above for the method are also applicable to the device 500 and the units included therein, and will not be described in detail here.

[0159] Reference below Figure 6 , which shows an electronic device 600 (eg, Figure 1 A schematic diagram of the structure of a server or terminal device in FIG. Figure 6 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0160] like Figure 6As shown, the electronic device 600 may include a processing device 601 (e.g., a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 602 or a program loaded from a storage device 608 into a random access memory (RAM) 603. In the RAM 603, various programs and data required for the operation of the electronic device 600 are also stored. The processing device 601, the ROM 602, and the RAM 603 are connected to each other via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0161] Typically, the following devices may be connected to the I / O interface 605: an input device 606 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 607 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 608 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 609. The communication device 609 may allow the electronic device 600 to communicate with other devices wirelessly or by wire to exchange data. Although Figure 6 The electronic device 600 is shown with various devices, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead. Figure 6 Each block shown in the figure may represent one device, or may represent multiple devices as required.

[0162] In particular, according to some embodiments of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, some embodiments of the present disclosure include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In some such embodiments, the computer program can be downloaded and installed from the network through the communication device 609, or installed from the storage device 608, or installed from the ROM 602. When the computer program is executed by the processing device 601, the above-mentioned functions defined in the method of some embodiments of the present disclosure are executed.

[0163] It should be noted that the computer-readable medium recorded in some embodiments of the present disclosure may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In some embodiments of the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device. In some embodiments of the present disclosure, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. The computer readable signal medium may also be any computer readable medium other than a computer readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0164] In some embodiments, the client and the server may communicate using any currently known or future developed network protocol such as HTTP (HyperText Transfer Protocol), and may be interconnected with any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), an internet (e.g., the Internet), and a peer-to-peer network (e.g., an ad hoc peer-to-peer network), as well as any currently known or future developed network.

[0165] The computer-readable medium may be included in the electronic device; or it may exist independently without being installed in the electronic device. The computer-readable medium carries one or more programs. When the one or more programs are executed by the electronic device, the electronic device: performs conversion operations on each image in a time-sequential image sequence to obtain an image information sequence; for each image information in the image information sequence, performs the following loop steps: inputs the image information into an encoder included in a pre-trained deep image generation model to obtain a feature map, wherein the deep image generation model includes an encoder, a memory module and a decoder; inputs the feature map into the memory module to perform attention operations on the feature map and the target memory feature to obtain an updated feature map; inputs the updated feature map into the decoder to obtain a depth image.

[0166] Computer program code for performing the operations of some embodiments of the present disclosure may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0167] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0168] The units described in some embodiments of the present disclosure may be implemented by software or hardware. The units described may also be provided in a processor, for example, may be described as: a processor including a conversion unit and an execution unit. The names of these units do not, in some cases, constitute limitations on the units themselves, for example, the conversion unit may also be described as "a unit that performs conversion operations on each image in a time-series image sequence to obtain an image information sequence".

[0169] The functions described above herein may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like.

[0170] Some embodiments of the present disclosure further provide a computer program product, including a computer program, which implements any of the above-mentioned depth image generation methods when executed by a processor.

[0171] The above descriptions are only some preferred embodiments of the present disclosure and an explanation of the technical principles used. Those skilled in the art should understand that the scope of the invention involved in the embodiments of the present disclosure is not limited to the technical solutions formed by a specific combination of the above-mentioned technical features, but should also cover other technical solutions formed by any combination of the above-mentioned technical features or their equivalent features without departing from the above-mentioned inventive concept. For example, the above-mentioned features are replaced with the technical features with similar functions disclosed in the embodiments of the present disclosure (but not limited to) and the technical solutions formed.

Claims

1. A method for generating a depth image, comprising: Performing a conversion operation on each image in the time-sequential image sequence to obtain an image information sequence; For each image information in the image information sequence, the following loop steps are performed: Inputting the image information into an encoder included in a pre-trained deep image generation model to obtain a feature map, wherein the deep image generation model includes an encoder, a memory module and a decoder; Inputting the feature map into the memory module to perform an attention operation on the feature map and the target memory feature to obtain an updated feature map; The updated feature map is input into the decoder to obtain a depth image.

2. The method according to claim 1, wherein: The method further comprises: The obtained depth images are combined into a depth image sequence corresponding to the image sequence.

3. The method according to claim 1, wherein: The memory module includes an attention module; And the step of inputting the feature map into the memory module to perform an attention operation on the feature map and the target memory feature to obtain an updated feature map comprises: The feature map is input into the attention module to perform an attention operation on the feature map and the target memory feature to obtain an updated feature map.

4. The method according to claim 3, wherein: The memory module also includes a memory encoding module and a memory storage module; And the cycle step also includes: Inputting the obtained depth image and the feature map into the memory encoding module to obtain memory features; The memory characteristics are stored in the memory storage module.

5. The method according to claim 4, wherein: The cycle step also includes: In response to determining that the loop step does not meet a preset loop number condition, skipping the attention module, and inputting the feature map into the decoder to obtain a depth image.

6. The method according to claim 1, wherein: The encoder includes feature extraction layers having a sequence; And the image information is input into an encoder included in a pre-trained deep image generation model to obtain a feature map, including: Determine the first feature extraction layer among the feature extraction layers as the current feature extraction layer; Determining the image information as current image information; Based on the current feature extraction layer and the current image information, perform the following feature extraction steps: Input the current image information into the current feature extraction layer to obtain the current feature map; In response to determining that the current feature extraction layer is the last feature extraction layer, determining the current feature map as the feature map; In response to determining that the current feature extraction layer is not the last feature extraction layer, the next feature extraction layer of the current feature extraction layer is used as the current feature extraction layer, the current feature map is used as the current image information, and the feature extraction step is performed again.

7. The method according to claim 6, wherein: The decoder comprises respective feature decoding layers corresponding to the respective feature extraction layers; And the updating feature map is input into the decoder to obtain a depth image, comprising: For each of the feature extraction layers, inputting a current feature map or an updated feature map corresponding to the feature extraction layer into a feature decoding layer corresponding to the feature extraction layer to obtain a depth feature map; Based on the obtained depth feature maps, a depth image is generated.

8. The method according to claim 7, wherein: The generating a depth image based on each obtained depth feature map includes: Performing fusion processing on the depth feature maps to obtain a fused depth feature map; The fused depth feature map is normalized to obtain a depth image.

9. The method according to claim 3, wherein: The attention module includes various attention layers; And the step of inputting the feature map into the attention module to perform an attention operation on the feature map and the target memory feature to obtain an updated feature map comprises: Inputting the feature map into the first attention layer of each attention layer to obtain an output feature as a current output feature; Determine a second attention layer among the attention layers as a current attention layer; Based on the current attention layer and the current output features, the following attention operations are performed: Based on the current output feature and the position code corresponding to the feature map, the current output feature is processed by a self-attention mechanism in the current attention layer to obtain a first output feature; Based on the first output feature, the target memory feature, the position code corresponding to the feature map, and the position code corresponding to the target memory feature, a cross attention mechanism is performed on the current output feature and the target memory feature in the current attention layer to obtain a second output feature; Based on the second output feature, generating an output feature corresponding to the current attention layer as a target output feature; In response to determining that the current attention layer is the last attention layer, normalizing the target output feature to obtain an updated feature map; In response to determining that the current attention layer is not the last attention layer, the next attention layer of the current attention layer is used as the current attention layer, the target output feature is used as the current output feature, and the attention operation is performed again.

10. The method according to claim 9, wherein: The step of generating an output feature corresponding to the current attention layer as a target output feature based on the second output feature includes: Based on the second output feature, generating an output feature of the current output feature at the current attention layer as a third output feature; The sum of the second output feature and the third output feature is determined as the output feature corresponding to the current attention layer.

11. The method according to claim 4, wherein: The memory encoding module includes various layers of building blocks; And the step of inputting the obtained depth image and the feature map into the memory encoding module to obtain the memory feature comprises: Performing resolution preprocessing on the depth image to obtain a processed depth image; Performing fusion processing on the processed depth image and the feature map to obtain fusion features; Inputting the fused features into the first layer building block in each layer building block to obtain a coding result as the current coding result; Determine the second layer building block among the layers of building blocks as the current building block; Based on the current encoding result and the current building block, the following encoding steps are performed: Input the current encoding result into the current building block to obtain the encoding result as the target encoding result; In response to determining that the current building block is the last building block, determining the target encoding result as a memory feature; In response to determining that the current building block is not the last building block, taking the next building block of the current building block as the current building block, taking the target encoding result as the current encoding result, and performing the encoding step again.

12. The method according to any one of claims 1 to 11, wherein: The deep image generation model is obtained through the following training steps: Acquire each group of sample image sequences and sample depth image sequences having a time sequence, wherein the sample depth image sequence in each group corresponds to the sample image sequence; Performing image preprocessing on each group of sample image sequences and sample depth image sequences; For each group of image preprocessing sample image sequence and sample depth image sequence, a sample group is generated, wherein each sample in the sample group includes sample image information and sample depth image information corresponding to the sample image information; Based on the generated sample groups, the initial depth image generation model is trained with the sample image information in each sample group as input and the sample depth image information corresponding to the sample image as the expected output to obtain a trained depth image generation model.

13. A depth image generating device, comprising: A conversion unit, configured to perform a conversion operation on each image in a time-sequential image sequence to obtain an image information sequence; The execution unit is configured to perform the following loop steps for each image information in the image information sequence: input the image information into an encoder included in a pre-trained deep image generation model to obtain a feature map, wherein the deep image generation model includes an encoder, a memory module and a decoder; input the feature map into the memory module to perform an attention operation on the feature map and a target memory feature to obtain an updated feature map; input the updated feature map into the decoder to obtain a depth image.

14. An electronic device comprising: one or more processors; a storage device having one or more programs stored thereon, When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 12.

15. A computer readable medium having a computer program stored thereon, wherein: When the computer program is executed by a processor, the method according to any one of claims 1 to 12 is implemented.

16. A computer program product comprising a computer program, which, when executed by a processor, implements the method according to any one of claims 1 to 12.

Images