Method for training image encoder, image processing method and computing device

Abstract

A method of training an image encoder, a method of image processing, and a computing device are provided. The training image coding method comprises the following steps: respectively dividing an RGB image and a corresponding original depth image into a plurality of first image blocks and a plurality of second image blocks; through the embedding operation, obtaining a plurality of first embedding vectors corresponding to the plurality of first blocks and a plurality of second embedding vectors corresponding to the plurality of second blocks; then, removing a part of the plurality of second embedded vectors to obtain a plurality of second embedded vectors; and inputting the plurality of first embedded vectors and the plurality of second embedded vectors into a Transform-based encoder, and carrying out self-attention processing in the encoder to obtain a fused representation vector group. Using a decoder to obtain a reconstructed depth image according to the representation vector group; and updating the encoder and the decoder according to the original depth image and the reconstructed depth image.

Description

Technical Field

[0001] This specification relates to one or more embodiments in the fields of artificial intelligence and machine learning, and more particularly to methods for training image encoders, image processing methods, and corresponding computing devices. Background Technology

[0002] A depth map is a special type of image that records the distance information from each point in a scene to the camera. It encodes the geometric relationships in three-dimensional space into a two-dimensional grayscale or pseudo-color image, with each pixel value corresponding to the depth value of the object's surface. In practice, depth maps can be acquired using devices such as LiDAR and binocular stereo vision sensors. These devices generate depth maps by actively emitting light or passively calculating distances using parallax.

[0003] Depth maps have wide applications in various technological scenarios. In the field of autonomous driving, they help vehicles perceive the distance to surrounding obstacles, enabling path planning and safe obstacle avoidance; in AR / VR, depth maps support spatial anchoring and occlusion calculation, allowing virtual objects to blend naturally with the real environment; smartphones use depth maps to achieve background blurring effects in portrait mode; and so on.

[0004] Compared to color images, depth maps are unaffected by lighting and shadows, are sensitive to texture defects, reflective surfaces, and transparent objects, and provide stable geometric information. However, depth maps typically have lower resolution, making it difficult to accurately capture fine structures and prone to holes or noise. Therefore, in many scenarios, depth maps are often processed together with color images to obtain comprehensive environmental data.

[0005] To address this, the RGBD model was proposed. The RGBD model is a multimodal deep learning architecture that simultaneously processes RGB color images and depth maps. By fusing color texture and geometric distance information, it improves the accuracy of scene understanding. The RGBD model is widely used in scenarios such as indoor semantic segmentation (identifying furniture, walls, etc.), transparent object detection (solving the depth loss problem of reflective surfaces such as glass), video object segmentation (using depth cues to handle occlusion), and robot grasping (combining color and geometric information to estimate pose).

[0006] However, existing RGBD models are insufficient in modeling the semantic relationships between RGB information and depth map information when fusing RGB color images with depth maps, and their performance needs further improvement. Summary of the Invention

[0007] This specification provides one or more embodiments of a method for training an image encoder that can better model the semantic relationship between RGB information and depth map information, thereby improving the representational power of the encoding.

[0008] According to the first aspect, a method for training an image encoder is provided, comprising: The RGB image and the corresponding original depth image are divided into multiple first patches and multiple second patches, respectively; Through the embedding operation, multiple first embedding vectors corresponding to multiple first tiles and multiple second embedding vectors corresponding to multiple second tiles are obtained; A subset of the plurality of second embedding vectors is removed to obtain a plurality of second embedding vectors; The plurality of first embedding vectors and the plurality of second embedding vectors are input into a Transformer-based encoder, and self-attention processing is performed in the encoder to obtain a fused representation vector group. The decoder is used to obtain the reconstructed depth image based on the representation vector set; The model is updated based on the original depth image and the reconstructed depth image, and the model update includes updating the encoder and decoder.

[0009] According to the second aspect, an image processing method is provided, comprising: Obtain the encoder trained according to the method in the first aspect; The target RGB image and the corresponding target depth image are divided into multiple first patches and multiple second patches, respectively; Through the embedding operation, multiple first embedding vectors corresponding to multiple first tiles and multiple second embedding vectors corresponding to multiple second tiles are obtained; The plurality of first embedding vectors and the plurality of second embedding vectors are input into the encoder, where self-attention processing is performed to obtain a fused representation vector group. Based on the set of representation vectors, perform the target task related to the depth map.

[0010] According to a third aspect, an apparatus for training an image encoder is provided, comprising: The tile division unit is configured to divide the RGB image and the corresponding original depth image into multiple first tiles and multiple second tiles, respectively; The embedding operation unit is configured to obtain multiple first embedding vectors corresponding to multiple first map tiles and multiple second embedding vectors corresponding to multiple second map tiles through embedding operations; The vector elimination unit is configured to eliminate a portion of the plurality of second embedded vectors to obtain a plurality of second embedded vectors; The encoding unit is configured to input the plurality of first embedding vectors and the plurality of second embedding vectors into a Transformer-based encoder, and perform self-attention processing in the encoder to obtain a fused representation vector group. The decoding unit is configured to use the decoder to obtain a reconstructed depth image based on the representation vector set; The model update unit is configured to update the model based on the original depth image and the reconstructed depth image, wherein the model update includes updating the encoder and the decoder.

[0011] According to a fourth aspect, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed in a computer, causes the computer to perform the method described in the first or second aspect.

[0012] According to a fifth aspect, a computer system is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that the processor implements the method of the first aspect or the second aspect when executing the program.

[0013] In the embodiments of this specification, during the training of the image encoder, since the tile tokens of the depth map of the input encoder are masked or removed, the encoder and decoder can only rely on the context of the tile tokens of the RGB image and the remaining depth map tokens to infer the missing depth information. This forces the encoder to inject depth geometric information into the RGB representation, so as to better model and learn the semantic relationship between RGB features and depth map features. Attached Figure Description

[0014] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0015] Figure 1 This is a schematic diagram of training an image encoder in one embodiment; Figure 2 A flowchart illustrating the training of an image encoder according to one embodiment is shown; Figure 3A This diagram illustrates the effect of random elimination. Figure 3B This diagram illustrates the effect of large-area removal. Figure 4 A schematic diagram of encoder processing is shown in one embodiment; Figure 5 A flowchart illustrating an image processing method according to one embodiment is shown; Figure 6 A schematic block diagram of the training device is shown; Figure 7 shows an example computer system 1000. Detailed Implementation

[0016] The solution provided in this specification will now be described with reference to the accompanying drawings.

[0017] The RGBD model integrates RGB color images and depth maps through multimodal fusion. Among related technologies, the multimodal fusion strategies of the RGBD model are mainly divided into three categories.

[0018] One type is early fusion, which involves stitching the depth map as an additional channel with the RGB image (3 channels) to form a 4-channel input, which is then directly fed into the neural network model. This method is computationally efficient, but it struggles to handle feature differences between modalities and is susceptible to depth noise.

[0019] The second type is mid-level fusion, which uses a dual-branch encoder to extract graph features from the RGB image and depth map separately, and then fuses them in the middle layer of the neural network. A typical example is FuseNet, which uses two parallel encoders to inject deep features layer by layer into the RGB branch through element-wise summation, giving the RGB feature map geometric perception capabilities. However, the direct element-wise summation method is difficult to capture and understand the deep semantics of the image.

[0020] The third type is late fusion, which involves independently extracting high-level semantic features from the RGB image and depth map, fusing these features at the decision layer, and then predicting the result. This approach lacks deep interaction between RGBD information, has poor generalization ability, and is susceptible to noise interference.

[0021] Furthermore, some related technical solutions propose using a cross-attention mechanism to fuse cross-modal features from RGB images and depth maps. However, this approach retains the complete depth map input during model training, causing the model to directly rely on the existing depth information at the input for prediction, without truly learning the semantic relationship between RGB information and depth map information. This results in weak generalization ability and the risk of shortcuts.

[0022] In view of this, this specification proposes a scheme for training an image encoder that can enhance the modeling and learning of semantic associations between RGB images and depth maps, and better integrate depth information into RGB information.

[0023] Figure 1 This is a schematic diagram of training an image encoder in one embodiment. Figure 1As shown, firstly, the RGB image and the original depth map are obtained for training. The RGB three-channel image is labeled R, G, and B respectively, while the depth map is labeled D. Then, the RGB image and the depth map D are each divided into N patches, and embedding processing is performed to obtain the embedding vector for each patch. Thus, the RGB image and the depth map each obtain N embedding vectors. Following the framework of the Transformer model, the embedding vectors of the patches will be referred to as token vectors, or simply tokens. For distinction, the embedding vectors corresponding to the RGB patches are denoted as token1, and the embedding vectors corresponding to the depth map patches are denoted as token2. Therefore, the RGB image yields a sequence T1 consisting of N token1s, and the depth map yields a sequence T2 consisting of N token2s.

[0024] For a sequence T2 corresponding to the depth map, a certain proportion of token2 are selected for masking or removal. Masked or removed token2s are shown in black in the image, while the remaining token2s are shown in gray. Assume that after masking or removal, L token2s remain, where L... <N。

[0025] Next, the sequence T1 of N RGB token1s and the sequence T2' of L token2s are concatenated to obtain the joint sequence TS. The joint sequence TS is then input into a Transformer-based encoder for attention processing, and the output is the attention-fused representation vector set H.

[0026] The decoder performs decoding based on the aforementioned representation vector set H to obtain the reconstructed depth map D'. By comparing the original depth map D and the reconstructed depth map D', the encoder and decoder can be trained.

[0027] In the above process, since some of the map tiles token2 in the depth map are removed, the encoder and decoder cannot directly copy the depth map information to reconstruct the depth map. Instead, they must infer the missing depth information through the context of RGB token1 and the remaining depth map token2, thus forcing the encoder to inject the depth geometry information into the RGB representation.

[0028] The following reference Figure 2 The flowchart and specific embodiments describe the process of training the image encoder.

[0029] To train the image encoder, a color image P and its corresponding original depth map D are first acquired as training samples. The color image P and depth map D are captured for the same environmental object. Then, the color image P is split into RGB channels to obtain an RGB image, including the red channel image R, the green channel image G, and the blue channel image B.

[0030] like Figure 2As shown in step S21, the RGB image and the corresponding original depth image are divided into multiple first patches and multiple second patches, respectively.

[0031] Typically, the same partitioning method is used to divide the RGB image and depth map D into patches. For example, each image is divided into patches according to m rows and k columns, resulting in N = m * k patches. For ease of distinction, the patches in the RGB image are called the first patch, and the patches in the depth map are called the second patch.

[0032] Next, in step S22, through the embedding operation, multiple first embedding vectors corresponding to multiple first tiles and multiple second embedding vectors corresponding to multiple second tiles are obtained.

[0033] The embedding operation described above aims to perform preliminary feature extraction and encoding of image patches, and can be implemented in various ways. In one embodiment, an existing encoding algorithm can be used for the embedding operation. In another embodiment, a trainable embedding network can be used to perform the embedding operation. This embedding network can be a shallow neural network with a limited number of layers. For example, the embedding network can be a shallow convolutional neural network (CNN) with fewer than a preset threshold, such as fewer than three layers. In another example, the embedding network can also employ a multilayer perceptron (MLP) for linear embedding operations.

[0034] Through the above embedding operation, the embedding vectors corresponding to each image patch can be obtained. Similarly, for the sake of distinction, the embedding vector corresponding to the first image patch is called the first embedding vector, hereinafter referred to as token1; and the embedding vector corresponding to the second image patch is called the second embedding vector, hereinafter referred to as token2. In practice, RGB three-channel images are generally embedded together, that is, each first embedding vector is a feature combination of the three channel image patches at the corresponding position. Thus, after the embedding operation, N first embedding vectors (token1) corresponding to N first image patches and N second embedding vectors (token2) corresponding to N second image patches can be obtained. The sequence of N first embedding vectors can be specifically represented as: {C 1, C2,…,C N}, where C 1i This represents the RGB three-channel feature corresponding to the i-th position. The N second embedding vectors can be represented as {D}. 1, D2,…,D N}

[0035] Next, in step S23, a portion of the multiple second embedding vectors is removed to obtain several second embedding vectors. That is, a portion of the N token2s obtained above is removed to obtain L token2s, where L... <N。

[0036] In terms of the number of tokens to be removed, this step can remove a portion of token2 according to a certain ratio, such as removing 75% of token2. This removal ratio can be a pre-set fixed ratio or a dynamically adjustable ratio.

[0037] As those skilled in the art will know, model training typically involves multiple training epochs, with a model update performed in each epoch. In one example, the aforementioned elimination ratio can increase with the number of training epochs. That is, the elimination ratio in the i-th training epoch can be higher than that in the (i-1)-th epoch. Alternatively, the elimination ratio can increase incrementally according to the training cycle, i.e., after each training cycle consisting of, for example, p training epochs, the elimination ratio increases by a certain value or percentage. This allows the model training to progress from easy to difficult, promoting stable convergence.

[0038] From the perspective of the removal method, this step can adopt random removal, rule-based removal, or large-block removal, etc.

[0039] Specifically, random removal involves randomly selecting a subset of tokens from N tokens to remove, resulting in L tokens. Rule-based removal, on the other hand, determines which tokens to remove according to a specific rule. This rule could be, for example, removing a token at regular intervals.

[0040] In the large block removal scheme, some token groups can be determined from the N token2s. The second tiles corresponding to multiple token2s included in the token group form a continuous region in the original depth map D, i.e., a large tile. Then, each token group is removed from the N token2s.

[0041] Figure 3A This diagram illustrates the effect of random elimination. Figure 3B This diagram illustrates the effect of large-area removal. Figure 3A and Figure 3B In the diagram, the second tile corresponding to the removed token2 is represented by a small black square. It can be seen that the random removal scheme is equivalent to randomly selecting some second tiles from the N second tiles in the original depth map and removing their corresponding token2 tiles. And... Figure 3B In the large block removal shown, the second block that is removed forms an even larger block.

[0042] In different training epochs of the model, the same elimination method can be used, or different elimination methods can be used in each epoch. When eliminating the same proportion of token2, compared with random elimination, large-block elimination increases the reconstruction difficulty because it masks more neighboring tile information. Therefore, it is preferable to use the large-block elimination method after a certain number of training epochs.

[0043] Through the above methods, some token2s can be masked or removed from N token2s to obtain the remaining L token2s, which are several second embedding vectors.

[0044] Next, in step S24, the multiple first embedding vectors and the aforementioned second embedding vectors are input into a Transformer-based encoder, where self-attention processing is performed to obtain a fused representation vector group.

[0045] Specifically, a sequence consisting of N tokens1 and L tokens2 can be concatenated to form a joint sequence. This joint sequence is then input into a Transformer-based encoder. The encoder performs self-attention processing on each sequence element in the input joint sequence, obtaining the fused attention representation vectors for each sequence element, which are then grouped into a representation vector set.

[0046] It should be understood that although the encoder processes according to the self-attention mechanism, since the joint sequence includes both token1 corresponding to the RGB image and token2 corresponding to the depth map, the self-attention processing of the joint sequence will fuse the information between token1 and token2, thus realizing cross-modal processing of RGB information and depth map information.

[0047] Following the processing approach of the Transformer architecture, by default, a special character token representing global information is appended to the input joint sequence. This special character token is often represented by the classification symbol CLS used for classification. After performing self-attention processing on the input sequence containing this special character token, a representation vector corresponding to this special character token can be obtained. This representation vector is a global representation vector that incorporates global attention information.

[0048] Therefore, the representation vector set obtained by the encoder includes the first representation vector h1 corresponding to each of the multiple tokens1 after fusion attention, the second representation vector h2 corresponding to each of the multiple tokens2 after fusion attention, and the global representation vector h that fuses global attention information. CLS .

[0049] In one embodiment, when inputting the above joint sequence into the encoder, positional encoding needs to be combined to enable the encoder to capture the positional relationship information between patches. Specifically, N tokens1 can be combined with their corresponding N first positional codes to obtain N first combination vectors; L tokens2 can be combined with their corresponding L second positional codes to obtain L second combination vectors. The total sequence of the N first combination vectors and the L second combination vectors is then input into the encoder as the processed joint sequence. Different images share the same positional encoding method. That is, in each first and second patch, two patches with the same position have the same positional code. Assuming a patch is located in the i-th row and j-th column of its original image, its positional code can be uniquely determined by the row number i and column number j, regardless of whether the patch is a first or second patch. In this way, the encoder can easily capture the positional information of each token in the RGB image and the depth image, and use this positional information for more effective attention processing.

[0050] Figure 4 The diagram illustrates the encoder's processing in one embodiment. As shown, the encoder's input includes N tokens: {C 1, C2,…,C N}, L tokens2 (exemplarily shown in the diagram as D2, D5, D 12 ), and the special character token: CLS. Each of token 1 and token 2 is concatenated with its corresponding position code (shown as a plus sign "+" in the diagram) to obtain a combined vector. Plot tokens at the same position have the same position code. For example, N tokens {C...} 1, C2,…,C N The corresponding position codes are {P} 1, P2,…,P N The three tokens exemplified here have position codes P2, P5, and P6 respectively. 12 .

[0051] The encoder performs self-attention processing on the special character token and the vector sequence consisting of N+L combined vectors to obtain the fused representation vectors corresponding to each sequence position, forming a representation vector group. This includes the global representation vector corresponding to CLS, the N first representation vectors h1 corresponding to the N tokens1, and the L second representation vectors h2 corresponding to the L tokens2.

[0052] Based on the representation vector set obtained by the encoder through self-attention fusion, Figure 2 In the next step S25, the decoder obtains the reconstructed depth image based on the above representation vector group.

[0053] Specifically, the decoder decodes the aforementioned representation vector set and reconstructs the image based on the decoding result to obtain a reconstructed depth image. The decoder can be implemented using various neural networks, and no limitation is made here. In one embodiment, the decoder can be implemented using a convolutional neural network, such as a U-Net CNN with multiple convolutional layers forming a U-shaped structure. In this case, the decoding process may include deconvolution processing, and optionally pooling processing, etc. In another embodiment, the decoder can be a Transformer-based decoder, such as a DETR-style decoder, which employs a decoding processing method similar to that of a Transformer.

[0054] In one implementation, the decoder decodes all the representation vectors in the representation vector group obtained by the encoder to obtain a reconstructed depth image.

[0055] In another implementation, only the global representation vector in the representation vector group and the first representation vector h1 corresponding to each token1 are input into the decoder for decoding. That is, in this implementation, the decoder cannot obtain any direct depth map information (token2 information) during decoding and reconstruction; it can only infer depth information based on global features and RGB features (represented by the first representation vector h1). This forces the encoder to fully integrate the feature information in the depth map (token2 information) into the RGB feature information, better encoding the semantic relationship between RGB information and depth map information.

[0056] After obtaining the reconstructed depth map, as shown in step S26, the model can be updated based on the original depth image and the reconstructed depth image. The model update includes at least jointly updating the encoder and decoder.

[0057] Specifically, in this step, the reconstruction loss can be determined based on the comparison between the original depth image and the reconstructed depth image. In one example, the L1 or L2 distance between the original depth image and the reconstructed depth image can be calculated as the reconstruction loss. In other implementations, other loss function forms can also be used to determine the reconstruction loss, such as calculating the perceptual loss or SSIM loss between the original depth image and the reconstructed depth image, etc. Furthermore, in the direction of decreasing reconstruction loss, the model parameters are updated through backpropagation, specifically including sequentially updating the model parameters in the decoder and the model parameters in the encoder.

[0058] It is understood that if the embedding operation in step S22 uses a trainable embedding network, the backpropagation described above continues, and the model update also includes updating the model parameters in the embedding network.

[0059] Understandable. Figure 2 This illustrates the processing of a set of samples (including RGB images and the original depth map) during a single training round. This is achieved by repeatedly performing this process across multiple iterations using a large number of samples. Figure 2 The process of training an encoder can yield a well-trained encoder.

[0060] Looking back at the training process above, we can see that during the training of the image encoder, because the tile tokens of the depth map input to the encoder are masked or removed, the encoder and decoder can only rely on the context of the tile tokens of the RGB image and the remaining depth map tokens to infer the missing depth information. This forces the encoder to inject depth geometric information into the RGB representation, so as to better model and learn the semantic relationship between RGB features and depth map features.

[0061] Based on this, in another embodiment, this specification also provides an image processing method that uses the encoder trained above to perform image processing. Figure 5 A flowchart illustrating an image processing method according to one embodiment is shown. Figure 5 As shown, the method includes the following steps.

[0062] In step S51, the encoder is obtained, which is based on the above combination Figure 2 The training method is described.

[0063] In step S52, the target RGB image and the corresponding target depth image are divided into multiple first patches and multiple second patches, respectively. Here, the target RGB image and the target depth image are the images to be processed. The patch division can be performed in a manner similar to that in step S22.

[0064] In practice, depth maps acquired by depth sensors often contain holes or gaps at object edges. Since the encoder is trained based on partial depth map patches during the aforementioned training process, it is permissible for the target depth image to contain some holes or gaps during image processing.

[0065] Next, in step S53, through the embedding operation, multiple first embedding vectors corresponding to multiple first tiles and multiple second embedding vectors corresponding to multiple second tiles are obtained. The processing procedure of this step is similar to that of step S23 above, and will not be repeated.

[0066] Further, in step S54, the above-mentioned multiple first embedding vectors and multiple second embedding vectors are input into the trained Transformer-based encoder, where self-attention processing is performed to obtain the fused representation vector group.

[0067] Slightly different from the training process, during image processing, it is not necessary to intentionally remove the depth map patch tokens (i.e., the second embedding vector). Instead, all of them can be input into the encoder to provide more comprehensive depth information and improve the encoding effect. The positional encoding methods of the RGB patch tokens (first embedding vector) and the depth map patch tokens, as well as the processing after inputting them into the encoder, are similar to those in the training process and will not be repeated here.

[0068] Next, in step S55, the target task related to the depth map is performed based on the representation vector set obtained by the encoder.

[0069] It is understandable that because the encoder is trained to inject depth map features into RGB image features more effectively and to fuse the two modalities more deeply, the resulting representation vector set has stronger representation capabilities and can be applied to a variety of downstream tasks.

[0070] In one embodiment, the objective task described above is depth map completion or enhancement, that is, repairing and refining the original depth map for sparse, noisy, or missing regions. This can be achieved by using a decoder obtained during training to perform depth map reconstruction, resulting in a reconstructed or restored depth map.

[0071] In another embodiment, the aforementioned target task can be a 3D scene understanding-based task, such as transparent object detection (using depth information to complete transparent and reflective objects in a color image) or robot grasping (combining color and geometric information to estimate pose).

[0072] In this way, by leveraging the encoder's semantic alignment and fusion capabilities with RGB features and depth map features, a set of representation vectors with stronger representation capabilities can be obtained, thereby improving the performance of various image processing tasks related to depth maps. According to another embodiment, an apparatus for training an image encoder is provided, which can be deployed in any computing device, platform, or device cluster with computing and processing capabilities. Figure 6 A schematic block diagram of the training device is shown. Figure 6 As shown, the training device 600 includes: The tile division unit 61 is configured to divide the RGB image and the corresponding original depth image into multiple first tiles and multiple second tiles, respectively; The embedding operation unit 62 is configured to obtain multiple first embedding vectors corresponding to multiple first tiles and multiple second embedding vectors corresponding to multiple second tiles through embedding operations. Vector elimination unit 63 is configured to eliminate a portion of the plurality of second embedded vectors to obtain a plurality of second embedded vectors; The encoding unit 64 is configured to input the plurality of first embedding vectors and the plurality of second embedding vectors into a Transformer-based encoder, and perform self-attention processing in the encoder to obtain a fused representation vector group. Decoding unit 65 is configured to use the decoder to obtain a reconstructed depth image based on the representation vector group; The model update unit 66 is configured to perform model updates based on the original depth image and the reconstructed depth image, wherein the model update includes updating the encoder and the decoder.

[0073] For ease of description, the above devices are described in terms of function, divided into various modules. Of course, in implementing this specification, the functions of each module can be implemented in one or more software and / or hardware.

[0074] This specification also provides a computer-readable non-volatile storage medium storing a computer program that, when executed by a processor, can be used to perform one or more steps of one or more methods described or illustrated herein, or to provide the functionality described or illustrated herein. In this document, a computer-readable non-volatile storage medium or medium may include one or more semiconductor or other integrated circuits (ICs) (e.g., field-programmable gate arrays (FPGAs) or application-specific integrated circuits (ASICs)), hard disk drives (HDDs), hybrid hard disk drives (HHDs), optical disks, optical disk drives (ODDs), magneto-optical disks, magneto-optical drives, floppy disks, floppy disk drives (FDDs), magnetic tape, solid-state drives (SSDs), RAM drives, secure digital cards or drives, any other suitable computer-readable non-volatile storage medium, or any suitable combination thereof where appropriate. A computer-readable non-volatile storage medium may be volatile, non-volatile, or a combination of volatile and non-volatile.

[0075] This specification also provides a computer system. Figure 7 An example computer system 1000 is illustrated. In a particular embodiment, one or more computer systems 1000 perform one or more steps of one or more methods described or illustrated herein. In a particular embodiment, one or more computer systems 1000 provide the functionality described or illustrated herein. In a particular embodiment, software running on one or more computer systems 1000 performs one or more steps of one or more methods described or illustrated herein, or provides the functionality described or illustrated herein. The particular embodiments include one or more portions of one or more computer systems 1000. References to computer systems herein may include computing devices, and vice versa, where appropriate. Furthermore, references to computer systems may include one or more computer systems, where appropriate.

[0076] This disclosure contemplates any suitable number of computer systems 1000. This disclosure contemplates computer systems 1000 taking any suitable physical form. By way of example and not limitation, computer system 1000 can be an embedded computer system, a system-on-a-chip (SOC), a single-board computer system (SBC) (e.g., a computer-on-module (COM) or system-on-module (SOM)), a desktop computer system, a notebook or laptop system, an interactive kiosk, a mainframe, a grid of computer systems, a mobile phone, a personal digital assistant (PDA), a server, a tablet computer system, or any suitable combination of these. Where appropriate, computer system 1000 can include one or more computer systems 1000; be single or distributed; span multiple locations; span multiple machines; span multiple data centers; or reside in the cloud, which may include one or more cloud components in one or more networks. Where appropriate, one or more computer systems 1000 can perform one or more steps of one or more methods described or illustrated herein without substantial spatial or temporal limitations. For example, and not by way of limitation, one or more computer systems 1000 can perform one or more steps of one or more methods described or illustrated herein in real-time or batch mode. One or more computer systems 1000 may perform one or more steps of the methods described or illustrated herein at different times or in different locations, where appropriate.

[0077] In a particular embodiment, computer system 1000 includes processor 1002, memory 1004, storage 1006, input / output (I / O) interface 1008, communication interface 1010, and bus 1012. Although this disclosure describes and illustrates a particular computer system having a particular number of particular components in a particular arrangement, this disclosure contemplates any suitable computer system having any suitable number of any suitable components in any suitable arrangement.

[0078] In certain embodiments, processor 1002 includes hardware for executing instructions, such as those that constitute a computer program. For example, but not limited to, to execute instructions, processor 1002 may retrieve (or fetch) instructions from internal registers, internal caches, memory 1004, or storage 1006; decode and execute them; and then write one or more results to internal registers, internal caches, memory 1004, or storage 1006. In certain embodiments, processor 1002 may include one or more internal caches for data, instructions, or addresses. This disclosure contemplates that processor 1002 may include any suitable number of suitable internal caches where appropriate. For example, but not limited to, processor 1002 may include one or more instruction caches, one or more data caches, and one or more translation back buffers (TLBs). Instructions in the instruction cache may be copies of instructions in memory 1004 or storage 1006, and the instruction cache may speed up the retrieval of these instructions by processor 1002. The data in the data cache may be a copy of data in memory 1004 or storage 1006 for operation by instructions executed at processor 1002; the result of instructions previously executed by processor 1002 for access or writing to memory 1004 or storage 1006 by subsequent instructions executed at processor 1002; or other suitable data. The data cache can speed up read or write operations on processor 1002. The TLB can speed up virtual address translation on processor 1002. In a particular embodiment, processor 1002 may include one or more internal registers for data, instructions, or addresses. This disclosure contemplates that processor 1002 may include any suitable number of suitable internal registers where appropriate. Where appropriate, processor 1002 may include one or more arithmetic logic units (ALUs); be a multi-core processor; or include one or more processors 1002. Although this disclosure describes and illustrates specific processors, this disclosure contemplates any suitable processor.

[0079] In a particular embodiment, memory 1004 includes main memory for storing instructions to be executed by processor 1002 or data to be operated on by processor 1002. For example, but not limited to, computer system 1000 may load instructions into memory 1004 from storage 1006 or other sources (e.g., another computer system 1000). Processor 1002 may then load instructions from memory 1004 into internal registers or internal caches. To execute instructions, processor 1002 may retrieve and decode instructions from internal registers or internal caches. During or after instruction execution, processor 1002 may write one or more results (potentially intermediate or final results) to internal registers or internal caches. Processor 1002 may then write one or more of these results into memory 1004. In a particular embodiment, processor 1002 executes instructions only in one or more internal registers or internal caches or memory 1004 (not in storage 1006 or elsewhere), and operates on data only in one or more internal registers or internal caches or memory 1004 (not in storage 1006 or elsewhere). One or more memory buses (each of which may include an address bus and a data bus) may couple processor 1002 to memory 1004. Bus 1012 may include one or more memory buses as described below. In a particular embodiment, one or more memory management units (MMUs) are located between processor 1002 and memory 1004 and facilitate access to memory 1004 requested by processor 1002. In a particular embodiment, memory 1004 includes random access memory (RAM). This RAM may be volatile memory, where appropriate. Where appropriate, this RAM may be dynamic RAM (DRAM) or static RAM (SRAM). Furthermore, where appropriate, this RAM may be single-port or multi-port RAM. This disclosure contemplates any suitable RAM. Memory 1004 may include one or more memory modules, where appropriate. Although this disclosure describes and illustrates specific memory, this disclosure contemplates any suitable memory.

[0080] In certain embodiments, storage 1006 includes mass storage for data or instructions. For example, but not limited to, storage 1006 may include a hard disk drive (HDD), floppy disk drive, flash memory, optical disk, magneto-optical disk, magnetic tape, or Universal Serial Bus (USB) drive, or a combination of two or more. Where appropriate, storage 1006 may include removable or non-removable (or fixed) media. Where appropriate, storage 1006 may be internal or external to computer system 1000. In certain embodiments, storage 1006 is a non-volatile solid-state memory. In certain embodiments, storage 1006 includes read-only memory (ROM). Where appropriate, this ROM may be a mask-programmable ROM, a programmable ROM (PROM), an erasable PROM (EPROM), an electrically erasable PROM (EEPROM), an electrically changeable ROM (EAROM), or flash memory, or a combination of two or more. This disclosure assumes that storage 1006 may take any suitable physical form. Where appropriate, storage 1006 may include one or more storage control units to facilitate communication between processor 1002 and storage 1006. Where appropriate, storage 1006 may include one or more storage units 1006. Although this disclosure describes and illustrates specific storage units, this disclosure considers any suitable storage unit.

[0081] In a particular embodiment, I / O interface 1008 includes hardware, software, or both, providing one or more interfaces between computer system 1000 and one or more I / O devices. Computer system 1000 may include one or more of these I / O devices, where appropriate. One or more of these I / O devices enable communication between a person and computer system 1000. For example, but not limited to, I / O devices may include a keyboard, keypad, microphone, display, mouse, printer, scanner, speaker, still camera, stylus, tablet computer, touchscreen, trackball, camera, other suitable I / O devices, or combinations thereof. I / O devices may include one or more sensors. This disclosure contemplates any suitable I / O devices and any suitable I / O interface 1008. Where appropriate, I / O interface 1008 may include one or more device or software drivers enabling processor 1002 to drive one or more of these I / O devices. I / O interface 1008 may include one or more I / O interfaces 1008, where appropriate. Although this disclosure describes and illustrates specific I / O interfaces, this disclosure considers any suitable I / O interface.

[0082] In a particular embodiment, communication interface 1010 includes hardware, software, or both (e.g., packet-based communication) that provides one or more communication interfaces between computer system 1000 and one or more other computer systems 1000 or one or more networks. For example, but not limited to, communication interface 1010 may include a network interface controller (NIC) or network adapter for communicating with Ethernet or other wired networks, or a wireless NIC (WNIC) or wireless adapter for communicating with wireless networks (e.g., Wi-Fi networks). This disclosure contemplates any suitable network and any suitable communication interface 1010. For example, but not limited to, computer system 1000 may communicate with one or more portions of an ad hoc network, a personal area network (PAN), a local area network (LAN), a wide area network (WAN), a metropolitan area network (MAN), or the Internet, or a combination of both. One or more portions of these networks may be wired or wireless. For example, computer system 1000 may communicate with a wireless personal area network (WPAN) (e.g., Bluetooth WPAN), a Wi-Fi network, a Wi-Max network, a cellular telephone network (e.g., a Global System for Mobile Communications (GSM) network), or other suitable wireless networks, or a combination of both. Where appropriate, computer system 1000 may include any suitable communication interface 1010 for any of these networks. Where appropriate, communication interface 1010 may include one or more communication interfaces 1010. Although specific communication interfaces are described and illustrated in this disclosure, any suitable communication interface is contemplated in this disclosure.

[0083] In a particular embodiment, bus 1012 includes hardware, software, or both that interconnect components of computer system 1000. For example, but not limited to, bus 1012 may include an Accelerated Graphics Port (AGP) or other graphics bus, an Enhanced Industry Standard Architecture (EISA) bus, a Front Side Bus (FSB), a HyperTransport (HT) interconnect, an Industry Standard Architecture (ISA) bus, an Infiniband interconnect, a Low Pin Count (LPC) bus, a memory bus, a Micro Channel Architecture (MCA) bus, a Peripheral Component Interconnect (PCI) bus, a PCI-Express (PCIe) bus, a Serial Advanced Technology Attachment (SATA) bus, a Video Electronics Standards Association Local (VLB) bus, or other suitable buses, or combinations thereof. Where appropriate, bus 1012 may include one or more buses 1012. Although this disclosure describes and illustrates specific buses, this disclosure contemplates any suitable bus or interconnect.

[0084] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0085] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to interchangeably. Each embodiment focuses on describing the differences from other embodiments. In particular, the system embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments.

[0086] The above description is merely an embodiment of this specification and is not intended to limit this specification. Various modifications and variations can be made to this specification by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this specification should be included within the scope of the claims of this application.

Claims

1. A method for training an image encoder, comprising: The RGB image and the corresponding original depth image are divided into multiple first patches and multiple second patches, respectively; Through the embedding operation, multiple first embedding vectors corresponding to multiple first tiles and multiple second embedding vectors corresponding to multiple second tiles are obtained; A subset of the plurality of second embedding vectors is removed to obtain a plurality of second embedding vectors; The plurality of first embedding vectors and the plurality of second embedding vectors are input into a Transformer-based encoder, and self-attention processing is performed in the encoder to obtain a fused representation vector group. The decoder is used to obtain the reconstructed depth image based on the representation vector set; The model is updated based on the original depth image and the reconstructed depth image, and the model update includes updating the encoder and decoder.

2. The method according to claim 1, wherein, The embedding operation is performed through an embedding network; the model update also includes updating the embedding network.

3. The method according to claim 2, wherein, The embedded network is a shallow convolutional neural network, in which the number of convolutional layers is less than a preset threshold.

4. The method according to claim 1, wherein, By removing a portion of the plurality of second embedding vectors, several second embedding vectors are obtained, including: A subset of the multiple second embedding vectors is randomly selected and removed to obtain several second embedding vectors.

5. The method according to claim 1, wherein, By removing a portion of the plurality of second embedding vectors, several second embedding vectors are obtained, including: A target vector group is determined from the plurality of second embedding vectors, and the plurality of target second patches corresponding to the plurality of target second embedding vectors included in the target vector group form a continuous region in the original depth map; The target vector group is removed from the plurality of second embedding vectors.

6. The method according to claim 1, wherein, By removing a portion of the plurality of second embedding vectors, several second embedding vectors are obtained, including: From the plurality of second embedding vectors, remove the second embedding vectors of a first proportion.

7. The method according to claim 6, wherein, The method includes multiple rounds of model updates, wherein the value of the first proportion in the i-th round is higher than that in the (i-1)-th round.

8. The method according to claim 1, wherein, The plurality of first embedding vectors and the plurality of second embedding vectors are input into a Transformer-based encoder, where self-attention processing is performed, specifically including: The plurality of first embedding vectors are combined with their corresponding plurality of first position codes to obtain a plurality of first combined vectors; the plurality of second embedding vectors are combined with their corresponding plurality of second position codes to obtain a plurality of second combined vectors; wherein, among the plurality of first blocks and the plurality of second blocks, two blocks with the same position have the same position code; The encoder is used to perform self-attention processing on the plurality of first combined vectors and the plurality of second combined vectors.

9. The method according to claim 1, wherein, The representation vector group includes: multiple first representation vectors corresponding to the multiple first embedding vectors, multiple second representation vectors corresponding to the multiple second embedding vectors, and a global representation vector that fuses global attention information. Using the decoder based on the representation vector set, a reconstructed depth image is obtained, including: The plurality of first representation vectors and the global representation vector are input into the decoder for decoding processing to obtain the reconstructed depth image.

10. The method according to claim 9, wherein, The decoder is implemented through a convolutional neural network, and the decoding process includes deconvolution processing.

11. The method according to claim 1, wherein, Model updates based on the original depth image and the reconstructed depth image include: Based on the comparison between the original depth image and the reconstructed depth image, the reconstruction loss is determined; Update the model in the direction that reduces the reconstruction loss.

12. An image processing method, comprising: Obtain the encoder trained according to the method of claim 1; The target RGB image and the corresponding target depth image are divided into multiple first patches and multiple second patches, respectively; Through the embedding operation, multiple first embedding vectors corresponding to multiple first tiles and multiple second embedding vectors corresponding to multiple second tiles are obtained; The plurality of first embedding vectors and the plurality of second embedding vectors are input into the encoder, where self-attention processing is performed to obtain a fused representation vector group. Based on the set of representation vectors, perform the target task related to the depth map.

13. An apparatus for training an image encoder, comprising: The tile division unit is configured to divide the RGB image and the corresponding original depth image into multiple first tiles and multiple second tiles, respectively; The embedding operation unit is configured to obtain multiple first embedding vectors corresponding to multiple first map tiles and multiple second embedding vectors corresponding to multiple second map tiles through embedding operations; The vector elimination unit is configured to eliminate a portion of the plurality of second embedded vectors to obtain a plurality of second embedded vectors; The encoding unit is configured to input the plurality of first embedding vectors and the plurality of second embedding vectors into a Transformer-based encoder, and perform self-attention processing in the encoder to obtain a fused representation vector group. The decoding unit is configured to use the decoder to obtain a reconstructed depth image based on the representation vector set; The model update unit is configured to update the model based on the original depth image and the reconstructed depth image, wherein the model update includes updating the encoder and the decoder.

14. A computer-readable storage medium having a computer program stored thereon, which, when executed in a computer, causes the computer to perform the method of any one of claims 1-12.

15. A computer system comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the method of any one of claims 1-12.

Images