Depth information processing method and device based on VLA large model

Abstract

The invention discloses a depth information processing method and device based on a VLA large model, and the method enables a model to master the surface features and space structure of a scene at the same time through the precise fusion of space depth information and visual appearance information, and forms more comprehensive feature representation. The space perception defect of an existing VLA large-model visual encoder is made up, the space understanding ability of the model to a real scene is improved, and then the learning and reasoning effects and the reasonability of decision response are optimized.

Description

A method and apparatus for deep information processing based on VLA large models Technical Field

[0001] This invention relates to the field of information processing technology, and specifically to a deep information processing method and apparatus based on a VLA large model. Background Technology

[0002] The existing VLA large model's visual encoder only relies on RGB three-channel image input, lacking depth information and unable to deeply perceive the spatial dimensions of the target scene (such as object distance, front and back occlusion, three-dimensional shape, etc.), resulting in errors that "do not conform to spatial characteristics" during model inference (such as misjudging the spatial positional relationship between objects, failing to recognize near and far occlusion leading to action decision bias, etc.). Summary of the Invention

[0003] The main objective of this invention is to provide a method and apparatus for deep information processing based on a VLA large model, so as to overcome the shortcomings of related technologies.

[0004] To achieve the above objectives, according to a first aspect of the present invention, a depth information processing method based on a VLA large model is provided, comprising: acquiring a depth map and an RGB image of a target scene; performing a patch extraction operation on the depth map and the RGB image to obtain a set of position-aligned depth map patches and a set of RGB image patches; and performing pixel flattening processing on each depth map patch in the depth map patch set and each RGB image patch in the RGB image patch set to obtain a vectorized representation of each depth map patch. Vectorized representation of each RGB image patch Where i is the index of the patch, corresponding to the patch pair aligned with the position on the depth map and the RGB image; the keypoint position is determined by a position-encoded multilayer perceptron. Encode the location to obtain a position encoding vector; based on the position encoding vector, the... The above Determine the embedding vector of the depth map patch and the embedding vector of the RGB image patch. , wherein The position of a specific pixel in the patch corresponding to index i; based on and The process involves determining the final feature vector that integrates spatial depth information and visual appearance information. , where the final feature vector The input is fed into the Projector module of the VLA large model to complete the feature space mapping to support the learning and inference process of the VLA large model.

[0005] Optionally, based on the location encoding vector, the The above Determine the embedding vector of the depth map patch and the embedding vector of the RGB image patch. Including: The intermediate features of the depth map patch are obtained by summing the intermediate features with the corresponding positional encoding vectors. Then, the correlation between key points in the intermediate features of the depth map patch is learned through a self-attention mechanism, and the embedding vector of the depth map patch is output. ;Will The intermediate features of the RGB image patch are obtained by summing the intermediate features with the corresponding positional encoding vectors. Then, the correlation between key points in the intermediate features of the RGB image patch is learned through a self-attention mechanism, and the embedding vector of the RGB image patch is output. Optionally, based on and The process involves determining the final feature vector that integrates spatial depth information and visual appearance information. Including: and corresponding Perform vector concatenation to obtain the initial features of the RGBD fusion patch; learn the correlation between key points in the initial features of the RGBD fusion patch through a self-attention mechanism, and output the final feature vector that fuses spatial depth information and visual appearance information. .

[0006] Optionally, performing patch extraction on the depth map and the RGB image includes: configuring a sliding window, placing the configured sliding window at a preset initial position on the depth map, and synchronously mapping it to the corresponding spatial position in the RGB image to ensure that the pixel coordinates of the covered area of ​​the sliding window on the depth map and the RGB image are completely aligned; controlling the sliding window to move along a preset direction with a set step size, and controlling the stopping of the movement according to the movement rules; determining the depth map area covered by the sliding window when it stops as a depth map patch, and recording the spatial coordinate range of the patch; controlling the sliding window to traverse the entire depth map according to a preset trajectory until all areas to be extracted have been patched, obtaining a depth map patch set; extracting the area with the corresponding coordinate range on the RGB image according to the spatial coordinate range of each depth map patch in the depth map patch set, as an RGB image patch aligned with the position of the depth map patch; traversing all depth map patches to complete the construction of the RGB image patch set.

[0007] Optionally, configuring the sliding window includes: setting the width of the sliding window. and height Set the sliding window's movement step size to... Set depth variance threshold .

[0008] Optionally, controlling the sliding window to move along a preset direction by a set step size, and controlling the stopping of the movement according to the movement rules, includes: controlling the sliding window to move along a preset direction by a set step size, and after each movement, calculating the variance s2 of the depth map pixels within the current window; if s2 ≥ If the current region's depth fluctuation is deemed too large, the sliding window will stop moving; if the sliding window has moved a cumulative distance... After each step, s2 still satisfies < Then stop moving the sliding window.

[0009] According to a second aspect of the present invention, a depth information processing method based on a VLA large model is provided, comprising: a patch extraction unit for acquiring a depth map and an RGB image of a target scene; performing a patch extraction operation on the depth map and the RGB image to obtain a set of position-aligned depth map patches and a set of RGB image patches; and a processing unit for performing pixel flattening processing on each depth map patch in the depth map patch set and each RGB image patch in the RGB image patch set to obtain a vectorized representation of each depth map patch. Vectorized representation of each RGB image patch Where i is the index of the patch, corresponding to the patch pair aligned with the position on the depth map and the RGB image; the keypoint position is determined by a position-encoded multilayer perceptron. Encode the location to obtain a position encoding vector; based on the position encoding vector, the... The above Determine the embedding vector of the depth map patch and the embedding vector of the RGB image patch. , wherein The position of a specific pixel in the patch corresponding to index i; the output unit, based on and The process involves determining the final feature vector that integrates spatial depth information and visual appearance information. , where the final feature vector The input is fed into the Projector module of the VLA large model to complete the feature space mapping to support the learning and inference process of the VLA large model.

[0010] According to a third aspect of the present invention, a computer-readable storage medium is provided, the computer-readable storage medium storing computer instructions for causing the computer to perform the method described in any one of the first aspects.

[0011] According to a fourth aspect of the present invention, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores a computer program executable by the at least one processor, the computer program being executed by the at least one processor to cause the at least one processor to perform the method described in any implementation of the first aspect.

[0012] According to a fourth aspect of the present invention, a computer program product is provided, wherein the computer program, when executed by a processor, implements the method described in any implementation of the first aspect.

[0013] This embodiment presents a depth information processing method and apparatus based on a VLA large-scale model. The method precisely fuses spatial depth information with visual appearance information, enabling the model to simultaneously grasp the surface features and spatial structure of a scene, forming a more comprehensive feature representation. This overcomes the spatial perception deficiencies of existing VLA large-scale model visual encoders, enhances the model's spatial understanding of real-world scenes, and thereby optimizes its learning, reasoning, and decision-making response rationality. Attached Figure Description

[0014] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0015] Figure 1 is a flowchart of the depth information processing method based on the VLA large model according to an embodiment of the present invention; Figures 2-4 are application diagrams of the depth information processing method based on the VLA large model according to an embodiment of the present invention; Figure 5 is a schematic diagram of the electronic device according to an embodiment of the present invention. Detailed Implementation

[0016] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0017] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate for the embodiments of the invention described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0018] It should be noted that, unless otherwise specified, the embodiments and features described in the present invention can be combined with each other. The present invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0019] According to an embodiment of the present invention, a depth information processing method based on a VLA large model is provided, as shown in Figure 1, including the following steps 101 to 103: Step 101: Obtain the depth map and RGB image of the target scene; perform a patch extraction operation on the depth map and the RGB image to obtain a set of position-aligned depth map patches and a set of RGB image patches.

[0020] In this step, the depth map is an intuitive representation of spatial depth information. Depth information can be easily obtained through various sensors. This method integrates depth information into the artificial neural network, enabling the VLA large model to better understand spatial information and make more reasonable decision-making responses based on this.

[0021] Obtain a depth map and an RGB image corresponding to the target scene. The spatial positions of the depth map and the RGB image are in one-to-one correspondence. The depth map is used to represent the spatial depth information of the target scene, and the RGB image is used to represent the visual appearance information of the target scene.

[0022] Step 102: Perform pixel flattening processing on each depth map patch in the depth map patch set and each RGB image patch in the RGB image patch set to obtain the vectorized representation of each depth map patch. Vectorized representation of each RGB image patch , where i is the index of the patch, corresponding to the patch pair that is aligned with the position on the depth map and the RGB image.

[0023] In this step, a depth-based patch extraction operation is performed on the depth map and the RGB image to obtain a depth map patch set and an RGB image patch set. The depth-based patch extraction operation must ensure that the patch positions on the depth map and the RGB image are strictly aligned. Each depth map patch in the depth map patch set is then pixel-flattened to obtain a vectorized representation of each depth map patch.

[0024] Each RGB image patch in the RGB image patch set is pixel-flattened to obtain a vectorized representation of each RGB image patch.

[0025] Step 103: Determine the location of key points using a position-encoded multilayer perceptron. Encode the location to obtain a position encoding vector; based on the position encoding vector, the... The above Determine the embedding vector of the depth map patch and the embedding vector of the RGB image patch. , wherein This represents the position of a specific pixel in the Patch corresponding to index i.

[0026] Index i clarifies the correspondence between position-aligned Patch pairs. The purpose of introducing index i is to assign a unique identifier to each pair of position-aligned depth map Patch + RGB map Patch.

[0027] As an optional implementation of this embodiment, based on the position encoding vector, the... The above Determine the embedding vector of the depth map patch and the embedding vector of the RGB image patch. Including: The intermediate features of the depth map patch are obtained by summing the intermediate features with the corresponding positional encoding vectors. Then, the correlation between key points in the intermediate features of the depth map patch is learned through a self-attention mechanism, and the embedding vector of the depth map patch is output. ;Will The intermediate features of the RGB image patch are obtained by summing the intermediate features with the corresponding positional encoding vectors. Then, the correlation between key points in the intermediate features of the RGB image patch is learned through a self-attention mechanism, and the embedding vector of the RGB image patch is output. .

[0028] In this optional implementation, through The location is encoded, and the location information of key points is mapped to a high-dimensional space. Then, through addition operations, the depth information and location information are fused. After fusion, the correlation between each key point is learned through the self-attention mechanism.

[0029] ,in , represents the position of key point i (a specific pixel).

[0030] Similarly, through The location is encoded, mapping the keypoint location information to a high-dimensional space. Then, through addition operations, the visual surface information and location information are fused. After fusion, the correlation between the keypoints is learned through the self-attention mechanism. .

[0031] Step 104: Based on and The process involves determining the final feature vector that integrates spatial depth information and visual appearance information. , where the final feature vector The input is fed into the Projector module of the VLA large model to complete the feature space mapping to support the learning and inference process of the VLA large model.

[0032] As an optional implementation method in this embodiment, based on and The process involves determining the final feature vector that integrates spatial depth information and visual appearance information. Including: and corresponding Perform vector concatenation to obtain the initial features of the RGBD fusion patch; learn the correlation between key points in the initial features of the RGBD fusion patch through a self-attention mechanism, and output the final feature vector that fuses spatial depth information and visual appearance information. .

[0033] In this step, a vector concatenation operation (⊕) is performed to obtain the initial features of the RGBD fusion patch. Then, a self-attention mechanism is used to learn the correlation between key points in the initial features of the RGBD fusion patch, and the final feature vector that fuses spatial depth information and visual appearance information is output. For example, the overall encoding process described above can be illustrated in Figure 2, which defines a novel visual encoder that integrates depth information for VLA. The depth map and RGB image are simultaneously input into the "Depth-based Patch Extraction" module. Patch data is extracted from both images, with the positions aligned. After extraction, pixel flattening is performed to form a vectorized representation of each patch, which is then fed into the MLP (Multilayer Perceptron) module, which outputs the embedding vector representation of the patch corresponding to the depth map. Embedding vector representation of the corresponding Patch in an RGB image .

[0034] The final feature vector is input into the Projector module of the VLA large model to complete the feature space mapping, supporting the learning and inference process of the VLA large model. As an optional implementation method in this embodiment, the patch extraction operation on the depth map and the RGB image includes: configuring a sliding window, placing the configured sliding window at a preset initial position on the depth map, and synchronously mapping it to the corresponding spatial position on the RGB image to ensure that the pixel coordinates of the coverage area of ​​the sliding window on the depth map and the RGB image are completely aligned; controlling the sliding window to move along a preset direction with a set step size, and controlling the stopping of the movement according to the movement rules; determining the depth map area covered by the sliding window when it stops as a depth map patch, and recording the spatial coordinate range of the patch; controlling the sliding window to traverse the entire depth map according to a preset trajectory until all areas to be extracted are patched, obtaining a depth map patch set; extracting the area with the corresponding coordinate range on the RGB image according to the spatial coordinate range of each depth map patch in the depth map patch set, as an RGB image patch aligned with the position of the depth map patch; traversing all depth map patches to complete the construction of the RGB image patch set.

[0035] In this optional implementation, as shown in Figure 3, the "Depth-based Patch Extraction" module first defines a sliding window for extracting patches, with a width and height of [missing information]. and ( and The threshold range is the width of the original image. Between (between), the window positions of the depth map and the RGB image are strictly aligned, as shown in Figure 4. The sliding window is initially in First move Each step size ( Threshold range is ), calculate the variance of the entire patch depth value ,if ( Threshold range is If the distance between the specified values ​​(in meters) indicates a large depth fluctuation, the sliding window is stopped; otherwise, the sliding window is moved, and depth variance is checked at the new position. This continues until the sliding window has moved a certain distance. Even with a step size of [number], no significant fluctuations in depth change were detected, confirming that the current window has [changed / remained] from [a certain point]. move Step size to .

[0036] In this optional implementation, the depth fluctuates significantly, corresponding to key areas of spatial structural change in real-world scenes (e.g., object edges, boundaries between different objects, and abrupt changes in distance—such as the boundary between a tabletop and the air, or the connection between walls and furniture). The depth information in these areas is crucial for the VLA model to understand spatial relationships: stopping the window's movement and extracting this area as a patch accurately preserves this key spatial feature of depth abrupt changes, preventing the fragmentation and loss of crucial depth information due to continued window movement. This ensures the model can learn the 3D boundaries of objects, occlusion, and other spatial logic through this patch, compensating for the lack of spatial awareness in existing technologies.

[0037] Without significant depth fluctuations, this corresponds to flat areas in real-world spatial structures (e.g., flat walls, open ground, smooth surfaces of the same object). The depth information in these areas is relatively uniform, eliminating the need for excessive patch subdivision (excessive subdivision leads to feature redundancy and reduces model processing efficiency). Therefore, after moving the window by Wu steps (Wu being the window width, a preset reasonable coverage threshold), the movement stops and the patch is extracted. This ensures complete coverage of the depth information in the flat area while avoiding unnecessary repetitive movements, achieving a balance between "extraction completeness" and "processing efficiency."

[0038] It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases the steps shown or described may be executed in a different order than that shown here.

[0039] According to an embodiment of the present invention, a method is also provided.

[0040] According to embodiments of the present invention, the present invention also provides an electronic device, the electronic device comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to implement the methods described in any of the above embodiments.

[0041] According to embodiments of the present invention, the present invention also provides a readable storage medium storing computer instructions that enable a computer to perform the methods described in any of the above embodiments when executed.

[0042] According to embodiments of the present invention, the present invention also provides a computer program product that, when executed by a processor, can implement the methods described in any of the above embodiments.

[0043] Figure 5 shows a schematic block diagram of an example electronic device 300 that can be used to implement embodiments of the present invention. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices.

[0044] As shown in Figure 5, the electronic device 300 includes a computing unit 301, which can perform various appropriate actions and processes based on a computer program stored in a read-only memory (ROM) 302 or a computer program loaded from a storage unit 308 into a random access memory (RAM) 303. The RAM 303 can also store various programs and data required for the operation of the electronic device 300. The computing unit 301, ROM 302, and RAM 303 are interconnected via a bus 304. An input / output (I / O) interface 305 is also connected to the bus 304.

[0045] Multiple components in electronic device 300 are connected to I / O interface 305, including: input unit 306, such as keyboard, mouse, etc.; output unit 307, such as various types of displays, speakers, etc.; storage unit 308, such as disk, optical disk, etc.; and communication unit 309, such as network card, modem, wireless transceiver, etc. Communication unit 309 allows electronic device 300 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0046] The computing unit 301 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 301 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 301 performs the various methods and processes described above, such as the object matching method. For example, in some embodiments, the object matching method may be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 308. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 300 via ROM 302 and / or communication unit 309. When the computer program is loaded into RAM 303 and executed by the computing unit 301, one or more steps of the methods described above may be performed.

[0047] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0048] The program code used to implement the methods of the present invention can be written in any combination of one or more programming languages. This program code can be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing device, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code can be executed entirely on the machine, partially on the machine, as a standalone software package partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0049] In the context of this invention, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. Machine-readable media can include, but are not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

Claims

1. A deep information processing method based on a VLA large model, characterized in that, The process includes: acquiring a depth map and an RGB image of the target scene; performing patch extraction operations on the depth map and the RGB image to obtain a set of position-aligned depth map patches and a set of RGB image patches; and performing pixel flattening processing on each depth map patch in the depth map patch set and each RGB image patch in the RGB image patch set to obtain a vectorized representation of each depth map patch. Vectorized representation of each RGB image patch Where i is the index of the patch, corresponding to the patch pair aligned with the position on the depth map and the RGB image; the keypoint position is determined by a position-encoded multilayer perceptron. Encode the location to obtain a position encoding vector; based on the position encoding vector, the... The above Determine the embedding vector of the depth map patch and the embedding vector of the RGB image patch. , wherein The position of a specific pixel in the patch corresponding to index i; based on and The process involves determining the final feature vector that integrates spatial depth information and visual appearance information. , where the final feature vector The input is fed into the Projector module of the VLA large model to complete the feature space mapping to support the learning and inference process of the VLA large model.

2. The deep information processing method based on the VLA large model according to claim 1, characterized in that, Based on the location encoding vector, the The above Determine the embedding vector of the depth map patch and the embedding vector of the RGB image patch. Including: The intermediate features of the depth map patch are obtained by summing the intermediate features with the corresponding positional encoding vectors. Then, the correlation between key points in the intermediate features of the depth map patch is learned through a self-attention mechanism, and the embedding vector of the depth map patch is output. ;Will The intermediate features of the RGB image patch are obtained by summing the intermediate features with the corresponding positional encoding vectors. Then, the correlation between key points in the intermediate features of the RGB image patch is learned through a self-attention mechanism, and the embedding vector of the RGB image patch is output. 。 3. The deep information processing method based on the VLA large model according to claim 1, characterized in that, based on and The process involves determining the final feature vector that integrates spatial depth information and visual appearance information. Including: and corresponding Perform vector concatenation operation to obtain the initial features of the RGBD fused patch; The correlation between key points in the initial features of the RGBD fusion patch is learned through a self-attention mechanism, and the final feature vector that fuses spatial depth information and visual appearance information is output. 。 4. The deep information processing method based on the VLA large model according to claim 1, characterized in that, The patch extraction operation on the depth map and the RGB image includes: configuring a sliding window, placing the configured sliding window at a preset initial position on the depth map, and synchronously mapping it to the corresponding spatial position on the RGB image to ensure that the pixel coordinates of the covered area of ​​the sliding window on the depth map and the RGB image are completely aligned; controlling the sliding window to move along a preset direction with a set step size, and controlling the stopping of the movement according to the movement rules; determining the depth map area covered by the sliding window when it stops as a depth map patch, and recording the spatial coordinate range of the patch; controlling the sliding window to traverse the entire depth map according to a preset trajectory until all areas to be extracted have been patched, obtaining a depth map patch set; extracting the area with the corresponding coordinate range on the RGB image according to the spatial coordinate range of each depth map patch in the depth map patch set, as an RGB image patch aligned with the position of the depth map patch; traversing all depth map patches to complete the construction of the RGB image patch set.

5. The deep information processing method based on the VLA large model according to claim 4, characterized in that, Configuring a sliding window includes setting the width of the sliding window. and height Set the sliding window's movement step size to... Set depth variance threshold 。 6. The deep information processing method based on the VLA large model according to claim 5, characterized in that, Controlling the sliding window to move along a preset direction by a set step size, and controlling the stopping of the movement according to the movement rules, includes: controlling the sliding window to move along a preset direction by a set step size, and after each movement, calculating the variance s2 of the depth map pixels within the current window; if s2 ≥ If the current region's depth fluctuation is deemed too large, the sliding window will stop moving; if the sliding window has moved a cumulative distance... After each step, s2 still satisfies < Then stop moving the sliding window.

7. A deep information processing method based on a VLA large model, characterized in that, include: The Patch extraction unit is used to obtain the depth map and RGB image of the target scene; Perform a patch extraction operation on the depth map and the RGB image to obtain a set of position-aligned depth map patches and a set of RGB image patches; The processing unit performs pixel flattening processing on each depth map patch in the depth map patch set and each RGB image patch in the RGB image patch set to obtain a vectorized representation of each depth map patch. Vectorized representation of each RGB image patch Where i is the index of the patch, corresponding to the patch pair aligned with the position on the depth map and the RGB image; the keypoint position is determined by a position-encoded multilayer perceptron. Encode the location to obtain a position encoding vector; based on the position encoding vector, the... The above Determine the embedding vector of the depth map patch and the embedding vector of the RGB image patch. , wherein The position of a specific pixel in the patch corresponding to index i; the output unit, based on and The process involves determining the final feature vector that integrates spatial depth information and visual appearance information. , where the final feature vector The input is fed into the Projector module of the VLA large model to complete the feature space mapping to support the learning and inference process of the VLA large model.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions for causing the computer to perform the method according to any one of claims 1-7.

9. An electronic device, characterized in that, include: At least one processor; And a memory communicatively connected to the at least one processor; wherein the memory stores a computer program executable by the at least one processor, the computer program being executed by the at least one processor to cause the at least one processor to perform the method according to any one of claims 1-7.

10. A computer program product, characterized in that, When executed by a processor, the computer program implements the method described in any one of claims 1-6.

Images