Intelligent optical flow image reconstruction method and device
By generating and using optical flow maps to compensate and reconstruct images in depth image videos, the problem of camera random jitter reducing image accuracy is solved, and the accuracy of embodied intelligent control is improved.
Patent Information
- Application Number
- CN202411702217.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-11-26
- Publication Date
- 2025-03-28
AI Technical Summary
Random jitter in the camera system will reduce the image display accuracy and affect the accuracy of embodied intelligent control.
By obtaining the depth image video stream of the binocular camera, splitting it into image frames, generating an image optical flow map, and inputting it into the optical flow image reconstruction model, obtaining a compensation pose, and then reconstructing the image frame.
It reduces the accuracy impact of camera random jitter, improves image display accuracy, and ensures the smooth implementation of embodied intelligent control.
Smart Images
Figure CN119722928A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of medical image recognition technology, and more specifically, to an optical flow image reconstruction method and device for embodied intelligence. Background Art
[0002] Random jitter of a camera system refers to image jitter caused by external factors (such as vibration, ambient temperature, mechanical error, etc.) during the camera shooting process. This jitter is random and cannot be predicted in advance, but random jitter will affect the clarity of the image, especially in high-resolution imaging systems.
[0003] The embodied intelligent control of surgical robots requires corresponding processing based on the depth image video during the operation. However, the random jitter of the camera itself will reduce the display accuracy of the depth image video, thereby affecting the subsequent embodied intelligent control. Summary of the invention
[0004] The problem solved by this application is that the random jitter of the current camera system will reduce the display accuracy of the image.
[0005] To solve the above problems, the first aspect of the present application provides an optical flow image reconstruction method for embodied intelligence, comprising:
[0006] Get the depth image video stream of the stereo camera and split it into consecutively set image frames;
[0007] generating an image optical flow map according to the image frame;
[0008] Inputting the optical flow map into an optical flow image reconstruction model to obtain a compensation posture corresponding to the image optical flow map;
[0009] The image frame is reconstructed based on the compensated pose.
[0010] A second aspect of the present application provides an optical flow image reconstruction device for embodied intelligence, comprising:
[0011] A video splitting module is used to obtain the depth image video stream of the binocular camera and split it into consecutively set image frames;
[0012] An optical flow generation module, which is used to generate an image optical flow map according to the image frame;
[0013] An optical flow processing module, which is used to input the optical flow map into an optical flow image reconstruction model to obtain a compensation posture corresponding to the image optical flow map;
[0014] An image reconstruction module is used to reconstruct the image frame based on the compensated posture.
[0015] A third aspect of the present application provides an electronic device, comprising: a memory and a processor;
[0016] The memory is used to store programs;
[0017] The processor, coupled to the memory, is configured to execute the program to:
[0018] Get the depth image video stream of the stereo camera and split it into consecutively set image frames;
[0019] generating an image optical flow map according to the image frame;
[0020] Inputting the optical flow map into an optical flow image reconstruction model to obtain a compensation posture corresponding to the image optical flow map;
[0021] The image frame is reconstructed based on the compensated pose.
[0022] A fourth aspect of the present application provides a computer-readable storage medium having a computer program stored thereon, wherein the program is executed by a processor to implement the above-mentioned optical flow image reconstruction method for embodied intelligence.
[0023] In the present application, an optical flow map of the image is generated and used to compensate and reconstruct the image in the depth image video, thereby reducing the accuracy impact caused by random camera jitter, thereby ensuring the smooth implementation of subsequent embodied intelligent control. BRIEF DESCRIPTION OF THE DRAWINGS
[0024] Figure 1 is a flowchart of an optical flow image reconstruction method for embodied intelligence according to an embodiment of the present application;
[0025] Figure 2 A model architecture diagram of an optical flow image reconstruction method for embodied intelligence according to an embodiment of the present application;
[0026] Figure 3 It is an architecture diagram of a spatial attention module of an optical flow image reconstruction method for embodied intelligence according to an embodiment of the present application;
[0027] Figure 4 is a structural block diagram of an optical flow image reconstruction device for embodied intelligence according to an embodiment of the present application;
[0028] Figure 5 is a structural block diagram of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION
[0029] In order to make the above-mentioned purposes, features and advantages of the present application more obvious and easy to understand, the specific embodiments of the present application are described in detail below in conjunction with the accompanying drawings. Although the exemplary embodiments of the present application are shown in the accompanying drawings, it should be understood that the present application can be implemented in various forms and should not be limited by the embodiments described herein. On the contrary, these embodiments are provided in order to enable a more thorough understanding of the present application and to fully convey the scope of the present application to those skilled in the art.
[0030] It should be noted that, unless otherwise specified, the technical terms or scientific terms used in this application should have the common meanings understood by technicians in the field to which this application belongs.
[0031] The present application embodiment provides an optical flow image reconstruction method for embodied intelligence. The specific scheme of the method is as follows: Figure 1-Figure 3 As shown, the method can be performed by an optical flow image reconstruction device for embodied intelligence, and the optical flow image reconstruction device for embodied intelligence can be integrated in electronic devices such as computers, servers, computers, server clusters, and data centers. Figure 1 , Figure 2 As shown, it is a flow chart of an optical flow image reconstruction method for embodied intelligence according to an embodiment of the present application; wherein the optical flow image reconstruction method for embodied intelligence includes:
[0032] S101, obtaining a depth image video stream of a binocular camera and splitting it into consecutively set image frames;
[0033] S102, generating an image optical flow map according to the image frame;
[0034] Optical flow is a motion vector field that describes the pixels in an image sequence, that is, how each pixel moves from one position to another between two consecutive frames. Optical flow information can be used to determine the direction and speed of movement of objects in an image. An optical flow map is a graph composed of optical flows, used to represent an image of pixel motion information in a video or image sequence.
[0035] S103, inputting the optical flow map into an optical flow image reconstruction model to obtain a compensation posture corresponding to the image optical flow map;
[0036] S104: Reconstruct the image frame based on the compensated posture.
[0037] In the present application, an optical flow map of the image is generated and used to compensate and reconstruct the image in the depth image video, thereby reducing the accuracy impact caused by random camera jitter, thereby ensuring the smooth implementation of subsequent embodied intelligent control.
[0038] In the present application, the characteristics of the optical flow graph are utilized to convert image frames into changes and errors between adjacent image frames, thereby achieving subsequent accurate recognition and compensation.
[0039] In the present application, after acquiring the image frame, two adjacent image frames are selected, and the Lucas-Kanade algorithm is used to calculate the sparse optical flow by assuming that the motion within the neighborhood is consistent to obtain the optical flow vector; the optical flow vector is converted into a color map, which is the image optical flow map of the image frame.
[0040] In the present application, the optical flow vector is converted into a color map, the direction of the optical flow vector corresponds to the hue of the color map, and the speed of the optical flow vector corresponds to the brightness of the color map.
[0041] It should be noted that the image optical flow map is obtained from adjacent image frames, so the image optical flow map can be uniformly divided into the optical flow map corresponding to the previous image frame.
[0042] In this application, the image frames are acquired when the binocular camera is in a shaking state, that is, acquired in an actual posture. However, since the shaking is not detected, the subsequent processing of the image frames is based on the standard posture. This error is part of the harm caused by random jitter.
[0043] In the present application, the image frame is reconstructed by compensating the posture, that is, the image frame is processed based on the standard posture and then reprocessed by compensating the posture, so as to reconstruct it into an image frame processed based on the actual posture.
[0044] In one embodiment, in combination Figure 2 As shown, the optical flow map is input into the optical flow image reconstruction model to obtain the compensation posture corresponding to the image optical flow map, including:
[0045] Input the optical flow map into the Transformer structure to obtain a supervised feature map;
[0046] Inputting the supervised feature map into an optical flow feature extraction structure to obtain an optical flow feature map;
[0047] The optical flow feature map and the standard pose are input into a fully connected structure to obtain a compensated pose.
[0048] In this application, the optical flow map enters the Transformer structure and uses its multi-head self-attention mechanism to generate a supervised feature map. This supervised feature map contains the motion information and spatial features in the optical flow map, which is suitable for further feature extraction.
[0049] In this application, the fully connected structure combines the optical flow feature map with the standard pose and outputs a compensated pose. The compensated pose is a correction of the standard pose, taking into account the position deviation caused by jitter.
[0050] In this application, the self-attention mechanism can help the model capture long-range dependencies in images.
[0051] In this application, the optical flow map is used as input, and Transformer processes the spatial and motion information of the image through its multi-layer self-attention mechanism.
[0052] In this application, the supervised feature map obtained from the Transformer is input into the optical flow feature extraction structure to extract the motion features of the optical flow.
[0053] In this application, the optical flow feature map and the standard pose are input into the fully connected structure to obtain the compensated pose, where the standard pose is the current pose of the binocular camera, that is, the pose when the corresponding depth image video stream / image frame is captured. The pose can be determined by a built-in gyroscope or a third-party calculation.
[0054] It should be noted that whether it is the built-in gyroscope or the third-party inference method, what is obtained is the theoretical current posture, not the posture after random jitter, so it is considered to be the standard posture.
[0055] In the present application, the compensated posture is a posture obtained after compensating the standard posture for random jitter, and can also be considered as a real posture. Based on this posture, the actual image frame after random jitter can be determined.
[0056] In this application, the standard posture is usually expressed in vector form, which is composed of position and posture.
[0057] In this application, the optical flow feature map and the standard pose are spliced together as the input of the fully connected structure. The fully connected layer outputs the compensated pose through a series of nonlinear transformations; the compensated pose is added to the standard pose to obtain the pose actually required to be executed.
[0058] Among them, the optical flow feature extraction structure can extract the corresponding features and output them as a feature vector of a fixed length (optical flow feature map).
[0059] In one implementation, before inputting the optical flow map into the Transformer structure to obtain the supervised feature map, the method further includes:
[0060] The optical flow map is subjected to random data enhancement processing to obtain a plurality of data-enhanced optical flow maps suitable for being used as the input optical flow map.
[0061] In this application, random data enhancement processing is performed on the optical flow map to improve the generalization ability of the model. At the same time, the Transformer is used to meet the transformation invariance and effectively extract the motion essence representation of the optical flow map.
[0062] In one implementation, the random data enhancement process includes: random cropping process, random color distortion process and random Gaussian blur process.
[0063] Random cropping process: randomly select the position and size of a cropping window from the optical flow map to ensure that the cropped area matches the target area; rescale the cropped image to the size of the original optical flow map to maintain data consistency.
[0064] Effect: Increasing the diversity of optical flow map data and obtaining feature changes in different regions helps improve the spatial generalization ability of the model.
[0065] Random color distortion: Randomly adjust the brightness, contrast, saturation or hue of the optical flow map. The following adjustment methods can be used:
[0066] Brightness: Randomly increase or decrease the brightness of the optical flow map. Contrast: Randomly increase or decrease the contrast to make bright areas brighter or dark areas darker. Saturation: Randomly adjust the saturation to change the color concentration. Hue: Randomly adjust the overall hue of the image. Each parameter can be randomly changed within a certain range to avoid excessive distortion.
[0067] Effect: Simulates different lighting conditions to make the model more robust to lighting changes.
[0068] Random Gaussian Blur: Randomly select a blur radius and perform Gaussian blur on the optical flow map. Common blur radius is between 1 and 5. Use different radius values to simulate different focusing effects.
[0069] Effect: Simulate different focusing effects in the scene and enhance the model's expressiveness on blurred optical flow images.
[0070] In one embodiment, in combination Figure 2 As shown, the supervised feature map is input into the optical flow feature extraction structure to obtain the optical flow feature map, including:
[0071] Inputting the supervised feature map into a visual attention module to obtain a visual feature map;
[0072] Inputting the visual feature map into a spatial attention module to obtain a spatial attention map;
[0073] The spatial attention map is input into a visual attention module to obtain the optical flow feature map.
[0074] In this application, the supervised feature map (features obtained from the Transformer model) is input into a visual attention module. This module usually uses a self-attention mechanism to focus on important areas in the image, enhance important features, and suppress irrelevant areas.
[0075] In this application, the goal of the visual attention module is to dynamically adjust the representation of each region in the feature map by calculating the similarity (attention weight) between various parts in the image, so that the model can focus on important spatial regions.
[0076] In this application, the obtained visual feature map will enter the spatial attention module, which focuses on the features at different positions in the image space. This module adjusts the feature strength at different positions in the image by calculating the attention weights in the spatial dimension.
[0077] The spatial attention module usually uses convolution operations (such as 3x3 convolution) to generate a spatial attention map, which can guide the model to spatially enhance the features of certain regions while suppressing other unimportant regions.
[0078] In this application, the generated spatial attention map will be input into the visual attention module again to further enhance the important features in the image and finally output the optical flow feature map.
[0079] At this stage, the spatial attention map acts as a “guide” to instruct the visual attention module to pay more attention to specific spatial regions, thereby extracting effective optical flow features.
[0080] In this application, the visual attention module: uses convolutional layers to generate query, key and value respectively, then calculates the self-attention matrix, and obtains the enhanced feature map through weighted aggregation.
[0081] In this application, the spatial attention module uses convolution to calculate the spatial attention map, which is used as a weight to adjust the feature values at different positions of the image.
[0082] In this application, the optical flow feature extraction structure is as follows: the supervised feature map is input into the visual attention module to obtain the visual feature map, then input into the spatial attention module to obtain the spatial attention map, and finally the enhanced optical flow feature map is returned.
[0083] In one implementation, the supervised feature map is input into a visual attention module to obtain parameter sharing between the optical flow maps enhanced by the same optical flow map data in the visual feature map.
[0084] Multiple views of the same data enhancement: After performing different enhancement methods on the optical flow map (such as random cropping, color distortion, Gaussian blur, etc.), multiple enhanced optical flow maps are generated to ensure that they still retain the key features of the original optical flow to some extent.
[0085] Shared parameter mechanism: In the visual attention module, the same weight parameters are set to process these enhanced optical flow maps. Because these enhanced data come from the same optical flow map, parameter sharing can ensure that the feature maps under different data enhancement perspectives maintain consistent feature extraction methods, enhancing the model's ability to recognize the same features under different perspectives.
[0086] Feature fusion: After the visual attention module processes the feature map, it obtains different enhanced visual feature maps. Due to the use of parameter sharing, the feature maps under each enhanced perspective share weights within the model to maintain feature consistency.
[0087] In this application, the feature map is screened and key area information is extracted through the visual attention module, and the parameter sharing mechanism is used to more effectively identify the key information in the optical flow map.
[0088] In this application, since the model has consistency and robustness in features from different perspectives of data enhancement, the generalization ability of unknown data is improved.
[0089] In this application, when processing optical flow maps in the visual attention module, multiple optical flow maps generated by data enhancement share the same network parameters. This sharing can enable the model to more effectively utilize the information of data enhancement while maintaining the efficiency and generalization ability of the network.
[0090] In this application, multiple enhanced optical flow maps are generated by performing data augmentation (such as rotation, flipping, scaling, etc.) on the original optical flow map. In the visual attention module, these enhanced optical flow maps will share the same set of parameters, thereby effectively utilizing the enhanced data without increasing the complexity of the model.
[0091] In one embodiment, all enhanced optical flow maps can be batch-inputted into the visual attention module, and the features of the optical flow maps can be jointly learned by sharing network parameters (i.e., weights of convolutional layers or self-attention layers). In this way, the network can learn the common features exhibited by the enhanced images, rather than learning different features for each enhanced image.
[0092] In this application, all enhanced optical flow maps are batch-input into the visual attention module to ensure that they share the same parameters (i.e., the weights of the convolutional layer). In this way, the model does not re-learn new features when processing different transformations, but extracts the common features of the optical flow map through the shared parameters.
[0093] In one embodiment, in combination Figure 3 As shown, the visual feature map is input into the spatial attention module to obtain a spatial attention map, including:
[0094] Inputting the visual feature map into the constructed hypernetwork for encoding processing to obtain an attention weight tensor;
[0095] Based on a dynamic spatial attention mechanism, retaining information in the optical flow map;
[0096] Generate optical flow markers based on the retained information in the optical flow map;
[0097] Based on the optical flow markers, a spatial attention map is generated.
[0098] In this application, a hypernetwork is a network for dynamically generating parameters, which generates attention weight tensors based on the input visual feature map, and these weights are used to adjust the degree of attention to features in the image.
[0099] The role of the hypernetwork is to dynamically generate adaptive attention weights based on the input visual feature map. This function can be implemented using a convolutional neural network (CNN) or other types of networks.
[0100] In this application, the dynamic spatial attention mechanism determines which areas in space should be retained (enhanced) by calculating the attention weights to help extract more important features and suppress unimportant parts. This step ensures that the key information in the optical flow map is retained.
[0101] By calculating the attention weights in space, useful information in the image is retained or strengthened. Convolutional layers and activation functions can be used to dynamically adjust the attention weights of each spatial position.
[0102] In this application, with the help of the spatial attention mechanism, optical flow marker points are generated, which can be considered as important motion or feature points in the image, and these points are used for further analysis or tracking.
[0103] Based on the dynamic spatial attention mechanism, the model identifies important image regions (such as moving objects, edges, or feature points). These markers are used to generate spatial attention maps.
[0104] In this application, finally, a spatial attention map is generated according to the position and importance of the optical flow markers, which will identify which positions in the feature map should receive more attention.
[0105] An embodiment of the present application provides an optical flow image reconstruction device for embodied intelligence, which is used to execute an optical flow image reconstruction method for embodied intelligence described in the above content of the present application. The optical flow image reconstruction device for embodied intelligence is described in detail below.
[0106] like Figure 4 As shown, the optical flow image reconstruction device for embodied intelligence includes:
[0107] The video splitting module 101 is used to obtain the depth image video stream of the binocular camera and split it into consecutively set image frames;
[0108] An optical flow generation module 102, which is used to generate an image optical flow map according to the image frame;
[0109] An optical flow processing module 103, which is used to input the optical flow map into an optical flow image reconstruction model to obtain a compensation posture corresponding to the image optical flow map;
[0110] The image reconstruction module 104 is configured to reconstruct the image frame based on the compensated posture.
[0111] In one implementation, the optical flow processing module 103 is further configured to:
[0112] The optical flow map is input into the Transformer structure to obtain a supervised feature map; the supervised feature map is input into the optical flow feature extraction structure to obtain an optical flow feature map; the optical flow feature map and the standard pose are input into the fully connected structure to obtain a compensated pose.
[0113] In one implementation, the optical flow processing module 103 is further configured to:
[0114] The optical flow map is subjected to random data enhancement processing to obtain a plurality of data-enhanced optical flow maps suitable for being used as the input optical flow map.
[0115] In one implementation, the random data enhancement process includes: random cropping process, random color distortion process and random Gaussian blur process.
[0116] In one implementation, the optical flow processing module 103 is further configured to:
[0117] The supervised feature map is input into a visual attention module to obtain a visual feature map; the visual feature map is input into a spatial attention module to obtain a spatial attention map; the spatial attention map is input into a visual attention module to obtain the optical flow feature map.
[0118] In one implementation, the supervised feature map is input into a visual attention module to obtain parameter sharing between the optical flow maps enhanced by the same optical flow map data in the visual feature map.
[0119] In one implementation, the optical flow processing module 103 is further configured to:
[0120] The supervised feature map is input into the constructed hypernetwork for encoding processing to obtain an attention weight tensor; based on a dynamic spatial attention mechanism, the information in the optical flow map is retained; based on the retained information in the optical flow map, optical flow markers are generated; based on the optical flow markers, a visual feature map is generated.
[0121] The above describes the internal functions and structure of an optical flow image reconstruction device for embodied intelligence, such as Figure 5 As shown, in practice, the optical flow image reconstruction device for embodied intelligence can be implemented as an electronic device, including: a memory 301 and a processor 303.
[0122] The memory 301 may be configured to store programs.
[0123] In addition, the memory 301 may also be configured to store various other data to support operations on the electronic device. Examples of such data include instructions for any application or method operating on the electronic device, contact data, phone book data, messages, pictures, videos, etc.
[0124] The memory 301 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.
[0125] The processor 303 is coupled to the memory 301 and is configured to execute a program in the memory 301 to:
[0126] Get the depth image video stream of the stereo camera and split it into consecutively set image frames;
[0127] generating an image optical flow map according to the image frame;
[0128] Inputting the optical flow map into an optical flow image reconstruction model to obtain a compensation posture corresponding to the image optical flow map;
[0129] The image frame is reconstructed based on the compensated pose.
[0130] In one implementation, the processor 303 is further configured to:
[0131] The optical flow map is input into the Transformer structure to obtain a supervised feature map; the supervised feature map is input into the optical flow feature extraction structure to obtain an optical flow feature map; the optical flow feature map and the standard pose are input into the fully connected structure to obtain a compensated pose.
[0132] In one implementation, the processor 303 is further configured to:
[0133] The optical flow map is subjected to random data enhancement processing to obtain a plurality of data-enhanced optical flow maps suitable for being used as the input optical flow map.
[0134] In one implementation, the random data enhancement process includes: random cropping process, random color distortion process and random Gaussian blur process.
[0135] In one implementation, the processor 303 is further configured to:
[0136] The supervised feature map is input into a visual attention module to obtain a visual feature map; the visual feature map is input into a spatial attention module to obtain a spatial attention map; the spatial attention map is input into a visual attention module to obtain the optical flow feature map.
[0137] In one implementation, the supervised feature map is input into a visual attention module to obtain parameter sharing between the optical flow maps enhanced by the same optical flow map data in the visual feature map.
[0138] In one implementation, the processor 303 is further configured to:
[0139] The supervised feature map is input into the constructed hypernetwork for encoding processing to obtain an attention weight tensor; based on a dynamic spatial attention mechanism, the information in the optical flow map is retained; based on the retained information in the optical flow map, optical flow markers are generated; based on the optical flow markers, a visual feature map is generated.
[0140] In this application, Figure 5 Only some components are shown schematically, which does not mean that the electronic device only includes Figure 5 Components shown.
[0141] The electronic device provided in this embodiment is based on the same inventive concept as the optical flow image reconstruction method for embodied intelligence provided in the embodiment of the present application, and has the same beneficial effects as the method adopted, run or implemented by the application stored in it.
[0142] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-readable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.
[0143] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.
[0144] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.
[0145] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.
[0146] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.
[0147] The memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash memory (Flash RAM). The memory is an example of a computer-readable medium.
[0148] The present application also provides a computer-readable storage medium corresponding to an optical flow image reconstruction method for embodied intelligence provided in the aforementioned embodiment, on which a computer program (i.e., a program product) is stored. When the computer program is run by a processor, it will execute an optical flow image reconstruction method for embodied intelligence provided in any of the aforementioned embodiments.
[0149] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.
[0150] The computer-readable storage medium provided in the above-mentioned embodiments of the present application and the optical flow image reconstruction method for embodied intelligence provided in the embodiments of the present application are based on the same inventive concept and have the same beneficial effects as the methods adopted, run or implemented by the application programs stored therein.
[0151] It should be noted that in the description provided herein, a large number of specific details are described. However, it is understood that the embodiments of the present application can be practiced without these specific details. In some instances, well-known structures and technologies are not shown in detail so as not to obscure the understanding of this description.
[0152] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.
[0153] The above is only an embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the scope of the claims of the present application.
Claims
1. An optical flow image reconstruction method for embodied intelligence, characterized in that: include: Get the depth image video stream of the stereo camera and split it into consecutively set image frames; generating an image optical flow map according to the image frame; Inputting the optical flow map into an optical flow image reconstruction model to obtain a compensation posture corresponding to the image optical flow map; The image frame is reconstructed based on the compensated pose.
2. The optical flow image reconstruction method for embodied intelligence according to claim 1, characterized in that: The step of inputting the optical flow map into an optical flow image reconstruction model to obtain a compensation posture corresponding to the image optical flow map includes: Input the optical flow map into the Transformer structure to obtain a supervised feature map; Inputting the supervised feature map into an optical flow feature extraction structure to obtain an optical flow feature map; The optical flow feature map and the standard pose are input into a fully connected structure to obtain a compensated pose.
3. The optical flow image reconstruction method for embodied intelligence according to claim 2, characterized in that: Before inputting the optical flow map into the Transformer structure to obtain the supervised feature map, the method further includes: The optical flow map is subjected to random data enhancement processing to obtain a plurality of data-enhanced optical flow maps suitable for being used as the input optical flow map.
4. The optical flow image reconstruction method for embodied intelligence according to claim 3, characterized in that: The random data enhancement process includes: random cropping process, random color distortion process and random Gaussian blur process.
5. The optical flow image reconstruction method for embodied intelligence according to any one of claims 2 to 4, characterized in that: The step of inputting the supervised feature map into an optical flow feature extraction structure to obtain an optical flow feature map comprises: Inputting the supervised feature map into a visual attention module to obtain a visual feature map; Inputting the visual feature map into a spatial attention module to obtain a spatial attention map; The spatial attention map is input into a visual attention module to obtain the optical flow feature map.
6. The optical flow image reconstruction method for embodied intelligence according to claim 5, characterized in that: The supervised feature map is input into a visual attention module to obtain parameter sharing between the optical flow maps enhanced by the same optical flow map data in the visual feature map.
7. The optical flow image reconstruction method for embodied intelligence according to claim 5, characterized in that: The step of inputting the supervised feature map into a visual attention module to obtain a visual feature map comprises: Inputting the supervised feature map into the constructed hypernetwork for encoding processing to obtain an attention weight tensor; Based on a dynamic spatial attention mechanism, retaining information in the optical flow map; Generate optical flow markers based on the retained information in the optical flow map; Based on the optical flow markers, a visual feature map is generated.
8. An optical flow image reconstruction device for embodied intelligence, characterized in that: include: A video splitting module is used to obtain the depth image video stream of the binocular camera and split it into consecutively set image frames; An optical flow generation module, which is used to generate an image optical flow map according to the image frame; An optical flow processing module, which is used to input the optical flow map into an optical flow image reconstruction model to obtain a compensation posture corresponding to the image optical flow map; An image reconstruction module is used to reconstruct the image frame based on the compensated posture.
9. An electronic device, characterized in that: include: Memory and processor; The memory is used to store programs; The processor, coupled to the memory, is configured to execute the program to: Get the depth image video stream of the stereo camera and split it into consecutively set image frames; generating an image optical flow map according to the image frame; Inputting the optical flow map into an optical flow image reconstruction model to obtain a compensation posture corresponding to the image optical flow map; The image frame is reconstructed based on the compensated pose.
10. A computer-readable storage medium having a computer program stored thereon, characterized in that: The program is executed by a processor to implement an optical flow image reconstruction method for embodied intelligence as described in any one of claims 1 to 7.