Intelligent obstacle target identification method and device
By segmenting and processing the real depth video captured by binocular cameras, combining obstacle recognition and spatiotemporal prediction models, the problem of obstacle recognition only identifying static targets in the prior art is solved, accurate identification of dynamic obstacles and motion trajectory prediction are achieved, and the reliability of robotic arm control is improved.
Patent Information
- Application Number
- CN202411537015.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-10-31
- Publication Date
- 2025-03-25
AI Technical Summary
The current barrier identification of real-depth videos can only identify static targets, resulting in uncertainty in subsequent robotic arm control.
By taking real depth videos taken by binocular cameras, segmenting them into continuous depth image frames, and inputting them into barrier identification models to identify static and dynamic barriers. Then, the marked depth image frame is input into the spatiotemporal prediction model to predict the spatial motion trajectory of the dynamic obstacle.
Accurate identification of obstacles in real-life videos and prediction of motion trajectory of dynamic obstacles are achieved, eliminating the uncertainty of robotic arm control.
Smart Images
Figure CN119693911A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of medical image recognition technology, and more specifically, to a method and device for identifying obstacle targets for embodied intelligence. Background Art
[0002] Embodied intelligence refers to intelligent entities that have bodies and support interaction with the physical world, such as robots and driverless cars. By processing multiple sensor data inputs, a control center such as a large model generates motion commands to drive the intelligent entity, replacing the traditional motion-driving method based on rules or mathematical formulas, and achieving a deep integration of virtual and reality.
[0003] Surgical robots can serve as the perfect carrier of embodied intelligence, realizing the functions of surgical robots based on embodied intelligence, such as enabling surgical robots to avoid obstacles based on real depth videos during surgery.
[0004] The current target recognition of real depth videos can only identify static targets, which can easily bring great uncertainty to the subsequent robotic arm control through target recognition alone. Summary of the invention
[0005] The problem addressed by this application is that current obstacle recognition only recognizes static targets.
[0006] To solve the above problems, the first aspect of the present application provides an obstacle target recognition method for embodied intelligence, comprising:
[0007] Get the real depth video captured by the binocular camera and split it into continuously set depth image frames;
[0008] Inputting the depth image frame into the obstacle recognition model to obtain an obstacle recognition result; the obstacle recognition result includes static obstacles and dynamic obstacles;
[0009] The continuous depth image frames with the obstacle recognition results marked are input into the spatiotemporal prediction model to obtain the obstacle prediction results; the obstacle prediction results include the spatial motion trajectory of the dynamic obstacle.
[0010] The second aspect of the present application provides an obstacle target identification device for embodied intelligence, which includes:
[0011] An image segmentation module is used to obtain the real depth video captured by the binocular camera and segment it into continuously set depth image frames;
[0012] An obstacle recognition module, which is used to input the depth image frame into the obstacle recognition model to obtain an obstacle recognition result; the obstacle recognition result includes static obstacles and dynamic obstacles;
[0013] The spatiotemporal prediction module is used to input continuous depth image frames with marked obstacle recognition results into the spatiotemporal prediction model to obtain obstacle prediction results; the obstacle prediction results include the spatial motion trajectory of dynamic obstacles.
[0014] A third aspect of the present application provides an electronic device, comprising: a memory and a processor;
[0015] The memory is used to store programs;
[0016] The processor, coupled to the memory, is configured to execute the program to:
[0017] Get the real depth video captured by the binocular camera and split it into continuously set depth image frames;
[0018] Inputting the depth image frame into the obstacle recognition model to obtain an obstacle recognition result; the obstacle recognition result includes static obstacles and dynamic obstacles;
[0019] The continuous depth image frames with the obstacle recognition results marked are input into the spatiotemporal prediction model to obtain the obstacle prediction results; the obstacle prediction results include the spatial motion trajectory of the dynamic obstacle.
[0020] A fourth aspect of the present application provides a computer-readable storage medium having a computer program stored thereon, wherein the program is executed by a processor to implement the above-mentioned obstacle target identification method for embodied intelligence.
[0021] In this application, obstacle recognition is first used to identify the obstacle target in the real depth video, and then the spatial motion trajectory of the dynamic obstacle is determined through spatiotemporal prediction, so as to truly identify the corresponding video and eliminate the uncertainty of subsequent robotic arm control. BRIEF DESCRIPTION OF THE DRAWINGS
[0022] Figure 1 is a flow chart of an obstacle target identification method for embodied intelligence according to an embodiment of the present application;
[0023] Figure 2 It is an architecture diagram of spatiotemporal prediction of an obstacle target identification method for embodied intelligence according to an embodiment of the present application;
[0024] Figure 3 is a structural block diagram of an obstacle target identification device for embodied intelligence according to an embodiment of the present application;
[0025] Figure 4 is a structural block diagram of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION
[0026] In order to make the above-mentioned purposes, features and advantages of the present application more obvious and easy to understand, the specific embodiments of the present application are described in detail below in conjunction with the accompanying drawings. Although the exemplary embodiments of the present application are shown in the accompanying drawings, it should be understood that the present application can be implemented in various forms and should not be limited by the embodiments described herein. On the contrary, these embodiments are provided in order to enable a more thorough understanding of the present application and to fully convey the scope of the present application to those skilled in the art.
[0027] It should be noted that, unless otherwise specified, the technical terms or scientific terms used in this application should have the common meanings understood by technicians in the field to which this application belongs.
[0028] The present application embodiment provides a method for identifying obstacles for embodied intelligence, and the specific scheme of the method is as follows: Figure 1-Figure 2 As shown, the method can be performed by an obstacle target recognition device for embodied intelligence, and the obstacle target recognition device for embodied intelligence can be integrated into electronic devices such as computers, servers, computers, server clusters, and data centers. Figure 1 , Figure 2 As shown, it is a flow chart of an obstacle target identification method for embodied intelligence according to an embodiment of the present application; wherein the obstacle target identification method for embodied intelligence includes:
[0029] S101, obtaining a real depth video captured by a binocular camera, and dividing it into continuously set depth image frames;
[0030] In this application, a binocular camera is used to shoot real-time depth video data, and the scene depth information recorded by the video is captured as the original input.
[0031] In the present application, continuous depth video data is segmented into a plurality of continuous depth image frames to form a frame sequence, so as to identify and process obstacles frame by frame.
[0032] S102, inputting the depth image frame into an obstacle recognition model to obtain an obstacle recognition result; the obstacle recognition result includes static obstacles and dynamic obstacles;
[0033] In the present application, the segmented depth image frames are input into the obstacle recognition model; the model analyzes each frame of the image, identifies and classifies the obstacle type, and obtains the obstacle recognition result.
[0034] In this application, static obstacles refer to instruments and equipment, etc., and dynamic obstacles refer to moving people, doctors, etc.
[0035] S103, inputting the continuous depth image frames with the obstacle recognition results marked into the spatiotemporal prediction model to obtain an obstacle prediction result; the obstacle prediction result includes the spatial motion trajectory of the dynamic obstacle.
[0036] In this application, continuous depth image frames with obstacle annotations are input into a spatiotemporal prediction model, which can predict the future movement trajectory of obstacles based on their historical position information.
[0037] In this application, obstacle recognition is first used to identify the obstacle target in the real depth video, and then the spatial motion trajectory of the dynamic obstacle is determined through spatiotemporal prediction, so as to truly identify the corresponding video and eliminate the uncertainty of subsequent robotic arm control.
[0038] In one implementation, before inputting the depth image frame into the obstacle recognition model to obtain the obstacle recognition result, the method further includes:
[0039] Obtaining the target position and the preparation position, and generating a connection channel between the target position and the preparation position;
[0040] Divide the space through which the connection channel passes in the depth image frame, and perform connected domain detection based on the entities in the space, retaining the connected parts beyond the entities;
[0041] The processed depth image frame is used as the image frame of the input obstacle recognition model.
[0042] In one implementation, the target position and the preparation position are acquired by an NDI device disposed on one side of the binocular camera.
[0043] In the present application, by fixing a positioning frame at the surgical site and fixing a positioning frame at the end of the robotic arm, the NDI device identifies the position of the positioning frame to determine the target position and the preparation position.
[0044] In the present application, a connection channel is generated based on the target position and the preparation position as a virtual path of the robot or robotic arm from the starting point to the end point in the depth image frame.
[0045] In the present application, a spatial area through which a connection channel passes is divided in a depth image frame, and the spatial area is used as a key monitoring range.
[0046] In the present application, within the divided space, connected domain detection is performed based on entities in the depth image to identify obstacles and connected parts of other entities in the connecting channel area.
[0047] In this application, entities that are not connected to the channel connection area are deleted, and only the connected parts beyond the entities are retained to ensure that there are only obstacles that must be avoided on the connection channel.
[0048] In the present application, the depth image frames that have been detected and processed by the connected domain are input into the obstacle recognition model. The model can more accurately identify static and dynamic obstacles on the connecting channel and obtain clear obstacle recognition results.
[0049] In the present application, connected domain detection is performed so that only obstacles associated with the connection channel are retained, thereby greatly reducing the amount of identification data.
[0050] It should be noted that the conventional obstacle recognition model processes the complete picture or depth image input and identifies the obstacles therein.
[0051] The obstacle recognition model in the present application divides the space through which the connection channel passes in the depth image frame, and performs connected domain detection based on the entities in the space, retains the connected parts beyond the entities, sets the non-retained parts in the depth image frame to white, and then inputs the entire depth image frame into the obstacle recognition model.
[0052] In this way, by setting it to white, the complete depth image input obstacle recognition model is maintained on the one hand, and the amount of data that needs to be recognized is greatly reduced on the other hand.
[0053] In the present application, after obtaining the training samples, the obstacle recognition model is trained by randomly generating connection channels within the training samples and obtaining the corresponding images (retaining the connected parts beyond the entity and setting the non-retained parts in the depth image frame to white) through the above processing process to obtain the processed training samples and the annotations of the training samples (the annotations only retain the obstacle recognition results of the connection channels).
[0054] In this application, the specific model training method will not be repeated.
[0055] In one embodiment, dividing the space through which the connection channel passes in the depth image frame, and performing connected domain detection based on entities in the space, and retaining the connected parts beyond the entities, includes:
[0056] Perform edge detection on the two-dimensional image of the depth image frame and obtain the contour formed by the closed edge;
[0057] Determine the pixel correspondence between the three-dimensional image and the two-dimensional image of the depth image frame;
[0058] Acquire pixel points in the space through which the connection channel passes in the three-dimensional image as channel pixel points;
[0059] According to the pixel correspondence, the channel pixel points are mapped into the two-dimensional image, and the contour of the two-dimensional image containing the channel pixel points is retained.
[0060] In the present application, edge detection is performed on the two-dimensional image of the depth image frame (eg, using the Canny edge detection algorithm) to identify edge information in the image.
[0061] In the present application, edge detection is performed to obtain contours formed by closed edges, and these contours can help identify potential obstacles and areas.
[0062] In the present application, the pixel correspondence between the three-dimensional image and the two-dimensional image of the depth image frame is determined. Usually, this can be achieved by the internal and external parameters of the camera, and the three-dimensional points are mapped to the two-dimensional image plane using a projection model.
[0063] In the present application, pixels in the space through which the connection channel passes are extracted from the three-dimensional image, and these pixels are regarded as channel pixels. The relevant pixels can be identified by defining the three-dimensional coordinate range of the channel.
[0064] In the present application, the channel pixels are mapped back to the two-dimensional image according to the previously determined pixel correspondence.
[0065] In the present application, the outline of the pixel points of the channel in the two-dimensional image is retained, so that the area through which the connecting channel passes can be effectively displayed.
[0066] In one embodiment, in combination Figure 2 As shown, the continuous depth image frames with the obstacle recognition results marked are input into the spatiotemporal prediction model to obtain the obstacle prediction results, including:
[0067] Based on the obstacle recognition results, the key points of the dynamic obstacles are marked in the depth image frame;
[0068] Generate a continuous feature map according to the position key points of continuous depth image frames;
[0069] The continuous feature map is input into a spatiotemporal prediction model to obtain the spatial motion trajectory of the dynamic obstacle.
[0070] In this application, based on the obstacle recognition results, the key points of the dynamic obstacle positions are marked in the depth image frame. These key points may include the center point or other geometric features (such as boundary points) of the dynamic obstacle, which are used to characterize the spatial position of the obstacle.
[0071] In this application, the annotated key points are tracked in continuous depth image frames to form continuous feature maps. These feature maps show the position changes of dynamic obstacles over a period of time, providing time series information for the spatiotemporal prediction model.
[0072] In one embodiment, the spatiotemporal prediction model is a gated dilated causal convolutional neural network.
[0073] In each frame of the depth image, the position key points of the dynamic obstacles are marked according to the obstacle recognition results. These key points are used to capture the precise position of the obstacles in space.
[0074] In the present application, the key point information annotated in the depth image frame is converted into continuous feature maps, and these feature maps reflect the position changes of dynamic obstacles in continuous frames.
[0075] In this application, the continuous feature map can reflect the time series information of dynamic obstacles and provide rich data input for spatiotemporal modeling.
[0076] In this application, the continuous feature map is input into the gated dilated causal convolutional neural network:
[0077] The gating mechanism: Gated dilation The gating mechanism in the causal convolutional neural network can control the flow of information, allowing the model to focus on key features and ignore unimportant background noise when processing time series data.
[0078] The dilated convolution allows the network to increase the receptive field without increasing the computational complexity, enabling it to capture feature changes over long time intervals, thereby improving the model's ability to model the long-term dependencies of dynamic obstacle motion.
[0079] Causal convolution: Causal convolution ensures the timing characteristics of the network, that is, the model can only use past frames to predict the trajectory of the current frame or future frames, preventing information leakage and conforming to causality.
[0080] In this application, a gated dilation causal convolutional neural network outputs the future motion trajectory of a dynamic obstacle, including the future position, speed, and possible direction of motion.
[0081] In one embodiment, in combination Figure 2 As shown, generating a continuous feature map according to the position key points of continuous depth image frames includes:
[0082] Constructing a domain space graph structure according to the position key points;
[0083] Based on the correlation between the key points of the same dynamic obstacle, a weighted adjacency matrix of the domain space graph structure is constructed;
[0084] According to the domain space graph structure of continuous depth image frames and the corresponding weighted adjacency matrix, feature representations of the domain space graph structure at different times are generated, namely, the continuous feature graph.
[0085] In this application, the key points of the positions of dynamic obstacles are used as nodes to construct a domain space graph structure. Each node represents a key point of the position of a dynamic obstacle in a depth image frame.
[0086] In this application, the edges between nodes indicate that these key points are associated with each other in space, and key points at adjacent positions are usually connected to each other.
[0087] In the present application, a weighted adjacency matrix of a domain space graph structure is constructed based on the association relationship between the position key points of the same dynamic obstacle in consecutive image frames.
[0088] In the present application, each weight value in the weighted adjacency matrix can be set according to the distance or the degree of association, and nodes that are closer or more strongly associated will have larger weights, thereby better representing the local motion characteristics of the dynamic obstacle.
[0089] In this application, the domain space graph structure and weighted adjacency matrix of continuous depth image frames are used to generate feature representations of dynamic obstacles at different times, namely, continuous feature maps.
[0090] In the present application, the feature representation at each moment describes the graph structure information at that moment, reflecting the motion characteristics of the dynamic obstacle in the spatial and temporal dimensions.
[0091] An embodiment of the present application provides an obstacle target identification device for embodied intelligence, which is used to execute an obstacle target identification method for embodied intelligence described in the above content of the present application. The obstacle target identification device for embodied intelligence is described in detail below.
[0092] like Figure 3 As shown, the obstacle target recognition device for embodied intelligence includes:
[0093] An image segmentation module 101 is used to obtain a real depth video captured by a binocular camera and segment it into continuously set depth image frames;
[0094] The obstacle recognition module 102 is used to input the depth image frame into the obstacle recognition model to obtain an obstacle recognition result; the obstacle recognition result includes static obstacles and dynamic obstacles;
[0095] The spatiotemporal prediction module 103 is used to input the continuous depth image frames with the obstacle recognition results marked into the spatiotemporal prediction model to obtain the obstacle prediction results; the obstacle prediction results include the spatial motion trajectory of the dynamic obstacle.
[0096] In one implementation, the obstacle identification module 102 is further configured to:
[0097] The target position and the preparation position are obtained, and a connecting channel between the target position and the preparation position is generated; the space through which the connecting channel passes is divided in the depth image frame, and connected domain detection is performed based on the entities in the space, and the connected parts beyond the entities are retained; the processed depth image frame is used as the image frame of the input obstacle recognition model.
[0098] In one implementation, the target position and the preparation position are acquired by an NDI device disposed on one side of the binocular camera.
[0099] In one implementation, the obstacle identification module 102 is further configured to:
[0100] Perform edge detection on the two-dimensional image of the depth image frame, and obtain the contour formed by the closed edge; determine the pixel correspondence between the three-dimensional image of the depth image frame and the two-dimensional image; obtain the pixel points in the space through which the connecting channel passes in the three-dimensional image as the channel pixel points; map the channel pixel points to the two-dimensional image according to the pixel correspondence, and retain the contour containing the channel pixel points in the two-dimensional image.
[0101] In one embodiment, the spatiotemporal prediction module 103 is further used to:
[0102] Based on the obstacle recognition results, the position key points of the dynamic obstacle are marked in the depth image frame; a continuous feature map is generated according to the position key points of the continuous depth image frames; the continuous feature map is input into the spatiotemporal prediction model to obtain the spatial motion trajectory of the dynamic obstacle.
[0103] In one embodiment, the spatiotemporal prediction model is a gated dilated causal convolutional neural network.
[0104] In one embodiment, the spatiotemporal prediction module 103 is further used to:
[0105] According to the position key points, a domain space graph structure is constructed; based on the correlation between the position key points of the same dynamic obstacle, a weighted adjacency matrix of the domain space graph structure is constructed; according to the domain space graph structure of continuous depth image frames and the corresponding weighted adjacency matrix, a feature representation of the domain space graph structure at different times is generated, which is the continuous feature graph.
[0106] The above describes the internal functions and structure of an obstacle target recognition device for embodied intelligence, such as Figure 4 As shown, in practice, the obstacle target recognition device for embodied intelligence can be implemented as an electronic device, including: a memory 301 and a processor 303.
[0107] The memory 301 may be configured to store programs.
[0108] In addition, the memory 301 may also be configured to store various other data to support operations on the electronic device. Examples of such data include instructions for any application or method operating on the electronic device, contact data, phone book data, messages, pictures, videos, etc.
[0109] The memory 301 may be implemented by any type of volatile or nonvolatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.
[0110] The processor 303 is coupled to the memory 301 and is configured to execute a program in the memory 301 to:
[0111] Get the real depth video captured by the binocular camera and split it into continuously set depth image frames;
[0112] Inputting the depth image frame into the obstacle recognition model to obtain an obstacle recognition result; the obstacle recognition result includes static obstacles and dynamic obstacles;
[0113] The continuous depth image frames with the obstacle recognition results marked are input into the spatiotemporal prediction model to obtain the obstacle prediction results; the obstacle prediction results include the spatial motion trajectory of the dynamic obstacle.
[0114] In one implementation, the processor 303 is further configured to:
[0115] The target position and the preparation position are obtained, and a connecting channel between the target position and the preparation position is generated; the space through which the connecting channel passes is divided in the depth image frame, and connected domain detection is performed based on the entities in the space, and the connected parts beyond the entities are retained; the processed depth image frame is used as the image frame of the input obstacle recognition model.
[0116] In one implementation, the target position and the preparation position are acquired by an NDI device disposed on one side of the binocular camera.
[0117] In one implementation, the processor 303 is further configured to:
[0118] Perform edge detection on the two-dimensional image of the depth image frame, and obtain the contour formed by the closed edge; determine the pixel correspondence between the three-dimensional image of the depth image frame and the two-dimensional image; obtain the pixel points in the space through which the connecting channel passes in the three-dimensional image as the channel pixel points; map the channel pixel points to the two-dimensional image according to the pixel correspondence, and retain the contour containing the channel pixel points in the two-dimensional image.
[0119] In one implementation, the processor 303 is further configured to:
[0120] Based on the obstacle recognition results, the position key points of the dynamic obstacle are marked in the depth image frame; a continuous feature map is generated according to the position key points of the continuous depth image frames; the continuous feature map is input into the spatiotemporal prediction model to obtain the spatial motion trajectory of the dynamic obstacle.
[0121] In one embodiment, the spatiotemporal prediction model is a gated dilated causal convolutional neural network.
[0122] In one implementation, the processor 303 is further configured to:
[0123] According to the position key points, a domain space graph structure is constructed; based on the correlation between the position key points of the same dynamic obstacle, a weighted adjacency matrix of the domain space graph structure is constructed; according to the domain space graph structure of continuous depth image frames and the corresponding weighted adjacency matrix, a feature representation of the domain space graph structure at different times is generated, which is the continuous feature graph.
[0124] In this application, Figure 4 Only some components are shown schematically, which does not mean that the electronic device only includes Figure 4 Components shown.
[0125] The electronic device provided in this embodiment is based on the same inventive concept as the obstacle target identification method for embodied intelligence provided in the embodiment of the present application, and has the same beneficial effects as the method adopted, run or implemented by the application stored in it.
[0126] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-readable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.
[0127] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.
[0128] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.
[0129] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.
[0130] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.
[0131] The memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.
[0132] The present application also provides a computer-readable storage medium corresponding to a method for identifying an obstacle target for embodied intelligence provided in the aforementioned embodiment, on which a computer program (i.e., a program product) is stored. When the computer program is run by a processor, it will execute a method for identifying an obstacle target for embodied intelligence provided in any of the aforementioned embodiments.
[0133] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.
[0134] The computer-readable storage medium provided in the above-mentioned embodiments of the present application and the obstacle target identification method for embodied intelligence provided in the embodiments of the present application are based on the same inventive concept and have the same beneficial effects as the methods adopted, run or implemented by the application programs stored therein.
[0135] It should be noted that in the description provided herein, a large number of specific details are described. However, it is understood that the embodiments of the present application can be practiced without these specific details. In some instances, well-known structures and technologies are not shown in detail so as not to obscure the understanding of this description.
[0136] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.
[0137] The above is only an embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the scope of the claims of the present application.
Claims
1. A method for identifying obstacles for embodied intelligence, characterized in that: include: Get the real depth video captured by the binocular camera and split it into continuously set depth image frames; Inputting the depth image frame into the obstacle recognition model to obtain an obstacle recognition result; the obstacle recognition result includes static obstacles and dynamic obstacles; The continuous depth image frames with the obstacle recognition results marked are input into the spatiotemporal prediction model to obtain the obstacle prediction results; the obstacle prediction results include the spatial motion trajectory of the dynamic obstacle.
2. The obstacle target identification method for embodied intelligence according to claim 1, characterized in that: Before inputting the depth image frame into the obstacle recognition model to obtain the obstacle recognition result, the method further includes: Obtaining the target position and the preparation position, and generating a connection channel between the target position and the preparation position; Divide the space through which the connection channel passes in the depth image frame, and perform connected domain detection based on the entities in the space, retaining the connected parts beyond the entities; The processed depth image frame is used as the image frame of the input obstacle recognition model.
3. The obstacle target identification method for embodied intelligence according to claim 2, characterized in that: The target position and the preparation position are acquired through an NDI device arranged on one side of the binocular camera.
4. The obstacle target identification method for embodied intelligence according to claim 2, characterized in that: The method of dividing the space through which the connection channel passes in the depth image frame, and performing connected domain detection based on entities in the space, and retaining connected parts beyond the entities, includes: Perform edge detection on the two-dimensional image of the depth image frame and obtain the contour formed by the closed edge; Determine the pixel correspondence between the three-dimensional image and the two-dimensional image of the depth image frame; Acquire pixel points in the space through which the connection channel passes in the three-dimensional image as channel pixel points; According to the pixel correspondence, the channel pixel points are mapped into the two-dimensional image, and the contour of the two-dimensional image containing the channel pixel points is retained.
5. The obstacle target identification method for embodied intelligence according to claim 2, characterized in that: The step of inputting the continuous depth image frames with the obstacle recognition results marked thereon into the spatiotemporal prediction model to obtain the obstacle prediction results comprises: Based on the obstacle recognition results, the key points of the dynamic obstacles are marked in the depth image frame; Generate a continuous feature map according to the position key points of continuous depth image frames; The continuous feature map is input into a spatiotemporal prediction model to obtain the spatial motion trajectory of the dynamic obstacle.
6. The obstacle target identification method for embodied intelligence according to claim 5, characterized in that: The spatiotemporal prediction model is a gated dilated causal convolutional neural network.
7. The obstacle target identification method for embodied intelligence according to claim 5, characterized in that: The step of generating a continuous feature map according to the position key points of the continuous depth image frames includes: Constructing a domain space graph structure according to the position key points; Based on the correlation between the key points of the same dynamic obstacle, a weighted adjacency matrix of the domain space graph structure is constructed; According to the domain space graph structure of continuous depth image frames and the corresponding weighted adjacency matrix, feature representations of the domain space graph structure at different times are generated, namely, the continuous feature graph.
8. An obstacle target recognition device for embodied intelligence, characterized in that: include: An image segmentation module is used to obtain the real depth video captured by the binocular camera and segment it into continuously set depth image frames; An obstacle recognition module, which is used to input the depth image frame into the obstacle recognition model to obtain an obstacle recognition result; the obstacle recognition result includes static obstacles and dynamic obstacles; The spatiotemporal prediction module is used to input continuous depth image frames with marked obstacle recognition results into the spatiotemporal prediction model to obtain obstacle prediction results; the obstacle prediction results include the spatial motion trajectory of dynamic obstacles.
9. An electronic device, characterized in that: include: Memory and processor; The memory is used to store programs; The processor, coupled to the memory, is configured to execute the program to: Get the real depth video captured by the binocular camera and split it into continuously set depth image frames; Inputting the depth image frame into the obstacle recognition model to obtain an obstacle recognition result; the obstacle recognition result includes static obstacles and dynamic obstacles; The continuous depth image frames with the obstacle recognition results marked are input into the spatiotemporal prediction model to obtain the obstacle prediction results; the obstacle prediction results include the spatial motion trajectory of the dynamic obstacle.
10. A computer-readable storage medium having a computer program stored thereon, characterized in that: The program is executed by a processor to implement an obstacle target identification method for embodied intelligence as described in any one of claims 1-7.