A method, device and storage medium for determining a grasping pose

Abstract

This invention discloses a method, apparatus, device, and storage medium for determining a grasping posture. It acquires a visual image and a depth image of the current scene, as well as natural language information input by the user. The natural language information describes the feature information of the object to be grasped. Based on the visual image and the natural language information, it determines the availability points of the object to be grasped in the visual image. Based on the availability points and the visual image, it determines a mask image of the object to be grasped. Based on the visual image, depth image, camera parameters, and the mask image, it determines the grasping posture of the object to be grasped. By directly locating the availability points of the object through natural language information, and using these points to guide local segmentation to generate an accurate mask, and then combining the depth image and camera parameters to calculate the 3D grasping posture, it avoids the blindness and category dependence of traditional methods in complex, multi-object, and similar object scenes, significantly improving the accuracy and efficiency of grasping.

Description

Technical Field

[0001] This invention relates to the field of robotics, and in particular to a method, apparatus, device, and storage medium for determining grasping posture. Background Technology

[0002] In the fields of modern automation and robotics, object grasping is one of the core issues in achieving intelligent production and services.

[0003] However, existing object grasping technologies still fall short in accuracy and efficiency when dealing with complex scenarios involving multiple or similar objects. Summary of the Invention

[0004] This invention provides a method, apparatus, device, and storage medium for determining grasping posture, in order to solve the problems of insufficient accuracy and efficiency in existing object grasping technologies.

[0005] According to one aspect of the present invention, a method for determining a grasping posture is provided, comprising: The system acquires visual and depth images of the current scene, as well as natural language information input by the user, wherein the natural language information is used to describe the feature information of the object to be grasped. Based on the visual image and the natural language information, determine the availability points of the object to be grasped in the visual image; Based on the available location points and the visual image, determine the mask image of the object to be grasped; The grasping posture of the object to be grasped is determined based on the visual image, the depth image, the camera parameters, and the mask image.

[0006] According to another aspect of the present invention, a grasping posture determination device is provided, comprising: The data acquisition module is used to acquire visual and depth images of the current scene, as well as natural language information input by the user, wherein the natural language information is used to describe the feature information of the object to be grasped. The positioning point determination module is used to determine the available positioning points of the object to be grasped in the visual image based on the visual image and the natural language information; A mask determination module is used to determine a mask image of the object to be grasped based on the available positioning points and the visual image. The pose determination module is used to determine the grasping pose of the object to be grasped based on the visual image, the depth image, camera parameters, and the mask image.

[0007] According to another aspect of the present invention, an electronic device is provided, the electronic device comprising: At least one processor; and A memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform the grasping posture determination method according to any embodiment of the present invention.

[0008] According to another aspect of the present invention, a computer-readable storage medium is provided, the computer-readable storage medium storing computer instructions for causing a processor to execute and implement the grasping posture determination method according to any embodiment of the present invention.

[0009] According to another aspect of the present invention, a computer program product is provided, the computer program product comprising a computer program that, when executed by a processor, implements the grasping posture determination method according to any embodiment of the present invention.

[0010] The technical solution of this invention involves acquiring a visual image and a depth image of the current scene, as well as natural language information input by the user. The natural language information describes the feature information of the object to be grasped. Based on the visual image and the natural language information, the available location points of the object to be grasped in the visual image are determined. Based on the available location points and the visual image, a mask image of the object to be grasped is determined. Based on the visual image, the depth image, camera parameters, and the mask image, the grasping posture of the object to be grasped is determined. By directly locating the available grasping points of the object using natural language information, and using these points to guide local segmentation to generate a precise mask, and then combining the depth image and camera parameters to calculate the 3D grasping posture, the invention avoids the blindness and category dependence of traditional methods in complex, multi-object, and similar object scenes, significantly improving the accuracy and efficiency of grasping.

[0011] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of the present invention, nor is it intended to limit the scope of the invention. Other features of the invention will become readily apparent from the following description. Attached Figure Description

[0012] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0013] Figure 1 This is a flowchart of a grasping posture determination method provided in Embodiment 1 of the present invention; Figure 2 This is a schematic diagram of an availability point provided in Embodiment 1 of the present invention; Figure 3 This is a schematic diagram of a grasping posture determination device provided in Embodiment 2 of the present invention; Figure 4 This is a schematic diagram of the structure of an electronic device provided in Embodiment 3 of the present invention. Detailed Implementation

[0014] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0015] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0016] Example 1 Figure 1 This is a flowchart of a grasping posture determination method provided in Embodiment 1 of the present invention. This embodiment is applicable to determining the grasping posture of an object to be grasped in complex multi-object scenarios. The method can be executed by a grasping posture determination device, which can be implemented in hardware and / or software and can be configured in an electronic device. Figure 1 As shown, the method includes: S110. Obtain the visual image and depth image of the current scene, as well as the natural language information input by the user, wherein the natural language information is used to describe the feature information of the object to be grasped.

[0017] In this embodiment, the visual image is a color image of the current scene captured by an image acquisition device (such as an RGB camera), used to provide appearance information such as color and texture of the current scene. The depth image is an image registered with the visual image, captured by a depth camera, containing distance information from each pixel in the current scene to the camera. Natural language information is a command input by the user through a human-computer interaction method, used to describe the feature information of the object to be grasped. The human-computer interaction method includes text input and voice input; that is, in some embodiments, natural language information is obtained through text input and / or voice input. The text includes, but is not limited to, Chinese and English text.

[0018] Specifically, in scenarios where robots perform grasping tasks, RGB cameras and depth cameras are used to acquire visual and depth images of the current scene, while audio acquisition devices and / or text input devices are used to acquire natural language information input by the user.

[0019] S120. Based on the visual image and the natural language information, determine the availability points of the object to be grasped in the visual image.

[0020] In this embodiment, the availability point can be understood as a position indicator point of the object to be grasped in the visual image. For example, in some embodiments, the availability point is a two-dimensional coordinate point that marks the object to be grasped in the visual image.

[0021] Specifically, cross-modal fusion of visual images and natural language information establishes a correspondence between linguistic descriptions and image regions, thereby determining the availability points of the object to be grasped in the visual image. Through cross-modal feature fusion, abstract semantic descriptions are mapped to specific locations in image space, providing accurate prompts for subsequent segmentation processing.

[0022] For example, Figure 2 This is a schematic diagram of an availability point provided in Embodiment 1 of the present invention. Figure 2 As shown, in a scenario containing multiple coffee capsules 20 of different colors, the natural language information input by the user is "the rightmost black coffee capsule". The visual image outputs availability points marked with red dots, which are used to indicate the position of the object to be grasped corresponding to the language description.

[0023] S130. Based on the availability points and the visual image, determine the mask image of the object to be grasped.

[0024] In this embodiment, the mask image is a pixel-level contour region of the object to be grasped, segmented from the visual image based on available location points, and the mask image is a binary image corresponding to the object to be grasped.

[0025] Specifically, after acquiring the availability points, based on these availability points and the visual image, an image segmentation algorithm is used to process the visual image, using the availability points as cue input, to segment the pixel-level contour region of the object to be grasped, thus obtaining a mask image. The image segmentation algorithm includes, but is not limited to, various deep learning-based segmentation models, edge detection segmentation algorithms, region growing segmentation algorithms, and clustering segmentation algorithms.

[0026] S140. Based on the visual image, the depth image, the camera parameters, and the mask image, determine the grasping posture of the object to be grasped.

[0027] In this embodiment, the grasping posture can be understood as the pose parameters of the robotic arm when grasping the object to be grasped, including the spatial position of the grasping point and the grasping posture angle.

[0028] Specifically, after acquiring the mask image, based on the visual image, depth image, camera parameters, and mask image, a grasping pose prediction algorithm comprehensively analyzes the geometric shape, spatial position, and other features of the object to be grasped, thereby predicting and outputting the optimal grasping pose. The grasping pose prediction algorithm includes, but is not limited to, deep learning-based grasping prediction models, reinforcement learning algorithms, and optimization algorithms.

[0029] The technical solution provided in Embodiment 1 of this invention acquires a visual image and a depth image of the current scene, as well as natural language information input by the user. The natural language information describes the feature information of the object to be grasped. Based on the visual image and the natural language information, the available location points of the object to be grasped in the visual image are determined. Based on the available location points and the visual image, a mask image of the object to be grasped is determined. Based on the visual image, the depth image, camera parameters, and the mask image, the grasping posture of the object to be grasped is determined. By directly locating the available grasping points of the object using natural language information, and using these points to guide local segmentation to generate a precise mask, and then combining the depth image and camera parameters to calculate the 3D grasping posture, the blindness and category dependence of full-image search in complex, multi-object, and similar object scenes of traditional methods are avoided, significantly improving the accuracy and efficiency of grasping.

[0030] In some embodiments, determining the availability points of the object to be grasped in the visual image based on the visual image and the natural language information includes: fusing the visual image and the natural language information using a preset visual language large model to obtain the availability points of the object to be grasped in the visual image.

[0031] In this embodiment, the preset visual language big model can be understood as a pre-trained deep learning model used for cross-modal fusion processing of visual images and natural language information.

[0032] Specifically, visual images and natural language information are input into a preset visual language model. The preset visual language model establishes a correspondence between language descriptions and image regions, mapping abstract semantic descriptions to specific locations in image space. The output indicates the location of the object to be captured in the visual image, i.e., a location point.

[0033] The above technical solution fully utilizes the cross-modal understanding capabilities of the visual language big model, enabling precise positioning of target objects in complex multi-object scenarios and improving the accuracy and efficiency of object positioning.

[0034] In some embodiments, the step of fusing the visual image and the natural language information using a preset visual language large model to obtain the availability points of the object to be grasped in the visual image includes: extracting the semantic features of the natural language information and the visual features of the visual image respectively; performing cross-modal fusion on the semantic features and the visual features to obtain fused features; and determining the availability points of the object to be grasped in the visual image based on the fused features.

[0035] In this embodiment, semantic features are vector representations extracted from natural language information that characterize the meaning of linguistic descriptions. Visual features are vector representations extracted from visual images that characterize the content information of the images. Cross-modal fusion is the process of establishing the association between language and image data to achieve the alignment and integration of semantic and visual information.

[0036] Specifically, after inputting visual images and natural language information into a pre-defined visual-language model, the model extracts semantic features from the natural language information and visual features from the visual images. Cross-modal fusion of these semantic and visual features generates a fused feature that combines semantic understanding and visual perception capabilities. Based on this fused feature, the position of the object to be grasped, corresponding to the natural language description, is predicted in the visual image space; this is the provisional localization point. This effectively achieves precise alignment between the language description and the image space, improving the accuracy and robustness of target object localization.

[0037] In some embodiments, determining a mask image of the object to be grasped based on the availability points and the visual image includes: processing the visual image using a segmentation network based on the availability points to obtain a mask image of the object to be grasped.

[0038] In this embodiment, the segmentation network is a pre-trained deep learning model used for instance segmentation of visual images, including but not limited to instance segmentation networks based on point cues, semantic segmentation networks, or panoptic segmentation networks.

[0039] Specifically, based on availability points, these availability points are used as foreground cues input into the segmentation network. The segmentation network locates the target region in the visual image based on these foreground cues. A foreground-background discrimination mechanism distinguishes the object to be grasped from the background and other objects, suppressing responses from non-target objects. The network then extracts the object region corresponding to the foreground cues, generating a precise pixel-level contour mask, thus obtaining the mask image of the object to be grasped. By using availability points as foreground cues input into the segmentation network, the network's point-cue instance segmentation capability is fully utilized, achieving accurate segmentation of the target object and improving the accuracy and reliability of object contour extraction in complex multi-object scenes.

[0040] In some embodiments, the feature information includes at least one of appearance feature information, spatial location feature information, and category feature information.

[0041] In this embodiment, appearance feature information is used to describe the visual appearance attributes of the object to be grasped, including but not limited to color, shape, size, and texture. Spatial location feature information is used to describe the relative or absolute position of the object to be grasped in space, including but not limited to absolute orientation (e.g., far left, far right, in the middle) and positional relationship relative to other objects (e.g., above or next to an object). Category feature information is used to describe the type of the object to be grasped, including but not limited to object name (e.g., coffee capsule, cup, box) or category label.

[0042] This embodiment enables a multi-dimensional and accurate description of the object to be grasped, improving the accuracy and flexibility of target differentiation and localization in complex multi-object scenarios.

[0043] The technical solution provided by this invention achieves multimodal fusion through a large vision-language model, comprehensively considers language and visual information for intelligent object recognition and localization, supports users to flexibly describe target objects in Chinese and English natural language, accurately locates similar objects in complex multi-object environments (localization accuracy exceeds 98%) and determines the optimal grasping posture, accurately distinguishes the position and posture of different objects, thereby completing efficient and reliable grasping tasks, significantly improving user experience and system adaptability.

[0044] Example 2 Figure 3 This is a schematic diagram of a grasping posture determination device provided in Embodiment 2 of the present invention. Figure 3 As shown, the device includes: The data acquisition module 21 is used to acquire visual images and depth images of the current scene, as well as natural language information input by the user, wherein the natural language information is used to describe the feature information of the object to be grasped. The positioning point determination module 22 is used to determine the available positioning points of the object to be grasped in the visual image based on the visual image and the natural language information; The mask determination module 23 is used to determine the mask image of the object to be grasped based on the availability positioning points and the visual image. The pose determination module 24 is used to determine the grasping pose of the object to be grasped based on the visual image, the depth image, camera parameters and the mask image.

[0045] Optionally, the positioning point determination module 22 is specifically used to fuse the visual image and the natural language information through a preset visual language large model to obtain the availability positioning point of the object to be grasped in the visual image.

[0046] The technical solution provided in Embodiment 2 of the present invention directly locates the available grasping points of objects through natural language information, uses these points to guide local segmentation to generate a precise mask, and then combines depth images and camera parameters to calculate the 3D grasping posture. This avoids the blindness and category dependence of the full-image search in complex, multi-object and similar object scenes of traditional methods, and significantly improves the accuracy and efficiency of grasping.

[0047] Optionally, the positioning point determination module 22 includes: The feature extraction unit is used to extract the semantic features of the natural language information and the visual features of the visual image, respectively. The feature fusion unit is used to perform cross-modal fusion of the semantic features and the visual features to obtain fused features; The positioning point determination unit is used to determine the available positioning points of the object to be grasped in the visual image based on the fused features.

[0048] Optionally, the mask determination module 23 is specifically used to process the visual image using a segmentation network based on the available positioning points to obtain a mask image of the object to be grasped.

[0049] Optionally, the feature information includes at least one of appearance feature information, spatial location feature information, and category feature information.

[0050] Optionally, the natural language information is obtained through text input and / or voice input.

[0051] Optionally, the availability point is a two-dimensional coordinate point in the visual image that marks the object to be grasped.

[0052] The grasping posture determination device provided in the embodiments of the present invention can execute the grasping posture determination method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the method execution.

[0053] Example 3 Figure 4 This is a schematic diagram of an electronic device provided in Embodiment 3 of the present invention. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices (such as helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the invention described and / or claimed herein.

[0054] like Figure 4 As shown, the electronic device 10 includes at least one processor 11 and a memory, such as a read-only memory (ROM) 12 or a random access memory (RAM) 13, communicatively connected to the at least one processor 11. The memory stores computer programs executable by the at least one processor. The processor 11 can perform various appropriate actions and processes based on the computer program stored in the ROM 12 or loaded from storage unit 18 into the RAM 13. The RAM 13 can also store various programs and data required for the operation of the electronic device 10. The processor 11, ROM 12, and RAM 13 are interconnected via a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0055] Multiple components in electronic device 10 are connected to I / O interface 15, including: input unit 16, such as keyboard, mouse, etc.; output unit 17, such as various types of displays, speakers, etc.; storage unit 18, such as disk, optical disk, etc.; and communication unit 19, such as network card, modem, wireless transceiver, etc. Communication unit 19 allows electronic device 10 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0056] Processor 11 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, digital signal processors (DSPs), and any suitable processor, controller, microcontroller, etc. Processor 11 performs the various methods and processes described above, such as the grasping pose determination method.

[0057] In some embodiments, the grasping posture determination method may be implemented as a computer program tangibly contained in a computer-readable storage medium, such as storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or installed on electronic device 10 via ROM 12 and / or communication unit 19. When the computer program is loaded into RAM 13 and executed by processor 11, one or more steps of the grasping posture determination method described above may be performed. Alternatively, in other embodiments, processor 11 may be configured to perform the grasping posture determination method by any other suitable means (e.g., by means of firmware).

[0058] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0059] Computer programs used to implement the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when executed by the processor, the computer programs cause the functions / operations specified in the flowcharts and / or block diagrams to be performed. The computer programs may be executed entirely on a machine, partially on a machine, or as a standalone software package, partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0060] In the context of this invention, a computer-readable storage medium can be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, apparatus, or device. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination thereof. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0061] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0062] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or middleware components (e.g., application servers), or frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), blockchain networks, and the Internet.

[0063] A computing system can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a hosting product within the cloud computing service system to address the shortcomings of traditional physical hosts and VPS services, such as high management difficulty and weak business scalability.

[0064] It should be understood that the various forms of processes shown above can be used, with steps reordered, added, or deleted. For example, the steps described in this invention can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this invention can be achieved, and this is not limited herein.

[0065] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

[0066] This invention also provides a computer program product, including a computer program and / or instructions, which, when executed by a processor, implements the grasping posture determination method provided in any embodiment of this application.

[0067] In implementing the computer program product, computer program code for performing the operations of this invention can be written in one or more programming languages ​​or a combination thereof. Programming languages ​​include object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages ​​such as C or similar languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0068] Note that the above description is merely a preferred embodiment of the present invention and the technical principles employed. Those skilled in the art will understand that the present invention is not limited to the specific embodiments described herein, and various obvious changes, readjustments, and substitutions can be made without departing from the scope of protection of the present invention. Therefore, although the present invention has been described in detail through the above embodiments, the present invention is not limited to the above embodiments, and may include many other equivalent embodiments without departing from the concept of the present invention, the scope of which is determined by the scope of the appended claims.

Claims

1. A method for determining grasping posture, characterized in that, include: The system acquires visual and depth images of the current scene, as well as natural language information input by the user, wherein the natural language information is used to describe the feature information of the object to be grasped. Based on the visual image and the natural language information, determine the availability points of the object to be grasped in the visual image; Based on the available location points and the visual image, determine the mask image of the object to be grasped; The grasping posture of the object to be grasped is determined based on the visual image, the depth image, the camera parameters, and the mask image.

2. The method according to claim 1, characterized in that, Based on the visual image and the natural language information, determining the availability points of the object to be grasped in the visual image includes: By fusing the visual image and the natural language information using a pre-defined visual language model, the availability points of the object to be grasped in the visual image are obtained.

3. The method according to claim 2, characterized in that, The step of fusing the visual image and the natural language information using a preset visual language model to obtain the availability points of the object to be grasped in the visual image includes: Semantic features of the natural language information and visual features of the visual image are extracted respectively; The semantic features and the visual features are fused across modally to obtain fused features; Based on the fusion features, the availability points of the object to be grasped in the visual image are determined.

4. The method according to claim 1, characterized in that, Based on the available location points and the visual image, determine the mask image of the object to be grasped, including: Based on the available location points, the visual image is processed using a segmentation network to obtain a mask image of the object to be grasped.

5. The method according to claim 1, characterized in that, The feature information includes at least one of appearance feature information, spatial location feature information, and category feature information.

6. The method according to claim 1, characterized in that, The natural language information is obtained through text input and / or voice input.

7. The method according to claim 1, characterized in that, The availability point is a two-dimensional coordinate point in the visual image that marks the object to be grasped.

8. A grasping posture determination device, characterized in that, include: The data acquisition module is used to acquire visual and depth images of the current scene, as well as natural language information input by the user, wherein the natural language information is used to describe the feature information of the object to be grasped. The positioning point determination module is used to determine the available positioning points of the object to be grasped in the visual image based on the visual image and the natural language information. A mask determination module is used to determine a mask image of the object to be grasped based on the available positioning points and the visual image. The pose determination module is used to determine the grasping pose of the object to be grasped based on the visual image, the depth image, camera parameters, and the mask image.

9. An electronic device, characterized in that, The electronic device includes: At least one processor; and A memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform the grasping posture determination method according to any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that cause a processor to execute the grasping posture determination method according to any one of claims 1-7.

Images