Data processing methods, apparatus, equipment and computer media

By acquiring user commands and environmental images, and using a target interaction model to analyze the environmental images, the intelligent robot can accurately identify and grasp target objects, solving the problem of low grasping efficiency and improving the success rate of human-computer interaction grasping.

CN117484502BActive Publication Date: 2026-11-17BEIJING YOUZHUJU NETWORK TECH CO LTD
View PDF 2 Cites 0 Cited by

Patent Information

Application Number
CN202311562350.2
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2023-11-21
Publication Date
2026-11-17
Estimated Expiration
2043-11-21

AI Technical Summary

Technical Problem

When grasping objects, intelligent robots cannot accurately identify target objects according to user instructions, resulting in low grasping efficiency.

Method used

By acquiring user commands and environmental images, the system analyzes the objects to be captured in the environmental images that correspond to the commands using a preset target interaction model, and then controls the capturing device to capture them.

Benefits of technology

It enables intelligent robots to accurately identify target objects in open environments and improve grasping efficiency, significantly increasing the success rate of human-computer interaction grasping.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN117484502B_ABST
    Figure CN117484502B_ABST
Patent Text Reader

Abstract

This disclosure discloses a data processing method, apparatus, device, and computer medium. The method includes: acquiring user instructions and an environmental image; determining, based on the instructions, the environmental image, and a preset target interaction model, an object to be grasped in the environmental image corresponding to the instructions; and controlling a grasping device to grasp the target item corresponding to the object to be grasped, which can improve the grasping efficiency of intelligent robots.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This disclosure pertains to the field of intelligent control, and particularly relates to a data processing method, apparatus, device, and computer medium. Background Technology

[0002] In related technologies, with the development of intelligent robot technologies, the application of intelligent robots is becoming increasingly widespread, and grasping ability is an important capability of intelligent robots. However, in some related technologies, intelligent robots cannot accurately identify the target object according to the user's instructions when grasping objects, resulting in low grasping efficiency. Summary of the Invention

[0003] This disclosure provides an implementation scheme different from related technologies to solve the technical problem in related technologies that intelligent robots cannot accurately identify corresponding target items according to user instructions, resulting in low grasping efficiency.

[0004] In a first aspect, this disclosure provides a data processing method, the method comprising:

[0005] Obtain user commands and environmental images;

[0006] Based on the instruction, the environmental image, and the preset target interaction model, determine the object to be captured in the environmental image that corresponds to the instruction;

[0007] Control the grasping device to grasp the target item corresponding to the object to be grasped.

[0008] Secondly, this disclosure provides a data processing apparatus, comprising:

[0009] The acquisition unit is used to acquire user commands and environmental images;

[0010] The determining unit is used to determine, based on the instruction, the environmental image, and a preset target interaction model, the object to be captured in the environmental image that corresponds to the instruction;

[0011] The control unit is used to control the gripping device to grip the target item corresponding to the object to be gripped.

[0012] Thirdly, this disclosure provides an electronic device, including:

[0013] Processor; and

[0014] Memory for storing the executable instructions of the processor;

[0015] The processor is configured to execute the first aspect, or any of the possible implementations of the first aspect, by executing the executable instructions.

[0016] Fourthly, embodiments of this disclosure provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the first aspect or any of the methods in various possible implementations of the first aspect.

[0017] The scheme disclosed herein, which acquires user instructions and environmental images; determines the object to be grasped in the environmental image corresponding to the instruction based on the instruction, the environmental image, and a preset target interaction model; and controls the grasping device to grasp the target item corresponding to the object to be grasped, can analyze the environmental image and instructions in real time through the preset target interaction model. It introduces a new algorithm, namely the target interaction model and the analysis of the environmental image, which enables the intelligent robot to accurately identify the corresponding target item according to the user's instructions when grasping items, resulting in high grasping efficiency. Attached Figure Description

[0018] To more clearly illustrate the technical solutions in the embodiments or related technologies of this disclosure, the accompanying drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, the accompanying drawings described below are some embodiments of this disclosure. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. In the drawings:

[0019] Figure 1 This is a schematic diagram of the structure of a system provided in one embodiment of the present disclosure;

[0020] Figure 2a A schematic flowchart illustrating a data processing method provided in an embodiment of this disclosure;

[0021] Figure 2b A schematic diagram of a scenario for a data processing method provided in an embodiment of this disclosure;

[0022] Figure 3 This is a schematic diagram of the structure of a data processing apparatus provided in an embodiment of the present disclosure;

[0023] Figure 4 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present disclosure. Detailed Implementation

[0024] Embodiments of this disclosure are described in detail below, with examples of these embodiments illustrated in the accompanying drawings. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain this disclosure, and should not be construed as limiting it.

[0025] The terms "first" and "second," etc., used in the specification, claims, and drawings of this disclosure are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that embodiments of the present disclosure described herein can be implemented, for example, in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0026] First, some terms used in the embodiments of this disclosure will be explained below to facilitate understanding by those skilled in the art.

[0027] Interactive Visual Semantic Disambiguation (IVSD) dialogue refers to resolving semantic ambiguity in visual information through human-machine interaction. Typically, IVSD dialogue employs a natural language-based interaction method, allowing human users to converse with the machine and provide additional information or explanations in natural language to help the machine better understand the semantic information in images.

[0028] The inventors discovered through research that, with the development of related technologies for intelligent robots and their increasingly widespread applications, grasping ability is a crucial capability for intelligent robots. However, in related technologies, intelligent robots often struggle to accurately identify target objects based on user instructions, resulting in low grasping efficiency.

[0029] Furthermore, when the intelligent robot performs tasks based on user instructions, it locates the target related to the task based on the interaction with the user and then performs the task accordingly. This method places high demands on the intelligent robot's comprehension ability; if the intelligent robot cannot accurately locate the target based on the interaction with the user, the accuracy of task execution is low.

[0030] Furthermore, the robot's ability to disambiguate visual semantics in interactive scenarios is weak. The intelligent robot cannot accurately understand its surroundings or analyze targets within the scene, all of which affect the accuracy of the intelligent robot's task execution.

[0031] The technical solutions of this disclosure and how they solve the aforementioned technical problems will be described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of this disclosure will now be described with reference to the accompanying drawings.

[0032] Figure 1 A schematic diagram of a system provided for an exemplary embodiment of this disclosure, the system including: a model training device 10 and an intelligent robot 11, wherein:

[0033] The intelligent robot 11 can be used to: acquire user instructions and environmental images; determine the object to be grasped in the environmental image corresponding to the instruction based on the instructions, the environmental images, and a preset target interaction model; and control the grasping device to grasp the target item corresponding to the object to be grasped.

[0034] Model training device 10 can be used for:

[0035] The task of obtaining samples;

[0036] Based on the type of the sample task, obtain the corresponding sample interaction information and the sample image corresponding to the sample interaction information; the sample interaction information includes: sample interaction content and sample task execution result;

[0037] The initial word processing unit in the initial interaction model performs word processing on the sample task and the sample interaction content to obtain the corresponding word vector sequence.

[0038] The sample image is encoded by the initial visual encoding unit in the initial interaction model to obtain the image feature sequence corresponding to the sample image;

[0039] The word vector sequence and the image feature sequence are analyzed using the initial transformation model in the initial interaction model to obtain the corresponding task prediction results;

[0040] The initial interaction model is trained based on the sample task execution results and the task prediction results to obtain the target interaction model.

[0041] Optionally, the aforementioned target interaction model can also be trained by the intelligent robot 11 itself.

[0042] The execution principles and interaction processes of each component unit in this system embodiment, such as the model training device 10 and the intelligent robot 11, can be found in the descriptions of the following method embodiments.

[0043] Figure 2aThis is a flowchart illustrating a data processing method provided as an exemplary embodiment of the present disclosure. The method can be performed by the aforementioned intelligent robot, or other devices with mobility and grasping capabilities, wherein the intelligent robot has mobility and grasping capabilities. The method includes at least the following steps S21-S23:

[0044] S21. Obtain user commands and environmental images;

[0045] In some optional embodiments of this application, the user's instructions can be any of the following types: voice instructions, gesture instructions, or instructions triggered through an interactive screen. The interactive screen can be the screen of the intelligent robot, or it can be the screen of a terminal device used by the user, such as a mobile phone or computer. When the interactive screen is the screen of a terminal device, the terminal device can send instructions to the intelligent robot.

[0046] Optionally, the aforementioned environmental images are acquired by the camera device of the intelligent robot, and the user's instructions are given at the same time as the acquisition of the environmental images.

[0047] S22. Based on the instruction, the environmental image, and the preset target interaction model, determine the object to be captured in the environmental image that corresponds to the instruction;

[0048] In some optional embodiments of this application, the user's instruction is the first interaction content with the user. In the aforementioned S22, the object to be captured in the environment image corresponding to the instruction is determined based on the instruction, the environment image, and the preset target interaction model, including the following S221-S222:

[0049] S221. Analyze the first preset task, the first interaction content, and the environmental image through the target interaction model to obtain the task execution result corresponding to the first preset task. The first preset task is a task based on the first interaction content to determine whether the object to be captured can be specified in the environmental image.

[0050] Specifically, in the aforementioned S221, the first preset task, the first interactive content, and the environmental image are analyzed by the target interaction model to obtain the task execution result corresponding to the first preset task, including: inputting the first preset task, the first interactive content, and the environmental image into the target interaction model to obtain the task execution result corresponding to the first preset task.

[0051] S222. Based on the task execution result, determine the object to be captured in the environmental image that corresponds to the instruction.

[0052] In some optional embodiments of this application, the aforementioned step S222, determining the object to be captured in the environmental image corresponding to the instruction based on the task execution result, includes the following steps S2221-S2223:

[0053] S2221. If the task execution result is yes, then obtain the second preset task. The task type of the second preset task is: based on the first interaction content, the task of pointing out the first region information of the image region occupied by the object to be captured in the environment image.

[0054] S2222: Input the second preset task, the first interactive content, and the environmental image into the target interactive model to obtain the first region information;

[0055] S2223. The object within the first area information is taken as the object to be captured in the environment image corresponding to the instruction.

[0056] In some optional embodiments of this application, the method further includes the following S01-S05:

[0057] S01. If the task execution result is negative, then obtain a third preset task. The task type of the third preset task is: a task to ask a question based on the first interactive content.

[0058] S02. Input the third preset task, the first interactive content, and the environmental image into the target interactive model to obtain the corresponding question content;

[0059] S03. Ask the user a question based on the question content;

[0060] This solution can disambiguate interactive content.

[0061] S04. Obtain the second interactive content of the user's reply based on the question content;

[0062] S05. Take the second interaction content and the question content as the new first interaction content, and return to execute the analysis of the first preset task, the first interaction content, and the environmental image through the target interaction model to obtain the task execution result corresponding to the first preset task, until the object to be captured can be specified in the environmental image.

[0063] S23. Control the grasping device to grasp the target item corresponding to the object to be grasped.

[0064] In some optional embodiments of this application, the aforementioned S23, controlling the grasping device to grasp the target item corresponding to the object to be grasped, includes the following S2031-S2032:

[0065] S2031. Control the grasping device to move to the target location where the target item corresponding to the object to be grasped is located;

[0066] S2032. Control the gripping device to grip the target item at the target location.

[0067] In some optional embodiments of this application, the method further includes the following S1-S2:

[0068] S1. Input the environmental image and the first region information into a preset segmentation model to obtain the mask corresponding to the environmental image;

[0069] The segmentation model can be implemented using Segment anything.

[0070] In the aforementioned mask, the mask information corresponding to the first region information in the environmental image is 1, and the mask information corresponding to the image regions other than the first region information in the environmental image is 0.

[0071] S2. Input the depth image information corresponding to the mask and the environment image into a preset grasping model to obtain the target position. The target position may specifically include pose information. This pose information can be a stable, collision-free grasping pose.

[0072] In some embodiments, the aforementioned grasping model may output multiple candidate locations. In this case, the method further includes: determining a target location from the multiple candidate locations. Determining the target location from the multiple candidate locations may include: selecting the target location closest to the target item from the candidate locations.

[0073] The crawling model can be implemented using ContactGraspNet or ContactGraspNet.

[0074] Optionally, the gripping device can be a robotic arm, and the target position can be the location of the object to be gripped in the real world.

[0075] In some optional embodiments of this application, the method further includes: performing a legality check on the first interactive content; and when the first interactive content is determined to be legal, determining to execute the analysis of the first preset task, the first interactive content, and the environmental image through the target interactive model to obtain the task execution result corresponding to the first preset task.

[0076] Optionally, the semantic legality of the first interaction content can be analyzed using ChatGPT. If the semantics are unclear, the first interaction content is determined to be illegal; if the semantics of the first interaction content are clear, the first interaction content is determined to be legal.

[0077] If the first interaction is determined to be invalid, the user can be prompted to re-enter the command.

[0078] In some optional embodiments of this application, after determining that the first interactive content is legal, the method further includes: detecting whether the environmental image contains candidate objects of the same type as the object corresponding to the first interactive content based on a preset detection model; if so, determining to execute the analysis of the first preset task, the first interactive content, and the environmental image through the target interaction model to obtain the task execution result corresponding to the first preset task.

[0079] In some optional embodiments of this application, the candidate objects belong to the same category as the object corresponding to the first interactive content; the category to which the aforementioned candidate objects belong is the object corresponding to the first interactive content. For example, when the object corresponding to the first interactive content is a beverage, the candidate objects would be cola, mineral water, etc.

[0080] Optionally, the aforementioned detection model can be a model trained by machine learning.

[0081] In some optional embodiments of this application, the aforementioned detection model can be a preset large-scale pre-trained vision-language model, and the aforementioned environmental image can be acquired by an RGB-D depth camera. If it is determined that the environmental image does not contain any candidate objects of the same type as the object corresponding to the first interactive content, the user can be directly prompted that no object corresponding to the first interactive content exists.

[0082] In some optional embodiments of this application, the method further includes the following S201-S206:

[0083] S201, Sample Acquisition Task;

[0084] In some optional embodiments of this application, the sample task is any of the following tasks: answering a question posed by a questioner, making a corresponding question based on the interaction content with the respondent, indicating the second region information of the image area occupied by the target object corresponding to the interaction content in the sample image based on the interaction content with the questioner or the respondent, answering whether the target object corresponding to the interaction content can be specified in the sample image based on the interaction content with the questioner or the respondent, or describing the sample image.

[0085] In some optional embodiments of this application, the task of answering the question raised by the questioner may include: answering the question raised by the questioner based on a sample image, specifically, it may include: answering the question raised by the questioner based on a preset area in the sample image.

[0086] In some alternative embodiments, the sample task can be text information or voice information, or it can be instruction information triggered through an interactive screen.

[0087] S202. Obtain the corresponding sample interaction information and the sample image corresponding to the sample interaction information based on the type of the sample task; the sample interaction information includes: sample interaction content and sample task execution result;

[0088] In some optional embodiments of this application, the sample interaction information may include interaction information between multiple roles, such as multiple dialogue messages. Specifically, a dialogue message may include the role's name and the dialogue content between the role and its opposing role. The dialogue content may include text information and / or voice information.

[0089] In some embodiments of this application, the sample interaction content precedes the sample task execution result in the sample interaction information, and the sample interaction content is adjacent to the sample task execution result. Both the sample interaction content and the sample task execution result belong to dialogue information.

[0090] In some optional embodiments of this application, the step of obtaining the corresponding sample interaction information based on the type of the sample task in S202 includes the following S2021-S2023:

[0091] S2021. Determine the corresponding interaction role based on the type of the sample task, wherein the interaction role is a questioner or a respondent;

[0092] In some optional embodiments of this application, determining the corresponding interaction role based on the type of the sample task in S2021 may include: determining the interaction role corresponding to the type of the sample task based on a preset correspondence. The preset correspondence may store different correspondences between sample task types and their corresponding interaction roles.

[0093] In some optional embodiments of this application, when the type of the sample task is to answer a question raised by a questioner, the interaction role corresponding to the type of sample task is the answerer.

[0094] In some optional embodiments of this application, when the type of the sample task is a task that asks a question based on the interaction content with the respondent, the interaction role corresponding to the type of sample task is the questioner.

[0095] In some optional embodiments of this application, when the type of sample task is a task based on the interaction content with the questioner, indicating the second region information of the image area occupied by the target object corresponding to the interaction content in the sample image, the interaction role corresponding to the type of sample task is the respondent.

[0096] In some optional embodiments of this application, when the type of sample task is a task that, based on the interaction content with the respondent, points out the second region information of the image area occupied by the target object corresponding to the interaction content in the sample image, the interaction role corresponding to the type of sample task is the questioner.

[0097] In some optional embodiments of this application, when the type of the sample task is a task based on the interaction content with the questioner, answering whether the target object corresponding to the interaction content can be specified in the sample image, the interaction role corresponding to the type of sample task is the respondent.

[0098] In some optional embodiments of this application, when the type of sample task is a task based on the interaction content with the respondent, answering whether the target object corresponding to the interaction content can be specified in the sample image, the interaction role corresponding to the type of sample task is the questioner.

[0099] In some optional embodiments of this application, when the type of the sample task is a task describing the sample image, the interaction role corresponding to the type of sample task is a responder or a questioner.

[0100] S2022. Based on the interaction role, determine the sample task execution result from the target sample interaction information set;

[0101] Optionally, the sample interaction information base may contain multiple sample interaction information sets. Each sample interaction information set can be considered as a group of dialogue information, and each group of dialogue information may include multiple dialogue messages. The target sample interaction information set is the sample interaction information set in the sample interaction information base that corresponds to the type of sample task.

[0102] Optionally, the above method further includes: acquiring multiple sample interaction information sets; and determining the target sample interaction information set corresponding to the type of the sample task from the multiple sample interaction information sets. Specifically, each type of sample task corresponds to one sample interaction information set.

[0103] In some optional embodiments of this application, in the aforementioned S2022, determining the sample task execution result from the target sample interaction information set based on the interaction role includes: randomly selecting a dialogue content between the interaction role and its opposing role from the target sample interaction information set; and using the dialogue content as the sample task execution result.

[0104] In some other optional embodiments of this application, in the aforementioned S2022, determining the sample task execution result from the target sample interaction information set based on the interaction role includes: determining the sample task execution result from the target sample interaction information set based on the interaction role and the type of the sample task.

[0105] Optionally, the aforementioned determination of the sample task execution result from the target sample interaction information set based on the type of the interaction role and the sample task includes: when the type of the sample task is to answer a question raised by the questioner, the dialogue content of the interaction role in the target sample interaction information set, where the preceding sentence is a question dialogue content of the opposing role, is taken as the sample task execution result, and the dialogue content of the interaction role is a declarative sentence.

[0106] Optionally, based on the type of the interaction role and the sample task, the sample task execution result is determined from the target sample interaction information set, including: when the type of the sample task is a task of making a corresponding question based on the interaction content with the respondent, the dialogue content of the interaction role in the target sample interaction information set, where the preceding sentence is the dialogue content of the opposing role, is taken as the sample task execution result, and the dialogue content of the interaction role is a question.

[0107] Optionally, the aforementioned determination of the sample task execution result from the target sample interaction information set based on the interaction role and the type of the sample task includes: when the type of the sample task is a task based on the interaction content with the questioner or the answerer, indicating the second region information of the image area occupied by the target object corresponding to the interaction content in the sample image, the dialogue content of the interaction role in the target sample interaction information set, where the preceding sentence is the dialogue content of the opposing role, is taken as the sample task execution result, and the dialogue content of the interaction role contains region information.

[0108] S2023. The target sample interaction information is collected, and the dialogue content before the sample task execution result is taken as the sample interaction content.

[0109] In some optional embodiments of this application, the target sample interaction information set contains a set of dialogue information.

[0110] Optionally, the set of dialogue information referred to in this application refers to the dialogue information that is clear and unambiguous to the target at the end of the dialogue.

[0111] In some other optional embodiments of this application, the target sample interaction information set may include multiple sets of dialogue information. In the aforementioned S2023, the dialogue content before the sample task execution result in the target sample interaction information set is used as the sample interaction content, which includes: using the dialogue content before the sample task execution result in any set of dialogue information in the target sample interaction information set as the sample interaction content.

[0112] When the type of the sample task is based on the interaction content with the questioner or responder, and the task is to answer whether a target object corresponding to the interaction content can be specified in the sample image, the step of obtaining the corresponding sample interaction information based on the type of the sample task includes the following S01-S03:

[0113] S01. Determine the corresponding interaction role based on the type of the sample task, wherein the interaction role is either a questioner or a respondent;

[0114] When the type of sample task is based on the interaction content with the questioner, and the task is to answer whether the target object corresponding to the interaction content can be specified in the sample image, the interaction role corresponding to the type of sample task is the respondent.

[0115] When the type of sample task is based on the interaction content with the respondent, and the task asks whether the target object corresponding to the interaction content can be specified in the sample image, the interaction role corresponding to the type of sample task is the questioner.

[0116] S02. Obtain the preset sample task execution result, wherein the preset sample task execution result is yes or no;

[0117] S03. When the sample task execution result is yes, and the target sample interaction information set includes only one set of dialogue information, then all dialogue content in the target sample interaction information set is taken as the sample interaction content; when the sample task execution result is no, and the target sample interaction information set includes only one set of dialogue information, then part of the dialogue content in the target sample interaction information set is taken as the sample interaction content. When the sample task execution result is yes, and the target sample interaction information set includes multiple sets of dialogue information, then all dialogue content in any set of dialogue information in the target sample interaction information set is taken as the sample interaction content; when the sample task execution result is no, and the target sample interaction information set includes multiple sets of dialogue information, then part of the dialogue content in any set of dialogue information in the target sample interaction information set is taken as the sample interaction content.

[0118] S203. The initial word processing unit in the initial interaction model performs word processing on the sample task and the sample interaction content to obtain the corresponding word vector sequence.

[0119] S204. The sample image is encoded by the initial visual encoding unit in the initial interaction model to obtain the image feature sequence corresponding to the sample image;

[0120] S205. Analyze the word vector sequence and the image feature sequence using the initial transformation model in the initial interaction model to obtain the corresponding task prediction result;

[0121] S206. The initial interaction model is trained based on the sample task execution results and the task prediction results to obtain the target interaction model.

[0122] Optionally, the target interaction model is used to: interact with the interaction object based on a first interaction content with the interaction object and an environmental image. Here, the interaction object may refer to a user.

[0123] In some optional embodiments of this application, in the aforementioned S206, the initial interaction model is trained based on the sample task execution result and the task prediction result to obtain the target interaction model, including the following S2061-S2062:

[0124] S2061. Determine loss information based on a preset loss function, the sample task execution result, and the task prediction result;

[0125] S2062. If the loss information is less than a preset threshold, the initial interaction model is used as the target interaction model; if the loss information is not less than the preset threshold, the parameters in the initial interaction model are adjusted, and the process returns to the step of performing word processing on the sample task and the sample interaction content through the initial word processing unit in the initial interaction model to obtain the corresponding word vector sequence, until the target interaction model is determined when the loss information is less than the preset threshold.

[0126] In some optional embodiments of this application, when the type of the sample task is: based on the interaction content with the questioner or answerer, the task of indicating the second region information of the image area occupied by the target object corresponding to the interaction content in the sample image, the second region information of the image area occupied by the target object in the sample image is used to indicate the coordinate range of the area occupied by the target object in the sample image.

[0127] Optionally, the target object corresponding to the interactive content is an object that is related to the interactive content.

[0128] In this application, the target interaction model includes: a target word processing unit, a target visual encoding unit, and a target transformation model.

[0129] Specifically, when the initial interaction model is trained to become the target interaction model, the initial word processing unit is trained to become the target word processing unit; the initial visual encoding unit is trained to become the target visual encoding unit; and the initial conversion model is trained to become the target conversion model.

[0130] Optionally, the aforementioned inputting the first preset task, the first interactive content, and the environmental image into the target interaction model to obtain the task execution result corresponding to the first preset task includes:

[0131] The first preset task and the first interaction content are processed by the target word processing unit in the target interaction model to obtain the corresponding word vector sequence.

[0132] The environmental image is encoded by the target visual encoding unit in the target interaction model to obtain the image feature sequence corresponding to the environmental image;

[0133] By analyzing the word vector sequence and the image feature sequence through the target transformation model in the target interaction model, the corresponding task execution results are obtained.

[0134] In some optional embodiments of this application, when the task type of the sample task is to answer a question raised by the questioner based on a preset region in the sample image, the aforementioned word processing of the sample task and the sample interaction content by the initial word processing unit in the initial interaction model to obtain the corresponding word vector sequence means: the initial word processing unit in the initial interaction model processes the sample task, the regional information of the preset region, and the sample interaction content to obtain the corresponding word vector sequence.

[0135] The following is an example of a scene. Figure 2b Further explanation of the scheme in this application:

[0136] The user inputs a voice command: "I'm thirsty, get me something to drink."

[0137] The intelligent robot detects whether the voice command is valid. If it is, it determines whether there is a beverage in the environmental image. If not, it prompts the user to re-enter the command.

[0138] When a beverage is present in the environmental image, the target interaction model analyzes the first preset task, the first interaction content, and the environmental image to obtain the task execution result corresponding to the first preset task; when no beverage is present in the environmental image, the user is prompted that no beverage is present.

[0139] If the task execution result is yes, then the second preset task is obtained, and the second preset task, the first interactive content, and the environmental image are input into the target interactive model to obtain the first region information; the object in the first region information is taken as the object to be captured in the environmental image corresponding to the instruction.

[0140] If the task execution result is negative, a third preset task is obtained. The third preset task, the first interactive content, and the environmental image are input into the target interaction model to obtain the corresponding question content. The user is asked a question based on the question content. The second interactive content in response to the question content is obtained. The second interactive content and the question content are used as the new first interactive content. The execution is returned to the target interaction model to analyze the first preset task, the first interactive content, and the environmental image to obtain the task execution result corresponding to the first preset task, until the object to be captured can be specified in the environmental image.

[0141] The environmental image and the information of the first region are input into a preset segmentation model to obtain a mask corresponding to the environmental image; the mask and the depth image information corresponding to the environmental image are input into a preset grasping model to obtain the target position.

[0142] Control the gripping device to grab the target item at the target location.

[0143] In real-world experiments with multiple ambiguous and unseen objects in real-world home scenarios, the proposed solution achieved a human-computer interaction capture success rate of >85% (within which the success rate is 0-50% without human-computer interaction), demonstrating significant effectiveness.

[0144] The scheme disclosed herein, which acquires user instructions and environmental images; determines the object to be grasped in the environmental image corresponding to the instruction based on the instruction, the environmental image, and a preset target interaction model; and controls the grasping device to grasp the target item corresponding to the object to be grasped, can analyze the environmental image and instructions in real time through the preset target interaction model. It introduces a new algorithm, namely the target interaction model and the analysis of the environmental image, which enables the intelligent robot to accurately identify the corresponding target item according to the user's instructions when grasping items, resulting in high grasping efficiency.

[0145] The solution proposed in this application enables intelligent robots to understand complex visual relationships, human state behaviors, and complex user expressions more accurately and robustly in more open scenarios. This facilitates natural and accurate interaction between intelligent robots and humans in most indoor and outdoor scenarios, enabling them to accurately understand and complete human language commands.

[0146] Figure 3 A schematic diagram of the structure of a data processing apparatus provided for an exemplary embodiment of this disclosure;

[0147] The device includes:

[0148] Acquisition unit 31 is used to acquire user commands and environmental images;

[0149] The determining unit 32 is used to determine the object to be captured in the environment image corresponding to the instruction based on the instruction, the environment image, and the preset target interaction model;

[0150] The control unit 33 is used to control the gripping device to grip the target item corresponding to the object to be gripped.

[0151] Optionally, when the aforementioned device is used to control the grasping device to grasp the target item corresponding to the object to be grasped, it is specifically used for:

[0152] Control the grasping device to move to the target location where the target item corresponding to the object to be grasped is located;

[0153] The gripping device is controlled to grip the target item at the target location.

[0154] Optionally, the user's instruction is the first interaction content with the user, and the aforementioned device, when determining the object to be captured in the environment image corresponding to the instruction based on the instruction, the environment image, and a preset target interaction model, is specifically used for:

[0155] The target interaction model is used to analyze the first preset task, the first interaction content, and the environmental image to obtain the task execution result corresponding to the first preset task. The first preset task is a task based on the first interaction content to determine whether the object to be captured can be specified in the environmental image.

[0156] Based on the task execution result, determine the object to be captured in the environmental image that corresponds to the instruction.

[0157] Optionally, when the aforementioned device is used to determine the object to be captured in the environmental image corresponding to the instruction based on the task execution result, it is specifically used for:

[0158] If the task execution result is yes, then a second preset task is obtained. The task type of the second preset task is: based on the first interaction content, the task of pointing out the first region information of the image region occupied by the object to be captured in the environment image.

[0159] The second preset task, the first interactive content, and the environmental image are input into the target interactive model to obtain the first region information;

[0160] The objects within the first region information are taken as the objects to be captured in the environment image corresponding to the instruction.

[0161] Optionally, the aforementioned device is also used for:

[0162] If the task execution result is negative, a third preset task is obtained. The task type of the third preset task is: a task to ask a question based on the first interaction content.

[0163] The third preset task, the first interactive content, and the environmental image are input into the target interactive model to obtain the corresponding question content;

[0164] Ask the user a question based on the content of the question;

[0165] Obtain the second interactive content of the user's reply based on the question;

[0166] The second interactive content and the question content are used as new first interactive content. The process of analyzing the first preset task, the first interactive content, and the environmental image through the target interactive model is returned to obtain the task execution result corresponding to the first preset task, until the object to be captured can be specified in the environmental image.

[0167] Optionally, the aforementioned device is also used for:

[0168] Based on a preset detection model, it is detected whether the environmental image contains candidate objects of the same type as the object corresponding to the first interactive content. If so, it is determined to execute the task execution result corresponding to the first preset task by analyzing the first preset task, the first interactive content, and the environmental image through the target interaction model.

[0169] Optionally, the aforementioned device is also used for:

[0170] The first interactive content is subjected to a legality check. If the first interactive content is determined to be legal, the first preset task, the first interactive content, and the environmental image are analyzed through the target interactive model to obtain the task execution result corresponding to the first preset task.

[0171] Optionally, the aforementioned device is also used for:

[0172] The environmental image and the information of the first region are input into a preset segmentation model to obtain a mask corresponding to the environmental image;

[0173] The depth image information corresponding to the mask and the environment image is input into a preset grasping model to obtain the target position.

[0174] Optionally, the aforementioned device is also used for:

[0175] The task of obtaining samples;

[0176] Based on the type of the sample task, obtain the corresponding sample interaction information and the sample image corresponding to the sample interaction information; the sample interaction information includes: sample interaction content and sample task execution result;

[0177] The initial word processing unit in the initial interaction model performs word processing on the sample task and the sample interaction content to obtain the corresponding word vector sequence.

[0178] The sample image is encoded by the initial visual encoding unit in the initial interaction model to obtain the image feature sequence corresponding to the sample image;

[0179] The word vector sequence and the image feature sequence are analyzed using the initial transformation model in the initial interaction model to obtain the corresponding task prediction results;

[0180] The initial interaction model is trained based on the sample task execution results and the task prediction results to obtain the target interaction model.

[0181] It should be understood that the device embodiments and method embodiments can correspond to each other, and similar descriptions can be referred to the method embodiments. To avoid repetition, they will not be repeated here. Specifically, the device can execute the above method embodiments, and the foregoing and other operations and / or functions of each module in the device correspond to the corresponding processes in the various methods in the above method embodiments, which will not be repeated here for the sake of brevity.

[0182] The apparatus of this disclosure embodiment has been described above from the perspective of functional modules in conjunction with the accompanying drawings. It should be understood that this functional module can be implemented in hardware, in software instructions, or in a combination of hardware and software modules. Specifically, the steps of the method embodiments in this disclosure can be completed by integrated logic circuits in the processor's hardware and / or by software instructions. The steps of the method disclosed in this disclosure embodiment can be directly embodied as being executed by a hardware decoding processor, or by a combination of hardware and software modules in the decoding processor. Optionally, the software module can reside in a mature storage medium in the art, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, etc. This storage medium is located in memory, and the processor reads information from the memory and, in conjunction with its hardware, completes the steps in the above method embodiments.

[0183] Figure 4This is a schematic block diagram of an electronic device provided in an embodiment of this disclosure. The electronic device may include:

[0184] The system includes a memory 401 for storing computer programs and a processor 402 for transferring program code to the processor 402. In other words, the processor 402 can retrieve and run the computer program from the memory 401 to implement the methods described in this embodiment.

[0185] For example, the processor 402 can be used to execute the above-described method embodiments according to instructions in the computer program.

[0186] In some embodiments of this disclosure, the processor 402 may include, but is not limited to:

[0187] General-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc.

[0188] In some embodiments of this disclosure, the memory 401 includes, but is not limited to:

[0189] Volatile memory and / or non-volatile memory. Non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. Volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as Static RAM (SRAM), Dynamic RAM (DRAM), Synchronous DRAM (SDRAM), Double Data Rate SDRAM (DDR SDRAM), Enhanced Synchronous DRAM (ESDRAM), Synchronous Link DRAM (SLDRAM), and Direct Rambus RAM (DR RAM).

[0190] In some embodiments of this disclosure, the computer program may be divided into one or more modules, which are stored in the memory 401 and executed by the processor 402 to perform the method provided in this disclosure. The one or more modules may be a series of computer program instruction segments capable of performing a specific function, which describe the execution process of the computer program in the electronic device.

[0191] like Figure 4 As shown, the electronic device may further include:

[0192] Transceiver 403, which can be connected to processor 402 or memory 401.

[0193] The processor 402 can control the transceiver 403 to communicate with other devices; specifically, it can send information or data to other devices or receive information or data sent by other devices. The transceiver 403 may include a transmitter and a receiver. The transceiver 403 may further include antennas, and the number of antennas may be one or more.

[0194] It should be understood that the various components in the electronic device are connected through a bus system, which includes a data bus, a power bus, a control bus, and a status signal bus.

[0195] This disclosure also provides a computer storage medium storing a computer program thereon, which, when executed by a computer, enables the computer to perform the methods of the above-described method embodiments. Alternatively, embodiments of this disclosure also provide a computer program product containing instructions that, when executed by a computer, cause the computer to perform the methods of the above-described method embodiments.

[0196] When implemented using software, it can be implemented wholly or partially as a computer program product. This computer program product includes one or more computer instructions. When these computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this disclosure are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium accessible to a computer or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., digital video disc (DVD)), or a semiconductor medium (e.g., solid-state disk (SSD)).

[0197] According to one or more embodiments of this disclosure, a data processing method is provided, the method comprising:

[0198] Obtain user commands and environmental images;

[0199] Based on the instruction, the environmental image, and the preset target interaction model, determine the object to be captured in the environmental image that corresponds to the instruction;

[0200] Control the grasping device to grasp the target item corresponding to the object to be grasped.

[0201] According to one or more embodiments of this disclosure, controlling the grasping device to grasp a target item corresponding to the object to be grasped includes:

[0202] Control the grasping device to move to the target location where the target item corresponding to the object to be grasped is located;

[0203] The gripping device is controlled to grip the target item at the target location.

[0204] According to one or more embodiments of this disclosure, the user's instruction is a first interaction content with the user. Based on the instruction, the environmental image, and a preset target interaction model, determining the object to be captured in the environmental image corresponding to the instruction includes:

[0205] The target interaction model is used to analyze the first preset task, the first interaction content, and the environmental image to obtain the task execution result corresponding to the first preset task. The first preset task is a task based on the first interaction content to determine whether the object to be captured can be specified in the environmental image.

[0206] Based on the task execution result, determine the object to be captured in the environmental image that corresponds to the instruction.

[0207] According to one or more embodiments of this disclosure, determining the object to be captured in the environmental image corresponding to the instruction based on the task execution result includes:

[0208] If the task execution result is yes, then a second preset task is obtained. The task type of the second preset task is: based on the first interaction content, the task of pointing out the first region information of the image region occupied by the object to be captured in the environment image.

[0209] The second preset task, the first interactive content, and the environmental image are input into the target interactive model to obtain the first region information;

[0210] The objects within the first region information are taken as the objects to be captured in the environment image corresponding to the instruction.

[0211] According to one or more embodiments of this disclosure, the method further includes:

[0212] If the task execution result is negative, a third preset task is obtained. The task type of the third preset task is: a task to ask a question based on the first interaction content.

[0213] The third preset task, the first interactive content, and the environmental image are input into the target interactive model to obtain the corresponding question content;

[0214] Ask the user a question based on the content of the question;

[0215] Obtain the second interactive content of the user's reply based on the question;

[0216] The second interactive content and the question content are used as new first interactive content. The process of analyzing the first preset task, the first interactive content, and the environmental image through the target interactive model is returned to obtain the task execution result corresponding to the first preset task, until the object to be captured can be specified in the environmental image.

[0217] According to one or more embodiments of this disclosure, the method further includes:

[0218] Based on a preset detection model, it is detected whether the environmental image contains candidate objects of the same type as the object corresponding to the first interactive content. If so, it is determined to execute the task execution result corresponding to the first preset task by analyzing the first preset task, the first interactive content, and the environmental image through the target interaction model.

[0219] According to one or more embodiments of this disclosure, the method further includes:

[0220] The first interactive content is subjected to a legality check. If the first interactive content is determined to be legal, the first preset task, the first interactive content, and the environmental image are analyzed through the target interactive model to obtain the task execution result corresponding to the first preset task.

[0221] According to one or more embodiments of this disclosure, the method further includes:

[0222] The environmental image and the information of the first region are input into a preset segmentation model to obtain a mask corresponding to the environmental image;

[0223] The depth image information corresponding to the mask and the environment image is input into a preset grasping model to obtain the target position.

[0224] According to one or more embodiments of this disclosure, the method further includes:

[0225] The task of obtaining samples;

[0226] Based on the type of the sample task, obtain the corresponding sample interaction information and the sample image corresponding to the sample interaction information; the sample interaction information includes: sample interaction content and sample task execution result;

[0227] The initial word processing unit in the initial interaction model performs word processing on the sample task and the sample interaction content to obtain the corresponding word vector sequence.

[0228] The sample image is encoded by the initial visual encoding unit in the initial interaction model to obtain the image feature sequence corresponding to the sample image;

[0229] The word vector sequence and the image feature sequence are analyzed using the initial transformation model in the initial interaction model to obtain the corresponding task prediction results;

[0230] The initial interaction model is trained based on the sample task execution results and the task prediction results to obtain the target interaction model.

[0231] According to one or more embodiments of the present disclosure, a data processing apparatus is provided, the apparatus comprising:

[0232] The acquisition unit is used to acquire user commands and environmental images;

[0233] The determining unit is used to determine, based on the instruction, the environmental image, and a preset target interaction model, the object to be captured in the environmental image that corresponds to the instruction;

[0234] The control unit is used to control the gripping device to grip the target item corresponding to the object to be gripped.

[0235] According to one or more embodiments of this disclosure, when the aforementioned device is used to control the grasping device to grasp the target item corresponding to the object to be grasped, it is specifically used for:

[0236] Control the grasping device to move to the target location where the target item corresponding to the object to be grasped is located;

[0237] The gripping device is controlled to grip the target item at the target location.

[0238] According to one or more embodiments of this disclosure, the user's instruction is a first interaction content with the user, and the aforementioned device, when determining the object to be captured in the environment image corresponding to the instruction based on the instruction, the environment image, and a preset target interaction model, is specifically used for:

[0239] The target interaction model is used to analyze the first preset task, the first interaction content, and the environmental image to obtain the task execution result corresponding to the first preset task. The first preset task is a task based on the first interaction content to determine whether the object to be captured can be specified in the environmental image.

[0240] Based on the task execution result, determine the object to be captured in the environmental image that corresponds to the instruction.

[0241] According to one or more embodiments of this disclosure, when the aforementioned apparatus is used to determine the object to be captured in the environmental image corresponding to the instruction based on the task execution result, it is specifically used for:

[0242] If the task execution result is yes, then a second preset task is obtained. The task type of the second preset task is: based on the first interaction content, the task of pointing out the first region information of the image region occupied by the object to be captured in the environment image.

[0243] The second preset task, the first interactive content, and the environmental image are input into the target interactive model to obtain the first region information;

[0244] The objects within the first region information are taken as the objects to be captured in the environment image corresponding to the instruction.

[0245] According to one or more embodiments of this disclosure, the aforementioned apparatus is further used for:

[0246] If the task execution result is negative, a third preset task is obtained. The task type of the third preset task is: a task to ask a question based on the first interaction content.

[0247] The third preset task, the first interactive content, and the environmental image are input into the target interactive model to obtain the corresponding question content;

[0248] Ask the user a question based on the content of the question;

[0249] Obtain the second interactive content of the user's reply based on the question;

[0250] The second interactive content and the question content are used as new first interactive content. The process of analyzing the first preset task, the first interactive content, and the environmental image through the target interactive model is returned to obtain the task execution result corresponding to the first preset task, until the object to be captured can be specified in the environmental image.

[0251] According to one or more embodiments of this disclosure, the aforementioned apparatus is further used for:

[0252] Based on a preset detection model, it is detected whether the environmental image contains candidate objects of the same type as the object corresponding to the first interactive content. If so, it is determined to execute the task execution result corresponding to the first preset task by analyzing the first preset task, the first interactive content, and the environmental image through the target interaction model.

[0253] According to one or more embodiments of this disclosure, the aforementioned apparatus is further used for:

[0254] The first interactive content is subjected to a legality check. If the first interactive content is determined to be legal, the first preset task, the first interactive content, and the environmental image are analyzed through the target interactive model to obtain the task execution result corresponding to the first preset task.

[0255] According to one or more embodiments of this disclosure, the aforementioned apparatus is further used for:

[0256] The environmental image and the information of the first region are input into a preset segmentation model to obtain a mask corresponding to the environmental image;

[0257] The depth image information corresponding to the mask and the environment image is input into a preset grasping model to obtain the target position.

[0258] According to one or more embodiments of this disclosure, the aforementioned apparatus is further used for:

[0259] The task of obtaining samples;

[0260] Based on the type of the sample task, obtain the corresponding sample interaction information and the sample image corresponding to the sample interaction information; the sample interaction information includes: sample interaction content and sample task execution result;

[0261] The initial word processing unit in the initial interaction model performs word processing on the sample task and the sample interaction content to obtain the corresponding word vector sequence.

[0262] The sample image is encoded by the initial visual encoding unit in the initial interaction model to obtain the image feature sequence corresponding to the sample image;

[0263] The word vector sequence and the image feature sequence are analyzed using the initial transformation model in the initial interaction model to obtain the corresponding task prediction results;

[0264] The initial interaction model is trained based on the sample task execution results and the task prediction results to obtain the target interaction model.

[0265] According to one or more embodiments of this disclosure, an electronic device is provided, comprising:

[0266] Processor; and

[0267] Memory for storing the executable instructions of the processor;

[0268] The processor is configured to execute the aforementioned methods by executing the executable instructions.

[0269] According to one or more embodiments of the present disclosure, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the aforementioned methods.

[0270] Those skilled in the art will recognize that the modules and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this disclosure.

[0271] In the embodiments provided in this disclosure, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple modules or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or modules may be electrical, mechanical, or other forms.

[0272] The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical modules; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. For example, the functional modules in the various embodiments of this disclosure may be integrated into one processing module, or each module may exist physically separately, or two or more modules may be integrated into one module.

[0273] The above are merely specific embodiments of this disclosure, but the scope of protection of this disclosure is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this disclosure should be included within the scope of protection of this disclosure. Therefore, the scope of protection of this disclosure should be determined by the scope of the claims.

Claims

1. A data processing method, characterized in that, The method includes: Obtain user commands and environmental images; Based on the instruction, the environmental image, and the preset target interaction model, determine the object to be captured in the environmental image that corresponds to the instruction; Control the grasping device to grasp the target item corresponding to the object to be grasped; The user's instruction is the first interaction content with the user. Determining the object to be crawled in the environment image corresponding to the instruction based on the instruction, the environment image, and a preset target interaction model includes: analyzing a first preset task, the first interaction content, and the environment image through the target interaction model to obtain a task execution result corresponding to the first preset task, where the first preset task is a task based on the first interaction content to determine whether the object to be crawled can be specified in the environment image; and determining the object to be crawled in the environment image corresponding to the instruction based on the task execution result. If the task execution result is negative, a third preset task is obtained. The task type of the third preset task is: a task to ask a question based on the first interaction content; the third preset task, the first interaction content, and the environmental image are input into the target interaction model to obtain the corresponding question content; the user is asked a question based on the question content; the second interaction content of the user's reply based on the question content is obtained; the second interaction content and the question content are used as new first interaction content, and the process of analyzing the first preset task, the first interaction content, and the environmental image through the target interaction model to obtain the task execution result corresponding to the first preset task is repeated until the object to be captured can be specified in the environmental image.

2. The method according to claim 1, characterized in that, Controlling the grasping device to grasp the target item corresponding to the object to be grasped includes: Control the grasping device to move to the target location where the target item corresponding to the object to be grasped is located; The gripping device is controlled to grip the target item at the target location.

3. The method according to claim 2, characterized in that, Based on the task execution result, the objects to be captured in the environmental image corresponding to the instruction are determined, including: If the task execution result is yes, then a second preset task is obtained. The task type of the second preset task is: based on the first interaction content, the task of pointing out the first region information of the image region occupied by the object to be captured in the environment image. The second preset task, the first interactive content, and the environmental image are input into the target interactive model to obtain the first region information; The objects within the first region information are taken as the objects to be captured in the environment image corresponding to the instruction.

4. The method according to claim 1, characterized in that, The method further includes: Based on a preset detection model, it is detected whether the environmental image contains candidate objects of the same type as the object corresponding to the first interactive content. If so, it is determined to execute the task execution result corresponding to the first preset task by analyzing the first preset task, the first interactive content, and the environmental image through the target interaction model.

5. The method according to claim 1, characterized in that, The method further includes: The first interactive content is subjected to a legality check. If the first interactive content is determined to be legal, the first preset task, the first interactive content, and the environmental image are analyzed through the target interactive model to obtain the task execution result corresponding to the first preset task.

6. The method according to claim 3, characterized in that, The method further includes: The environmental image and the information of the first region are input into a preset segmentation model to obtain a mask corresponding to the environmental image; The depth image information corresponding to the mask and the environment image is input into a preset grasping model to obtain the target position.

7. The method according to claim 1, further comprising: The task of obtaining samples; Based on the type of the sample task, obtain the corresponding sample interaction information and the sample image corresponding to the sample interaction information; The sample interaction information includes: sample interaction content and sample task execution results; The initial word processing unit in the initial interaction model performs word processing on the sample task and the sample interaction content to obtain the corresponding word vector sequence. The sample image is encoded by the initial visual encoding unit in the initial interaction model to obtain the image feature sequence corresponding to the sample image; The word vector sequence and the image feature sequence are analyzed using the initial transformation model in the initial interaction model to obtain the corresponding task prediction results; The initial interaction model is trained based on the sample task execution results and the task prediction results to obtain the target interaction model.

8. A data processing apparatus, characterized in that, include: The acquisition unit is used to acquire user instructions and environmental images, wherein the user instructions are the first interaction content with the user; The determining unit is configured to determine, based on the instruction, the environmental image, and a preset target interaction model, the object to be crawled in the environmental image corresponding to the instruction, including: analyzing a first preset task, the first interaction content, and the environmental image through the target interaction model to obtain a task execution result corresponding to the first preset task, wherein the first preset task is a task based on the first interaction content to determine whether the object to be crawled can be specified in the environmental image; determining the object to be crawled in the environmental image corresponding to the instruction based on the task execution result; if the task execution result is negative, obtaining a third preset task, wherein the task of the third preset task... The type is: a task that asks a question based on the first interactive content; inputting the third preset task, the first interactive content, and the environmental image into the target interaction model to obtain the corresponding question content; asking the user a question based on the question content; obtaining the second interactive content in which the user answers based on the question content; using the second interactive content and the question content as new first interactive content, and returning to execute the analysis of the first preset task, the first interactive content, and the environmental image through the target interaction model to obtain the task execution result corresponding to the first preset task, until the object to be captured can be specified in the environmental image; The control unit is used to control the gripping device to grip the target item corresponding to the object to be gripped.

9. An electronic device, characterized in that, include: processor; as well as Memory for storing the executable instructions of the processor; The processor is configured to execute the method of any one of claims 1-7 by executing the executable instructions.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method described in any one of claims 1-7.

Citation Information

Patent Citations

  • Visual question and answer oriented method of context awareness based on multi-modal interaction

    CN114970517A

  • Robot target object grabbing method, system and equipment and storage medium

    CN116993965A