A method and device for object manipulation based on low-level natural language instructions

Abstract

The present invention discloses an object manipulation method and device based on low-level natural language instructions, belonging to the field of artificial intelligence and automatic control technology. The method comprises: fusing the visual features of the object to be manipulated, the state features of the manipulation device and the high-level natural language instruction features to obtain a fusion feature; matching the fusion feature with the low-level natural language instruction feature set to predict the low-level natural language instruction applicable to the current environment; based on the fusion feature and the prior features of the functional diagram of the object to be manipulated, selecting the point with the highest probability as the manipulation contact point; based on the fusion feature and the prior features of the action to be performed, obtaining the state of the manipulation device required to implement the action to be performed; and implementing the manipulation task by executing the predicted low-level natural language instruction. The present invention forms a comprehensive solution by integrating multiple innovative strategies such as data set expansion, model fine-tuning, multimodal fusion and prior indication, so as to better realize manipulation prediction and execution.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence and automatic control technology, and in particular to an object manipulation method and device based on low-level natural language instructions. Background Art

[0002] "Visual language manipulation" is playing an extremely important and diverse role in various fields. Especially in the field of smart home, this technology has successfully realized the intelligent control of home appliances and systems by cleverly coordinating and analyzing images and language, allowing users to easily remotely control and manage home appliances such as lights, electrical appliances, and security systems without any effort, just with simple language instructions and efficient image recognition. This task not only runs through the intersection of computer vision and natural language processing, but also aims to achieve intelligent manipulation of objects by combining images and language.

[0003] However, despite the rapid development of this field, some problems still exist: For example, in terms of data requirements, the currently available datasets are mainly high-level manipulation instructions, such as "rearrange the table". Although such high-level instructions facilitate human communication, they are prone to ambiguity, making it difficult for machines to successfully perform tasks directly. In addition, how to efficiently and accurately combine images with text information to achieve meaningful manipulation is a challenging topic. Although current multimodal fusion methods have made some progress, they usually remain at the level of direct fusion and lack the in-depth design and optimization required specifically for manipulation tasks.

[0004] Therefore, how to more intelligently integrate information from these different modalities for more precise manipulation still requires further research and exploration. Summary of the invention

[0005] In order to solve the problems existing in the prior art, the present invention provides the following technical solutions.

[0006] A first aspect of the present invention provides an object manipulation method based on low-level natural language instructions, comprising:

[0007] Acquire visual features of the object to be manipulated, state features of the manipulation device, high-level natural language instruction features, and a low-level natural language instruction feature set;

[0008] The visual features of the object to be manipulated, the state features of the manipulation device and the high-level natural language instruction features are fused to obtain a fused feature; and the fused feature is matched with the low-level natural language instruction feature set to predict the low-level natural language instruction applicable to the current environment;

[0009] Based on the fusion feature and the prior features of the function map of the object to be manipulated, the probability of each point on the function map of the object to be manipulated being successfully manipulated is obtained, and the point with the largest probability is selected as the manipulation contact point;

[0010] Based on the fused features and the prior features of the action to be performed, obtaining the state of the manipulation device required to implement the action to be performed;

[0011] A manipulation task is determined according to the manipulation contact point and the required state of the manipulation device, and the manipulation task is implemented by executing the predicted low-level natural language instruction.

[0012] Preferably, the step of acquiring visual features of the object to be manipulated, state features of the manipulation device, high-level natural language instruction features, and low-level natural language instruction feature sets includes:

[0013] Obtaining point cloud data of the object to be manipulated, the state of the manipulation device, high-level natural language instructions, and low-level natural language instruction sets;

[0014] Extracting visual features of the object to be manipulated according to point cloud data of the object to be manipulated;

[0015] Extracting state features of the operating device according to the state of the operating device;

[0016] Extracting high-level natural language instruction features according to the high-level natural language instructions;

[0017] A low-level natural language instruction feature set is extracted according to the low-level natural language instruction set.

[0018] Preferably, the low-level natural language instruction set is obtained by the following steps:

[0019] Collect and manipulate videos;

[0020] Generates a set of low-level natural language instructions corresponding to manipulating the video.

[0021] Preferably, the step of generating a low-level natural language instruction set corresponding to manipulating the video further includes the following steps:

[0022] By manipulating the video and its corresponding low-level natural language instruction set, the video language model is fine-tuned to obtain a fine-tuned video language model;

[0023] The newly collected manipulation video is input into the fine-tuned video language model to generate a low-level natural language instruction set corresponding to the newly collected manipulation video.

[0024] Preferably, matching the fusion feature with the low-level natural language instruction feature set to predict the low-level natural language instruction applicable to the current environment includes:

[0025] In the multi-layer perceptron, the concatenation of each low-level natural language instruction feature and the fusion feature is input, and the score of each low-level natural language instruction feature is output;

[0026] Normalizing the score of each low-level natural language instruction feature to obtain a corresponding normalized score;

[0027] The weighted sum of the score of each low-level natural language instruction feature and its corresponding normalized score is calculated, and the low-level natural language instruction applicable to the current environment is predicted based on the low-level natural language instruction feature corresponding to the largest weighted sum.

[0028] Preferably, the predicting of the low-level natural language instructions applicable to the current environment further includes:

[0029] The local features in the visual features, the state features of the operating device and the features corresponding to the low-level natural language instructions applicable to the current environment are fused to obtain new fused features;

[0030] In the multi-layer perceptron, the concatenation of each low-level natural language instruction feature and the new fusion feature is input, and the score of each low-level natural language instruction feature is output;

[0031] Normalizing the score of each low-level natural language instruction feature to obtain a corresponding normalized score;

[0032] The weighted sum of the score of each low-level natural language instruction feature and its corresponding normalized score is calculated, and the final low-level natural language instruction applicable to the current environment is predicted based on the low-level natural language instruction feature corresponding to the largest weighted sum.

[0033] Preferably, the prediction of low-level natural language instructions applicable to the current environment also includes the steps of: constructing a manipulation prompt noun a priori dictionary and a manipulation prompt verb a priori dictionary, wherein the manipulation prompt noun a priori dictionary records objects and their corresponding functional diagram information; the manipulation prompt verb a priori dictionary records actions and their corresponding execution status information.

[0034] The prior features of the functional map of the object to be manipulated are obtained by the following method: extracting a noun representing the object to be manipulated from the predicted low-level natural language instructions applicable to the current environment, and finding the prior features of the functional map corresponding to the noun using the manipulation prompt noun prior dictionary, which is the prior features of the functional map of the object to be manipulated;

[0035] The prior features of the action to be executed are obtained by the following method: a verb representing the action to be executed is extracted from the low-level natural language instructions applicable to the predicted current environment, and the prior features of the execution state corresponding to the verb are found using the manipulation prompt verb prior dictionary, which are the prior features of the action to be executed.

[0036] Preferably, obtaining the probability of successful manipulation of each point on the functional map of the object to be manipulated based on the fusion feature and the priori feature of the functional map of the object to be manipulated, and selecting the point with the largest probability as the manipulation contact point comprises:

[0037] After the fusion feature and the prior feature of the function map of the object to be manipulated are fused through a multi-layer perceptron, the probability of successful manipulation of each point on the function map of the object to be manipulated is generated, and the point with the highest probability of successful manipulation is selected as the manipulation contact point.

[0038] Preferably, obtaining the state of the manipulation device required to implement the action to be performed based on the fusion feature and the priori feature of the action to be performed includes:

[0039] After the fusion feature and the prior feature of the action to be performed are fused through a multi-layer perceptron, the state of the manipulation device required to realize the action to be performed is obtained by combining the probability of successful manipulation of each point on the generated function map of the object to be manipulated.

[0040] A second aspect of the present invention provides an object manipulation device based on low-level natural language instructions, comprising:

[0041] A feature acquisition module, used to acquire visual features of the object to be manipulated, state features of the manipulation device, high-level natural language instruction features, and a low-level natural language instruction feature set;

[0042] The low-level natural language instruction prediction module is used to fuse the visual features of the object to be manipulated, the state features of the manipulation device and the high-level natural language instruction features to obtain a fused feature; and match the fused feature with the low-level natural language instruction feature set to predict the low-level natural language instruction applicable to the current environment;

[0043] A manipulation contact point selection module, used to obtain the probability of each point on the function map of the object to be manipulated being successfully manipulated based on the fusion feature and the prior features of the function map of the object to be manipulated, and select the point with the largest probability as the manipulation contact point;

[0044] A state acquisition module of a manipulation device, used for obtaining a state of the manipulation device required to implement the action to be performed based on the fusion feature and the priori feature of the action to be performed;

[0045] The low-level natural language instruction execution module is used to determine the manipulation task according to the manipulation contact point and the required state of the manipulation device, and implement the manipulation task by executing the predicted low-level natural language instruction.

[0046] The present invention also provides a memory storing a plurality of instructions, wherein the instructions are used to implement the object manipulation method based on low-level natural language instructions as described in the first aspect.

[0047] The present invention also provides an electronic device, comprising a processor and a memory connected to the processor, wherein the memory stores a plurality of instructions, which can be loaded and executed by the processor so that the processor can execute the object manipulation method based on low-level natural language instructions as described in the first aspect.

[0048] The beneficial effects of the present invention are as follows: the technical solution provided by the present invention collects rich object manipulation video data in a simulation environment and manually annotates to generate corresponding low-level natural language instruction sets, and uses existing manipulation videos and low-level natural language instruction sets to fine-tune the large video language model to achieve rapid expansion of the low-level natural language instruction data set. A low-level natural language instruction following mechanism is introduced, so that the model can perform manipulation actions in an orderly manner according to the sequence of the low-level natural language instruction set. A manipulation prompt prior dictionary of objects and manipulation actions is constructed, and manipulation prior indications are introduced for the multimodal fusion process. The present invention integrates multiple innovative strategies such as data set expansion, model fine-tuning, multimodal fusion and prior indications to form a comprehensive solution. This comprehensive approach enables the model to obtain more comprehensive and powerful capabilities in visual language manipulation tasks, thereby better realizing manipulation prediction and execution. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1 It is a flowchart of the object manipulation method based on low-level natural language instructions according to the present invention;

[0050] Figure 2 It is a functional structure diagram of the object manipulation device based on low-level natural language instructions described in the present invention. DETAILED DESCRIPTION

[0051] In order to better understand the above technical solution, the above technical solution will be described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0052] The method provided by the present invention can be implemented in the following terminal environment, and the terminal may include one or more of the following components: a processor, a memory, and a display screen. The memory stores at least one instruction, and the instruction is loaded and executed by the processor to implement the method described in the following embodiment.

[0053] The processor may include one or more processing cores. The processor uses various interfaces and lines to connect various parts of the entire terminal, and executes various functions of the terminal and processes data by running or executing instructions, programs, code sets or instruction sets stored in the memory, and calling data stored in the memory.

[0054] The memory may include random access memory (RAM) or read-only memory (ROM). The memory may be used to store instructions, programs, codes, code sets or instructions.

[0055] The display screen is used to display the user interface of each application.

[0056] In addition, those skilled in the art can understand that the structure of the above terminal does not constitute a limitation on the terminal, and the terminal may include more or fewer components, or combine certain components, or arrange the components differently. For example, the terminal also includes components such as a radio frequency circuit, an input unit, a sensor, an audio circuit, and a power supply, which will not be described in detail here.

[0057] Embodiment 1

[0058] like Figure 1 As shown, an embodiment of the present invention provides an object manipulation method based on low-level natural language instructions, including:

[0059] S101, obtaining visual features of an object to be manipulated, state features of a manipulation device, high-level natural language instruction features, and a low-level natural language instruction feature set;

[0060] S102, fusing the visual features of the object to be manipulated, the state features of the manipulation device, and the high-level natural language instruction features to obtain a fused feature; and matching the fused feature with the low-level natural language instruction feature set to predict the low-level natural language instruction applicable to the current environment;

[0061] S103, based on the fusion feature and the prior features of the function map of the object to be manipulated, obtaining the probability of each point on the function map of the object to be manipulated being successfully manipulated, and selecting the point with the largest probability as the manipulation contact point;

[0062] S104, obtaining a state of a manipulation device required to implement the action to be performed based on the fusion feature and the priori feature of the action to be performed;

[0063] S105 , determining a manipulation task according to the manipulation contact point and the required state of the manipulation device, and implementing the manipulation task by executing the predicted low-level natural language instruction.

[0064] It should be noted that the above method provided by the present invention can be performed in a simulation environment. In practical applications, the manipulation device can be a robot, a mechanical arm or a mechanical claw.

[0065] Among them, high-level natural language instructions refer to instructions for the overall goal of the task, such as "open winebottle".

[0066] Low-level natural language instructions refer to a series of instructions for the task completion process, such as: instruction 1) approach and grab the cap of the wine bottle, instruction 2) twist the cap clockwise, and instruction 3) pull out the cap.

[0067] In step S101, firstly, the point cloud data of the object to be manipulated, the state of the manipulation device, the high-level natural language instructions and the low-level natural language instruction set are obtained; then the global features and local features of the object to be manipulated can be extracted using the general point cloud feature extractor PointNet++; the state of the manipulation device can be encoded using a multi-layer perceptron (MLP) network to obtain the state features of the manipulation device. In addition, the text feature extractor in the Clip model can be used to input the high-level natural language instructions and the low-level natural language instruction set to extract the corresponding high-level natural language instruction features and the low-level natural language instruction feature set.

[0068] Among them, the low-level natural language instruction set can be obtained by the following steps: collecting the manipulation video; generating the low-level natural language instruction set corresponding to the manipulation video. Specifically, for example, in the simulation environment V-REP, manipulation videos covering 15 types of tasks (high-level natural language instruction tasks) are collected (such as 'close the door', 'open the drawer', 'open the wine bottle' and other household scene tasks). These videos are then submitted to the annotation personnel, and the low-level natural language instruction set corresponding to the manipulation video is generated through manual annotation. For example, for the high-level natural language instruction task 'slide the top drawer open' (corresponding to the corresponding manipulation video), the corresponding low-level natural language instruction set is generated as follows: Step 1: face to the front of the top drawer panel; Step 2: close the gripper to grasp the handle; Step 3: keep the gripper closed and pull out the handle. For another example, for the high-level natural language instruction task of 'close the door' (corresponding to the corresponding manipulation video), the corresponding low-level natural language instruction set is generated as follows: Step 1: position the gripper in front of the door handle; Step 2: grasp the handle with the gripper; Step 3: push the door. In this process, 4,500 corresponding videos and low-level natural language instruction sets are generated.

[0069] In order to further expand the data set size of low-level natural language instructions, the present invention also includes the following steps: using the manipulation video and its corresponding low-level natural language instruction set to fine-tune the video language model to obtain a fine-tuned video language model; inputting the newly collected manipulation video into the fine-tuned video language model to generate a low-level natural language instruction set corresponding to the newly collected manipulation video.

[0070] Among them, the video language big model used may include minigpt4 and video-llama. The video language big model is fine-tuned so as to generate a corresponding series of low-level natural language instructions using the manipulated video. Specifically, the manipulated video with existing text instructions (high-level natural language instructions) can be used as the input of the video language big model, and the corresponding low-level natural language instruction set can be used as the supervision signal to fine-tune the video language big model to obtain the fine-tuned video language big model. Then, with the help of the fine-tuned video language big model, the corresponding low-level natural language instruction set can be generated for more simulation environment manipulated videos, thereby expanding the scale of the low-level natural language instruction data set.

[0071] In step S102, the visual features of the object to be manipulated may include global features and local features. First, the visual global features of the object to be manipulated, the state features of the manipulation device, and the high-level natural language instruction features may be fused to obtain fused features; and then matched with a series of low-level natural language instruction features. Thus, it is predicted which instruction in a series of low-level instruction sets should be executed in the current environment (including the state of the object to be manipulated and the state of the manipulation device), ensuring that the low-level natural language instructions can be followed to perform tasks. Specifically, in the module composed of the multi-layer perceptron, each low-level natural language instruction feature and the splicing of the visual global features, the state features of the manipulation device (such as the state features of the mechanical claw) and the high-level natural language instruction features can be input, and the score of each low-level natural language instruction feature can be output. These scores can be normalized by the softmax function to obtain a normalized score. According to the normalized score, it is determined which specific low-level natural language instruction should be executed in the current environment to guide the subsequent prediction of the specific manipulation action of the manipulation device. Then, the weighted sum of the score of each low-level natural language instruction feature and its corresponding normalized score is calculated, and the low-level natural language instruction applicable to the current environment is predicted based on the low-level natural language instruction feature corresponding to the largest weighted sum. It should be noted that many instructions are given to the model at one time, such as instruction 1) xxx, instruction 2) xxx, and instruction 3) xxxx. Then the model needs to first determine which instruction should be executed among instruction 1), instruction 2) and instruction 3) based on the state of the current environment (such as objects and robots). The selection method is to score each instruction. The higher the score, the more it matches the current environment information. Then the instruction selected for execution is the instruction with the highest score.

[0072] In a preferred embodiment of the present invention, after predicting the low-level natural language instructions applicable to the current environment, it also includes: fusing local features in the visual features, state features of the operating device, and features corresponding to the predicted low-level natural language instructions applicable to the current environment to obtain new fused features; in a multi-layer perceptron, inputting the concatenation of each low-level natural language instruction feature and the new fused features, and outputting the score of each low-level natural language instruction feature; normalizing the score of each low-level natural language instruction feature to obtain a corresponding normalized score; calculating the weighted sum of the score of each low-level natural language instruction feature and its corresponding normalized score, and predicting the final low-level natural language instruction applicable to the current environment based on the low-level natural language instruction feature corresponding to the largest weighted sum. That is, the local visual features, the operating device state features and the predicted low-level natural language instruction features are multimodally fused to obtain new fused features, absorb information from multiple modalities, and use the new fused features and each low-level natural language instruction feature as the input of the multi-layer perceptron to predict the low-level natural language instructions applicable to the current environment again, so as to promote the model to more accurately predict the low-level natural language instructions applicable to the current environment.

[0073] In a preferred embodiment of the present invention, the step of predicting the low-level natural language instructions applicable to the current environment may also include the following steps: constructing a priori dictionary of manipulation prompt nouns and a priori dictionary of manipulation prompt verbs, wherein the a priori dictionary of manipulation prompt nouns records objects and their corresponding functional diagram information; the a priori dictionary of manipulation prompt verbs records actions and their corresponding execution status information. In visual language manipulation tasks, it is crucial to understand the objects to be manipulated and the actions to be performed. To this end, relevant prior instructions are designed and extracted in this embodiment. By constructing a priori dictionary of manipulation prompt nouns, objects and their corresponding functional diagram information are recorded. For example, the a priori dictionary of manipulation prompt nouns records the object "manipulation door". No matter what the style of the "manipulation door" is, whether it is opening or closing the door, the a priori features of the "manipulation door" are consistent, that is, it is difficult to manipulate when it is close to the door axis, and it is easy to manipulate when it is far away from the door axis. Then, a priori dictionary of manipulation prompt verbs is constructed to record actions and their corresponding execution status information. For example, for the action of "pulling", whether it is pulling a door or a drawer, the operating direction of the operating device should be toward the outside, that is, the verb "pull" and the execution direction "towards the outside" are a paired set of prior information. Another example is the verb "push", the direction of the corresponding execution state is toward the inside. Then the verb "push" and the execution direction "towards the inside" are a paired set of prior information. These prior cues will play an important role in subsequent manipulation predictions because they can unify tasks with similar characteristics, allowing the model to ensure more accurate predictions.

[0074] In step S103, the prior features of the functional map of the object to be manipulated can be obtained by the following method: extracting the noun representing the object to be manipulated from the predicted low-level natural language instructions applicable to the current environment, and finding the prior features of the functional map corresponding to the noun using the manipulation prompt noun prior dictionary, that is, the prior features of the functional map of the object to be manipulated. Wherein, the probability of each point on the functional map of the object to be manipulated being successfully manipulated based on the fusion feature and the prior features of the functional map of the object to be manipulated, and selecting the point with the highest probability as the manipulation contact point includes: fusing the fusion feature and the prior features of the functional map of the object to be manipulated through a multi-layer perceptron, generating the probability of each point on the functional map of the object to be manipulated being successfully manipulated, and selecting the point with the highest probability of being successfully manipulated as the manipulation contact point. The probability range of each point on the functional map (Affordance map, which can also be understood as the manipulable map) being successfully manipulated ranges from 0 to 1, and the larger the value, the greater the probability of being successfully manipulated. For example, on the functional map of a door, the point farthest from the door axis has a value of 1, and if it is manipulated, it is most likely to be successful. If the value of the point on the door axis is 0, no matter how you push it, you cannot move the door, that is, it cannot be operated.

[0075] For example, for a low-level natural language instruction 'keep the gripper closed and pull out the handle', the noun 'handle door handle' is extracted from it, and the prior features of the functional map corresponding to 'door handle' are found from the prior dictionary of manipulation prompt nouns. After the multimodal fusion features and the prior features of the functional map are fused through MLP, a fine-grained functional map of the current object is generated to express the probability of successful manipulation of each point of the current object. The point with the highest probability is selected as the contact point between the manipulation device and the object.

[0076] In step S104, the prior features of the action to be performed are obtained by the following method: extracting the verb representing the action to be performed from the predicted low-level natural language instructions applicable to the current environment, and finding the prior features of the execution state corresponding to the verb using the prior dictionary of the manipulation prompt verb, which is the prior features of the action to be performed. Wherein, obtaining the state of the manipulation device required to implement the action to be performed based on the fused features and the prior features of the action to be performed includes: fusing the fused features and the prior features of the action to be performed through a multi-layer perceptron, and combining them with the generated fine-grained functional map of the object to be manipulated, to obtain the state of the manipulation device required to implement the action to be performed.

[0077] For example, for the above-mentioned low-level natural language instruction 'keep the gripper closed and pull out the handle', the verb 'pull' is extracted, and the prior information of the execution state corresponding to 'pull' is found in the manipulation prompt verb prior dictionary. The generated pixel-level functional map, multimodal fusion features and prior information of the execution state are combined through MLP fusion to generate the state of the manipulation device required to implement the action to be executed, such as the angle of the mechanical claw, the switch state and other information.

[0078] Step S105 is executed to determine a manipulation task according to the manipulation contact point and the required state of the manipulation device, and to implement the manipulation task by executing the predicted low-level natural language instruction.

[0079] Specifically, the predicted manipulation contact points and manipulation action postures can be input into the robot in the simulation environment, so that it can perform manipulation tasks one by one according to the low-level natural language instruction set.

[0080] The object manipulation method based on low-level natural language instructions provided by the present invention has the following beneficial effects:

[0081] 1. Dataset expansion strategy: By collecting rich object manipulation video data in a simulation environment and manually annotating the corresponding low-level natural language instruction set, as well as using existing manipulation videos and low-level natural language instruction sets, the low-level natural language instruction dataset is rapidly expanded by fine-tuning the video language large model. This strategy effectively solves the problem of insufficient existing low-level natural language instruction data and provides more and more diverse data samples for model training.

[0082] 2. Fine-tuning the video language model: By fine-tuning the video language model, the model's pre-trained knowledge is combined with data from specific manipulation tasks to further optimize the model's performance. This method can further efficiently expand the size of the dataset.

[0083] 3. Multimodal fusion mechanism: A low-level natural language instruction following mechanism is introduced, which enables the model to perform manipulation actions in an orderly manner according to the sequence of low-level natural language instruction sets. This mechanism plays a key role in the fusion of multimodal information, enabling the model to better coordinate visual and textual information, thereby more accurately predicting and executing manipulation actions.

[0084] 4. Manipulation cue prior dictionary: A manipulation cue prior dictionary of objects and manipulation actions is constructed, which introduces manipulation prior indications into the multimodal fusion process. These indications not only provide task-specific information, but also enhance the model's ability to extract useful features from multimodal data, further improving the accuracy of manipulation prediction.

[0085] 5. Comprehensive solution: It integrates multiple innovative strategies such as dataset expansion, model fine-tuning, multimodal fusion and prior indication to form a comprehensive solution. This comprehensive approach enables the model to obtain more comprehensive and powerful capabilities in visual language manipulation tasks, thereby better achieving manipulation prediction and execution.

[0086] Embodiment 2

[0087] like Figure 2 As shown, another aspect of the present invention also includes a functional module architecture that is completely consistent with the aforementioned method flow, that is, the embodiment of the present invention also provides an object manipulation device based on low-level natural language instructions, including:

[0088] A feature acquisition module 201 is used to acquire visual features of the object to be manipulated, state features of the manipulation device, high-level natural language instruction features, and a low-level natural language instruction feature set;

[0089] The low-level natural language instruction prediction module 202 is used to fuse the visual features of the object to be manipulated, the state features of the manipulation device and the high-level natural language instruction features to obtain a fused feature; and match the fused feature with the low-level natural language instruction feature set to predict the low-level natural language instruction applicable to the current environment;

[0090] A manipulation contact point selection module 203 is used to obtain the probability of each point on the function map of the object to be manipulated being successfully manipulated based on the fusion feature and the prior features of the function map of the object to be manipulated, and select the point with the largest probability as the manipulation contact point;

[0091] A state acquisition module 204 of a manipulation device, used to obtain a state of the manipulation device required to implement the action to be performed based on the fusion feature and the prior features of the action to be performed;

[0092] The low-level natural language instruction execution module 205 is used to determine a manipulation task according to the manipulation contact point and the required state of the manipulation device, and implement the manipulation task by executing the predicted low-level natural language instruction.

[0093] The device can be implemented through the object manipulation method based on low-level natural language instructions provided in the above-mentioned embodiment 1. The specific implementation method can be found in the description of embodiment 1 and will not be repeated here.

[0094] The present invention also provides a memory storing a plurality of instructions, wherein the instructions are used to implement the object manipulation method based on low-level natural language instructions as described in the first embodiment.

[0095] The present invention also provides an electronic device, comprising a processor and a memory connected to the processor, wherein the memory stores a plurality of instructions, and the instructions can be loaded and executed by the processor so that the processor can execute the object manipulation method based on low-level natural language instructions as described in Example 1.

[0096] Although preferred embodiments of the present invention have been described, additional changes and modifications may be made to these embodiments by those skilled in the art once the basic inventive concepts are known. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present invention. Obviously, those skilled in the art may make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.

Claims

1. A method for object manipulation based on low-level natural language instructions, characterized in that: include: Acquire visual features of the object to be manipulated, state features of the manipulation device, high-level natural language instruction features, and a low-level natural language instruction feature set; The visual features of the object to be manipulated, the state features of the manipulation device and the high-level natural language command features are fused to obtain fused features; The fused features are matched with the low-level natural language instruction feature set to predict the low-level natural language instructions applicable to the current environment; After fusing the fusion feature and the prior features of the functional map of the object to be manipulated through a multi-layer perceptron, the probability of successful manipulation of each point on the functional map of the object to be manipulated is generated, and the point with the highest probability of successful manipulation is selected as the manipulation contact point; After fusing the fusion features and the prior features of the action to be performed through a multi-layer perceptron, the state of the manipulation device required to implement the action to be performed is obtained by combining the probability of successful manipulation of each point on the generated function map of the object to be manipulated; Determine a manipulation task based on the manipulation contact point and the required state of the manipulation device, and implement the manipulation task by executing the predicted low-level natural language instruction; The prior features of the functional map of the object to be manipulated are obtained by the following method: extracting a noun representing the object to be manipulated from the predicted low-level natural language instructions applicable to the current environment, and finding the prior features of the functional map corresponding to the noun using the manipulation prompt noun prior dictionary, which is the prior features of the functional map of the object to be manipulated; The prior features of the action to be executed are obtained by the following method: a verb representing the action to be executed is extracted from the low-level natural language instructions applicable to the predicted current environment, and the prior features of the execution state corresponding to the verb are found using the manipulation prompt verb prior dictionary, which are the prior features of the action to be executed.

2. The object manipulation method based on low-level natural language instructions as claimed in claim 1, characterized in that: The acquisition of visual features of the object to be manipulated, state features of the manipulation device, high-level natural language instruction features and low-level natural language instruction feature sets includes: Obtaining point cloud data of the object to be manipulated, the state of the manipulation device, high-level natural language instructions, and low-level natural language instruction sets; Extracting visual features of the object to be manipulated according to point cloud data of the object to be manipulated; Extracting state features of the operating device according to the state of the operating device; Extracting high-level natural language instruction features according to the high-level natural language instructions; A low-level natural language instruction feature set is extracted according to the low-level natural language instruction set.

3. The object manipulation method based on low-level natural language instructions as claimed in claim 2, characterized in that: The low-level natural language instruction set is obtained by the following steps: Collect and manipulate videos; Generates a set of low-level natural language instructions corresponding to manipulating the video.

4. The object manipulation method based on low-level natural language instructions as claimed in claim 3, characterized in that: The step of generating a low-level natural language instruction set corresponding to manipulating the video further includes: By manipulating the video and its corresponding low-level natural language instruction set, the video language model is fine-tuned to obtain a fine-tuned video language model; The newly collected manipulation video is input into the fine-tuned video language model to generate a low-level natural language instruction set corresponding to the newly collected manipulation video.

5. The object manipulation method based on low-level natural language instructions as claimed in claim 1, characterized in that: The matching of the fusion feature with the low-level natural language instruction feature set to predict the low-level natural language instruction applicable to the current environment includes: In the multi-layer perceptron, the concatenation of each low-level natural language instruction feature and the fusion feature is input, and the score of each low-level natural language instruction feature is output; Normalizing the score of each low-level natural language instruction feature to obtain a corresponding normalized score; The weighted sum of the score of each low-level natural language instruction feature and its corresponding normalized score is calculated, and the low-level natural language instruction applicable to the current environment is predicted based on the low-level natural language instruction feature corresponding to the largest weighted sum.

6. The object manipulation method based on low-level natural language instructions as claimed in claim 5, characterized in that: The predicting of the low-level natural language instructions applicable to the current environment also includes: The local features in the visual features, the state features of the operating device and the features corresponding to the low-level natural language instructions applicable to the current environment are fused to obtain new fused features; In the multi-layer perceptron, the concatenation of each low-level natural language instruction feature and the new fusion feature is input, and the score of each low-level natural language instruction feature is output; Normalizing the score of each low-level natural language instruction feature to obtain a corresponding normalized score; The weighted sum of the score of each low-level natural language instruction feature and its corresponding normalized score is calculated, and the final low-level natural language instruction applicable to the current environment is predicted based on the low-level natural language instruction feature corresponding to the largest weighted sum.

7. The object manipulation method based on low-level natural language instructions as claimed in claim 1, characterized in that: The step of predicting the low-level natural language instructions applicable to the current environment also includes: constructing a priori dictionary of manipulation prompt nouns and a priori dictionary of manipulation prompt verbs, wherein the a priori dictionary of manipulation prompt nouns records objects and their corresponding functional diagram information; the a priori dictionary of manipulation prompt verbs records actions and their corresponding execution status information.

8. An object manipulation device based on low-level natural language instructions, characterized in that: include: A feature acquisition module, used to acquire visual features of the object to be manipulated, state features of the manipulation device, high-level natural language instruction features, and a low-level natural language instruction feature set; A low-level natural language instruction prediction module is used to fuse the visual features of the object to be manipulated, the state features of the manipulation device, and the high-level natural language instruction features to obtain a fused feature; The fused features are matched with the low-level natural language instruction feature set to predict the low-level natural language instructions applicable to the current environment; A manipulation contact point selection module is used to fuse the fusion feature and the prior features of the functional map of the object to be manipulated through a multi-layer perceptron, generate the probability of each point on the functional map of the object to be manipulated being successfully manipulated, and select the point with the highest probability of successful manipulation as the manipulation contact point; A state acquisition module for a manipulation device, used to fuse the fusion features and the prior features of the action to be performed through a multi-layer perceptron, and combine the probability of successful manipulation of each point on the generated function map of the object to be manipulated to obtain the state of the manipulation device required to implement the action to be performed; A low-level natural language instruction execution module, configured to determine a manipulation task according to a manipulation contact point and a required state of a manipulation device, and implement the manipulation task by executing a predicted low-level natural language instruction; The prior features of the functional graph of the object to be manipulated are: extracting a noun representing the object to be manipulated from the low-level natural language instructions predicted to be applicable to the current environment, and finding the prior features of the functional graph corresponding to the noun using the manipulation prompt noun prior dictionary, which is the prior features of the functional graph of the object to be manipulated; The prior features of the action to be executed are to extract the verb representing the action to be executed from the low-level natural language instructions predicted to be applicable to the current environment, and use the manipulation prompt verb prior dictionary to find the prior features of the execution state corresponding to the verb, which are the prior features of the action to be executed.

9. A memory, characterized in that: A plurality of instructions are stored, and the instructions are used to implement the object manipulation method based on low-level natural language instructions as described in any one of claims 1-7.

10. An electronic device, characterized in that: It includes a processor and a memory connected to the processor, the memory stores multiple instructions, and the instructions can be loaded and executed by the processor to enable the processor to execute the object manipulation method based on low-level natural language instructions as described in any one of claims 1-7.

Images