Instruction generation method, robot control method, electronic equipment and medium
By determining the action and target objects in the processing model and generating target instructions, the problem of low instruction generation efficiency in the prior art is solved, and the robot can be faster response and real-time control.
Patent Information
- Application Number
- CN202510186193.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-02-19
- Publication Date
- 2025-05-02
Smart Images
Figure CN119910652A_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to the field of data processing, and in particular to an instruction generation method, a robot control method, an electronic device and a medium. Background Art
[0002] At present, the deep learning model processes the user's language to obtain instructions that the robot can understand, so as to control the robot to complete the instructions. For example, if the user asks the deep learning model "I want you to put the fruit in the basket, what will you do?", the deep learning model will process the question and obtain the instruction "Put the fruit in the basket", which can be further parsed and executed by the program to control the robot to complete the instruction.
[0003] However, directly processing the user's language through a deep learning model to obtain instructions that the robot can understand has the problem of low efficiency in generating instructions. Summary of the invention
[0004] Various aspects of the present disclosure provide an instruction generation method, a robot control method, an electronic device, and a medium to solve the technical problem of low efficiency in generating instructions in the prior art.
[0005] A first aspect of an embodiment of the present disclosure provides an instruction generation method, including:
[0006] Obtaining the first language instruction and determining the action and target object indicated by the first language instruction through the processing model;
[0007] Acquire a target image, the target image comprising: at least one first object;
[0008] determining, by processing the model, whether each of the at least one first object matches the target object;
[0009] If so, determining that the first object matching the target object is the object to be processed;
[0010] Target instructions are generated according to the actions and the objects to be processed, and the target instructions are used to control the robot.
[0011] A second aspect of an embodiment of the present disclosure provides an instruction generation device, including:
[0012] A first determination module is used to obtain a first language instruction and determine an action and a target object indicated by the first language instruction through a processing model;
[0013] An acquisition module, used to acquire a target image, the target image comprising: at least one first object;
[0014] A second determination module, configured to determine whether each first object of at least one first object matches the target object by processing the model;
[0015] A third determination module, configured to determine, if yes, that the first object matching the target object is the object to be processed;
[0016] The generation module is used to generate target instructions according to the actions and the objects to be processed, and the target instructions are used to control the robot.
[0017] A third aspect of the present disclosure provides a robot control device, including:
[0018] An acquisition module, used for acquiring a target instruction, the target instruction includes: an action and an object to be processed, and the target instruction is generated by the instruction generation method of the first aspect;
[0019] The control module is used to control the robot to perform actions on the object to be processed according to the target instructions.
[0020] A fourth aspect of an embodiment of the present disclosure provides an electronic device, comprising: a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the processor implements the method of the first aspect or the second aspect when executing the computer program.
[0021] A fifth aspect of an embodiment of the present disclosure provides a computer-readable storage medium, in which computer-executable instructions are stored. When the computer-executable instructions are executed by a processor, they are used to implement the method of the first aspect or the second aspect.
[0022] A sixth aspect of an embodiment of the present disclosure provides a computer program product, the program product comprising: a computer program, the computer program is stored in a readable storage medium, at least one processor of an electronic device can read the computer program from the readable storage medium, and at least one processor executes the computer program so that the electronic device executes the first aspect or the second aspect method.
[0023] The disclosed embodiment is applied in a robot control scenario, by acquiring a first language instruction, and determining the action and target object indicated by the first language instruction through a processing model; acquiring a target image, the target image including: at least one first object; determining through a processing model whether each first object in the at least one first object matches the target object; if so, determining that the first object matching the target object is an object to be processed; generating a target instruction according to the action and the object to be processed, the target instruction is used to control the robot, and the processing model only needs to output a small amount of text to generate the target instruction to control the robot, thereby improving the generation efficiency of the target instruction. BRIEF DESCRIPTION OF THE DRAWINGS
[0024] The drawings described herein are used to provide a further understanding of the present disclosure and constitute a part of the present disclosure. The illustrative embodiments of the present disclosure and their descriptions are used to explain the present disclosure and do not constitute an improper limitation on the present disclosure. In the drawings:
[0025] Figure 1 An application scenario diagram of an instruction generation method provided for an exemplary embodiment of the present disclosure;
[0026] Figure 2 A flowchart of a method for generating instructions provided for an exemplary embodiment of the present disclosure;
[0027] Figure 3 A flowchart of another instruction generation method provided for an exemplary embodiment of the present disclosure;
[0028] Figure 4 A flowchart of a robot control method provided for an exemplary embodiment of the present disclosure;
[0029] Figure 5 A structural block diagram of an instruction generation device provided for an exemplary embodiment of the present disclosure;
[0030] Figure 6 A structural block diagram of a robot control device provided for an exemplary embodiment of the present disclosure;
[0031] Figure 7 A schematic structural diagram of an electronic device provided by an exemplary embodiment of the present disclosure. DETAILED DESCRIPTION
[0032] In order to make the purpose, technical solutions and advantages of the present disclosure clearer, the technical solutions of the present disclosure will be clearly and completely described below in combination with the specific embodiments of the present disclosure and the corresponding drawings. Obviously, the described embodiments are only part of the embodiments of the present disclosure, not all of the embodiments. Based on the embodiments in the present disclosure, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present disclosure.
[0033] In the related art, a deep learning model trained with a large amount of text data can generate natural language text or understand the meaning of language text. This deep learning module can be used for robots that interact with natural language. Among them, the deep learning model requires a large amount of data for fine-tuning to solve the problem of insufficient capabilities of general deep learning models, which requires a large amount of data resources, hardware resources and time resources. In addition, the control of the robot requires real-time response. For example, after the user gives a language instruction, the robot needs to execute it immediately. However, the efficiency of the existing deep learning model to generate instructions is strongly related to the number of output words. The more output words, the lower the output efficiency. Therefore, there is a problem of low efficiency in directly outputting instructions to control the robot through a deep learning model.
[0034] Based on the above problems, the present disclosure provides an instruction generation method, in which the processing model only needs to output the action and the target object and output "yes" or "no", and a small amount of text can generate the target instruction, thereby improving the generation efficiency of the target instruction.
[0035] In addition, an application scenario of the embodiment of the present disclosure is as follows Figure 1 , is a robot system 10, which includes: an instruction generating device 11, a robot 12 and a workbench 13. A plurality of first objects, such as a first object a1, a first object a2, a first object a3 and a first object a4, can be placed on the workbench 13. A user inputs a first language instruction, and the first language instruction is processed by the instruction generating device of the present disclosure to obtain a target instruction, which can control the robot to perform a corresponding action on the first object.
[0036] in, Figure 1 This is just an exemplary application scenario, and the embodiment of the present disclosure can be applied to the instruction generation of any robot. The embodiment of the present disclosure does not limit the specific application scenario.
[0037] Figure 2 A flowchart of a method for generating an instruction provided by an exemplary embodiment of the present disclosure specifically includes the following steps:
[0038] S201, obtaining a first language instruction, and determining an action and a target object indicated by the first language instruction through a processing model.
[0039] The first language instruction is in voice format or text format. If the first language instruction is in voice format, it may be the first language instruction collected by a microphone. If the first language instruction is input by a user.
[0040] The first language instruction is a natural language instruction, such as "Please put the fruit in the box", "I asked you to put the fruit in the box", "I want you to put the fruit in the basket, what will you do?".
[0041] In the present disclosure, the processing model is a trained neural network model, which can be a large language model (LLM). Further, in the present disclosure, the large language model can select a small large language model, for example, the storage space occupied by the large language model is less than 20B. In addition, the processing model of the present disclosure does not need to be fine-tuned again and can be directly applied in the present disclosure, thereby avoiding the consumption of resources.
[0042] In the present disclosure, an action includes at least one of the following: grasping, placing, moving or positioning. It can be understood that grasping means that the robot needs to perform a grasping action. Placing means that the robot needs to perform a placing action, and the placing action also includes a placing method, such as stacking. Moving means that the robot needs to perform a moving action. Positioning means that the robot needs to perform a positioning action, that is, the robot is positioned at the target position. In addition, the action may also include other actions that can be performed by the robot, which are not limited.
[0043] In addition, the target object is the object that the first language instruction instructs the robot to perform an action on. The target object is the name of various objects, such as "fruit", "banana", "apple", "round object", "square object", etc.
[0044] In the disclosure, a first language instruction indicates at least one action and at least one target object.
[0045] For example: If the first language instruction is "I want you to put the fruit in the basket, what will you do?", the corresponding action is "grab and place", and the target object is "fruit". If the first language instruction is "move all round objects to platform A", the corresponding actions are "grab", "move and place", and the target object is "round object". If the robot is in the process of grabbing a round object and moving it to platform A, and the first language instruction received is "move the round object to platform B", the corresponding action is "move and place". If the first language instruction is "locate the banana", the corresponding action is "move and locate", and the target object is "banana".
[0046] In the present disclosure, the processing model only needs to output simple words and the output text is less, so the processing effect is higher.
[0047] S202, acquiring a target image.
[0048] The target image includes: at least one first object.
[0049] In the present disclosure, the target image is acquired for the first object on the workbench. It can be understood that Figure 1The robot system may further include a camera, which is controlled to capture an image of the first object on the workbench to obtain a target object.
[0050] Furthermore, after acquiring the target object, the present disclosure may use image recognition technology to identify at least one first object included in the target image. Figure 1 For example, the first object a1 is recognized as a peach, the first object a2 is recognized as an apple, the first object a3 is recognized as a cucumber, and the first object a4 is recognized as a tomato.
[0051] S203: Determine whether each first object in the at least one first object matches the target object by processing the model.
[0052] The target object and the first object match each other in that the first object is the same as the target object, or the first object belongs to the category indicated by the target object.
[0053] For example, refer to Figure 1 , the first object a1 is peach, the first object a2 is apple, the first object a3 is cucumber and the first object a4 is tomato. If the target object is peach, the first object a1 is the same as the target object, that is, it matches, and the first objects a2 to a4 do not match the target object. If the target object is fruit, where peach and apple are classified as fruit, the first object a1 and the first object a2 match the target object, and the first object a3 and the first object a4 do not match the target object.
[0054] In the present disclosure, each first object and target object is sequentially input into the processing model for processing, and the processing model only needs to output "yes" or "no", wherein "yes" is output if the first object matches the target object, and "no" is output if they do not match.
[0055] Furthermore, a prompt word is input into the processing model, and the prompt word, the first object, and the target object are processed in combination with the input processing model so that the processing model determines whether the object and the target object match. For example, the content of the input processing model is "Please determine whether peaches and fruits match, and only need to answer yes or no."
[0056] It can be understood that the processing model in the present disclosure only needs to output one word in this step, so the processing efficiency is high and the processing accuracy is high.
[0057] S204: If yes, determine that the first object matching the target object is the object to be processed.
[0058] Among them, the first objects that do not match the target object are not used to generate the target instruction, that is, the robot does not process these first objects.
[0059] For example: if the target object is fruit, where peaches and apples are classified as fruits, the first object a1 and the first object a2 match the target object, and the first object a1 and the first object a2 are objects to be processed, and the first object a3 and the first object a4 do not match the target object, and the first object a3 and the first object a4 are not used to generate target instructions, and the robot does not process the first object a3 and the first object a4.
[0060] S205: Generate a target instruction according to the action and the object to be processed.
[0061] Among them, the target instruction is used to control the robot.
[0062] In the embodiment of the present disclosure, the target instruction is an instruction that can be read by the robot, and the format of the target instruction is preset.
[0063] For example, if the action is “grab and place” and the objects to be processed are “apple” and “peach”, then the target instruction is “grab and place (apple, peach)”.
[0064] Furthermore, the target position can be pre-configured, and the robot will place the grasped object to be processed at the preset target position when executing the target instruction. For example, if the target instruction is "grab and place (apple, peach)" and the target position is "box", the robot will grab apples and peaches and place them in the box.
[0065] In one embodiment, the robot also has the ability to process data. For example, the action may also include counting or determining relative positions. For the instructions obtained for counting or determining relative positions, for example, the first language instruction is "counting fruits." The processing model determines that the action indicated by the first language instruction is counting, and the target object is fruit. Then, multiple first objects in the target image are obtained, and the first objects are determined to be objects to be processed, such as (apple, peach). Then, the corresponding target instruction generated is "count (apple, peach)", and then the target instruction can be input into the robot for execution, and the robot outputs a result of "2". For another example: the action is to determine the relative position, and the first language instruction is "determine the relative position of apples and peaches", then the corresponding target instruction is "determine relative position (apple, peach)", then the robot can be controlled to execute the target instruction, and the robot outputs a result such as "the apple is on the left of the peach".
[0066] In summary, the present disclosure can generate target instructions with a small amount of text by processing the model only by outputting actions and target objects and outputting "yes" or "no", thereby improving the generation efficiency of target instructions.
[0067] Reference Figure 3 , is a flowchart of a method for generating an instruction provided by an exemplary embodiment of the present disclosure, which specifically includes the following steps:
[0068] S301, obtaining a first language instruction.
[0069] The specific implementation process of this step refers to S201 and will not be repeated here.
[0070] S302: Determine whether the first language instruction is reasonable by processing the model.
[0071] Among them, reasonable first language instructions are executable by the robot.
[0072] In the disclosed embodiment, the processing model only needs to output “yes” or “no” according to the rationality of the first language instruction.
[0073] In the present disclosure, the prompt word and the first language instruction can be combined and then input into the processing model to determine the rationality. For example, the first language instruction is "I want you to put the fruit into the basket, what will you do?", and the input into the processing model is "Please confirm the rationality of I want you to put the fruit into the basket, what will you do?", and only need to answer yes or no.
[0074] In one embodiment, determining whether the first language instruction is reasonable through the processing model includes: determining whether the action indicated in the first language instruction is a preset action through the processing model; if so, determining that the first language instruction is reasonable.
[0075] In the present disclosure, a preset action is an action that a robot can perform. For example, a robot can only perform the two actions of "grab" and "place".
[0076] Among them, the preset action can be used as a prompt word and processed with the first language instruction input processing model to determine whether the first language instruction is reasonable. For example, the input processing model is "Please confirm whether 'I want you to put the fruit in the basket, what will you do?' indicates 'grab' and / or 'place', just need to reply yes or no". Further, if the processing model outputs "yes", it is determined that the first language instruction is reasonable. If the processing model outputs "no", it is determined that the first language instruction is unreasonable.
[0077] In another embodiment, determining whether the first language instruction is reasonable through the processing model includes: determining whether the first language instruction is common sense through the processing model; if so, determining that the first language instruction is reasonable.
[0078] Examples that are consistent with common sense and examples that are inconsistent with common sense can be used as prompt words and processed with the first language instruction input processing model to determine whether the first language instruction is reasonable. For example, the input processing model is "Please confirm whether 'I want you to put the fruit in the basket, what will you do?' is consistent with common sense, and you only need to reply yes or no, among which, similar to 'putting bananas on the moon' is inconsistent with common sense, and 'putting bananas in the box' is consistent with common sense." Further, if the processing model outputs "yes", it is determined that the first language instruction is reasonable. If the processing model outputs "no", it is determined that the first language instruction is unreasonable.
[0079] In the present disclosure, the processing model only needs to output one word "yes" or "no" for the rationality of the first language instruction, so the processing efficiency of the processing model can also be improved. In addition, the processing model can also determine whether the first language instruction is reasonable in other ways, which are not limited here.
[0080] Further, if the first language instruction is reasonable, S303 is executed; if the first language instruction is not reasonable, prompt information may be output to prompt the user that the first language instruction is unreasonable.
[0081] S303: Acquire historical language instructions of the first language instructions.
[0082] Among them, before S303, it also includes determining whether the first language instruction needs to be understood in conjunction with the historical language instruction. If so, obtaining the historical language instruction of the first language instruction and executing S304 and S305. If not, executing the step of processing the first language instruction through the processing model to obtain the action and target object indicated by the first language instruction.
[0083] Specifically, the first language instruction may be input into the processing model to determine whether the first language instruction needs to be understood in conjunction with the historical language instruction, and the processing model only needs to output "yes" or "no".
[0084] For example, if the first language instruction is "Let's move these to the basket", the content of the input processing model is "Please confirm whether 'Let's move these to the basket' needs to be understood in conjunction with the historical language instruction, and only need to reply yes or no." If the processing model outputs "yes", the historical language instruction of the first language instruction is obtained, such as "move all the fruits into the basin".
[0085] In another embodiment, for any first language instruction, the historical language instructions of the first language instruction are obtained, and then S304 and S305 are executed.
[0086] In the present disclosure, the historical language instruction of the first language instruction refers to at least one language instruction before the first language instruction. For example, if the first language instruction is "Let's move these to the basket", the previous language instruction of the first language instruction is "Move all the fruits to the basin", and the first two language instructions of the first language instruction are "Grab the fruits", then the historical language instructions include "Grab the fruits" and "Move all the fruits to the basin".
[0087] S304, concatenating the first language instruction and the historical language instruction to obtain a second language instruction.
[0088] In the present disclosure, the historical language instruction is spliced before the first language instruction to obtain the second language instruction. For example, the first language instruction is "Let's move these to the basket", and the historical language instruction is "Move all the fruits to the basin", and the corresponding second language instruction is "Move all the fruits to the basin, or move these to the basket".
[0089] S305: Process the second language instruction by using the processing model to obtain the action and target object indicated by the first language instruction.
[0090] The processing of the second language instruction by the processing model may refer to the processing of the first language instruction by the processing model in S201, which will not be described in detail here.
[0091] S306, controlling the camera to collect images of the plurality of first objects placed on the target table to obtain a target image.
[0092] In the present disclosure, the target image is acquired for the first object on the workbench. It can be understood that Figure 1 The robot system may further include a camera, which is controlled to capture an image of the first object on the workbench to obtain a target object.
[0093] Furthermore, after acquiring the target object, the present disclosure may use image recognition technology to identify at least one first object included in the target image. Figure 1 For example, the first object a1 is recognized as a peach, the first object a2 is recognized as an apple, the first object a3 is recognized as a cucumber, and the first object a4 is recognized as a tomato.
[0094] S307: Determine whether each first object in the at least one first object matches the target object by processing the model.
[0095] The specific implementation process of this step refers to S203 and will not be repeated here.
[0096] S308: Determine the first object that matches the target object as the object to be processed.
[0097] The specific implementation process of this step refers to S204 and will not be repeated here.
[0098] S309, determining the target position indicated by the first language instruction by processing the model.
[0099] In one embodiment, the first language instruction further indicates a target location, that is, a location corresponding to the action to be performed on the object to be processed. For example, if the first language instruction is "put the fruit in the basket", the target location is represented by "basket".
[0100] In the present disclosure, the target position is included in the first language instruction, and may also be included in the prompt word given to the processing model. For example, if the first language instruction is "Put the fruit in the basket", where the target position is "basket", the first language instruction includes "basket". If the first language instruction is "Pick up all the fruits", the input to the processing model is "Please confirm the target position indicated by 'Pick up all the fruits', where the target position for fruits is the basket and the target position for vegetables is the basin", and the target position can be identified as "basket".
[0101] In summary, the processing model can use a variety of methods to determine the target position indicated by the first language instruction, and the processing model only needs to output the corresponding "target position", thereby improving the processing efficiency of the processing model.
[0102] S310, determining a target instruction according to the action, the object to be processed and the target position.
[0103] In the embodiment of the present disclosure, the target instruction is an instruction that can be read by the robot, and the format of the target instruction is preset.
[0104] For example, if the action is “grab and place”, the objects to be processed are “apple” and “peach”, and the target location is a basket, then the target instruction is “grab and place ([apple, peach], basket)”.
[0105] In summary, the present disclosure can use a small processing model to allow the processing model to perform tasks that only require the output of a small number of words, combine multiple small tasks, complete the task of determining the target instruction, and improve the efficiency of generating the target instruction. In addition, accurate target instructions can be obtained through rationality and combining historical language instructions. In addition, the small processing model has a fast running rate and only requires the input of a small number of words each time, so the output accuracy is high.
[0106] Reference Figure 4 , showing a step flow image of a robot control method, specifically comprising the following steps:
[0107] S401, obtaining a target instruction.
[0108] The target instruction includes: action and object to be processed. The target instruction is based on Figure 2 or Figure 3 Generated by the instruction generation method.
[0109] In the present disclosure, the target instruction is an instruction that can be understood by the robot, such as “grab and place (apple, peach)”.
[0110] In addition, the target instruction also includes: a target position, for example, the target instruction is "grab and place ([apple, peach], basket)".
[0111] S402, controlling the robot to perform actions on the object to be processed according to the target instruction.
[0112] In the disclosed embodiment, the robot can perform corresponding tasks according to the target instruction. For example, if the target instruction is "grab and place ([apples, peaches], basket)", the robot grabs apples and peaches, and then places the apples and peaches in the basket.
[0113] In summary, the instruction generation method provided by the present disclosure can generate target instructions for controlling a robot, wherein the effect of the generated target instructions is improved, and the robot's response efficiency to the user's first language instructions can be further improved.
[0114] Reference Figure 5 , is a structural block diagram of an instruction generation device 50 provided by the present disclosure, and the instruction generation device 50 includes:
[0115] A first determination module 51 is used to obtain a first language instruction and determine an action and a target object indicated by the first language instruction through a processing model;
[0116] The acquisition module 52 is used to acquire a target image, where the target image includes: at least one first object;
[0117] A second determination module 53, configured to determine whether each first object in at least one first object matches the target object by processing the model;
[0118] A third determination module 54, configured to determine, if yes, that the first object matching the target object is the object to be processed;
[0119] The generating module 55 is used to generate a target instruction according to the action and the object to be processed, and the target instruction is used to control the robot.
[0120] In an optional embodiment, the first determining module 51 is specifically configured to obtain a historical language instruction of the first language instruction;
[0121] splicing the first language instruction and the historical language instruction to get the second language instruction;
[0122] The second language instruction is processed by the processing model to obtain the action and target object indicated by the first language instruction.
[0123] In an optional embodiment, the first determining module 51, when acquiring the historical language instruction of the first language instruction, is specifically configured to:
[0124] Determining whether the first language instruction is reasonable by processing the model, wherein a reasonable first language instruction is executable by the robot;
[0125] If so, obtain the historical language instruction of the first language instruction.
[0126] In an optional embodiment, the first determination module 51, when determining whether the first language instruction is reasonable through the processing model, is specifically configured to:
[0127] determining, by the processing model, whether the action indicated in the first language instruction is a predetermined action;
[0128] If so, determine that the first language instruction is reasonable.
[0129] In an optional embodiment, the first determination module 51, when determining whether the first language instruction is reasonable through the processing model, is specifically used to:
[0130] Determine whether first language instruction is common sense through processing models;
[0131] If so, determine that the first language instruction is reasonable.
[0132] In an optional embodiment, the generating module 55 is specifically configured to determine the target position indicated by the first language instruction by processing the model;
[0133] Determine the target instruction based on the action, the object to be processed and the target position.
[0134] In an optional embodiment, the acquisition module 52 is specifically used to control the camera to collect images of multiple first objects placed on the target table to obtain the target image.
[0135] In an optional embodiment, the action includes at least one of the following: grabbing, placing, moving or positioning.
[0136] The instruction generation device provided by the present disclosure can implement the above-mentioned instruction generation method. Please refer to the above for details and will not be repeated here.
[0137] Reference Figure 6 , is a structural block diagram of a robot control device 60 provided by the present disclosure, and the robot control device 60 includes:
[0138] An acquisition module 61 is used to acquire a target instruction, the target instruction includes: an action and an object to be processed, and the target instruction is generated by the instruction generation method of the first aspect;
[0139] The control module 62 is used to control the robot to perform actions on the object to be processed according to the target instruction.
[0140] The robot control device provided by the present disclosure can implement the above-mentioned robot control method. Please refer to the above for details and will not be repeated here.
[0141] In addition, in some of the processes described in the above embodiments and the accompanying drawings, multiple operations that appear in a specific order are included, but it should be clearly understood that these operations may not be executed in the order in which they appear in this article or executed in parallel, and are only used to distinguish between different operations, and the sequence number itself does not represent any execution order. In addition, these processes may include more or fewer operations, and these operations may be executed in sequence or in parallel. It should be noted that the descriptions of "second", "first", etc. in this article are used to distinguish different messages, devices, modules, etc., do not represent the order of precedence, and do not limit "second" and "first" to be different types.
[0142] Figure 7 A schematic diagram of the structure of an electronic device provided by an exemplary embodiment of the present disclosure. Figure 7 As shown, the electronic device 70 includes: a processor 71, and a memory 72 communicatively connected to the processor 71, and the memory 72 stores computer-executable instructions.
[0143] Among them, the processor executes the computer execution instructions stored in the memory to implement the instruction generation method provided by any of the above method embodiments, and the specific functions and technical effects that can be achieved are not repeated here.
[0144] The embodiment of the present disclosure further provides a computer-readable storage medium, in which computer-executable instructions are stored. When the computer-executable instructions are executed by a processor, they are used to implement any of the above methods.
[0145] The embodiments of the present disclosure also provide a computer program product, which includes: a computer program, which is stored in a readable storage medium, and at least one processor of an electronic device can read the computer program from the readable storage medium, and at least one processor executes the computer program so that the electronic device executes any of the above methods.
[0146] In the several embodiments provided in the present disclosure, it should be understood that the disclosed systems and methods can be implemented in other ways. For example, the system embodiments described above are only schematic, for example, the division of units is only a logical function division, and there may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of systems or units, which can be electrical, mechanical or other forms.
[0147] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
[0148] In addition, each functional unit in each embodiment of the present disclosure may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of hardware plus software functional units.
[0149] The above-mentioned integrated unit implemented in the form of a software functional unit can be stored in a computer-readable storage medium. The above-mentioned software functional unit is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor (processor) to perform some steps of the methods of various embodiments of the present disclosure. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, ROM), random access memory (Random Access Memory, RAM), disk or optical disk and other media that can store program codes.
[0150] Those skilled in the art can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional modules is used as an example for illustration. In actual applications, the above-mentioned functions can be assigned to different functional modules as needed, that is, the internal structure of the system is divided into different functional modules to complete all or part of the functions described above. The specific working process of the system described above can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.
[0151] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. The present disclosure is intended to cover any variations, uses or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The description and examples are to be considered exemplary only, and the true scope and spirit of the present disclosure are indicated by the following claims.
[0152] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.
Claims
1. A method for generating an instruction, characterized in that: include: Obtaining a first language instruction, and determining an action and a target object indicated by the first language instruction through a processing model; Acquire a target image, the target image comprising: at least one first object; determining, by the processing model, whether each of the at least one first object matches the target object; If so, determining that the first object matching the target object is the object to be processed; A target instruction is generated according to the action and the object to be processed, and the target instruction is used to control the robot.
2. The instruction generation method according to claim 1, characterized in that: The determining the action and target object indicated by the first language instruction by processing the model includes: obtaining historical language instructions of the first language instructions; splicing the first language instruction and the historical language instruction to obtain a second language instruction; The second language instruction is processed by the processing model to obtain the action and target object indicated by the first language instruction.
3. The instruction generation method according to claim 2, characterized in that: The step of obtaining the historical language instruction of the first language instruction includes: Determining whether the first language instruction is reasonable by using the processing model, wherein a reasonable first language instruction is executable by the robot; If so, obtain the historical language instructions of the first language instructions.
4. The instruction generation method according to claim 3, characterized in that: The determining whether the first language instruction is reasonable by using the processing model includes: Determining, by the processing model, whether the action indicated in the first language instruction is a preset action; If so, it is determined that the first language instruction is reasonable.
5. The instruction generation method according to claim 3, characterized in that: Determining whether the first language instruction is reasonable by using the processing model includes: determining, by the processing model, whether the first language instruction is common sense; If so, it is determined that the first language instruction is reasonable.
6. The instruction generation method according to any one of claims 1 to 5, characterized in that: The generating a target instruction according to the action and the to-be-processed pair comprises: Determining, by means of the processing model, a target location indicated by the first language instruction; The target instruction is determined according to the action, the object to be processed and the target position.
7. The instruction generation method according to any one of claims 1 to 5, characterized in that: The acquiring of the target image comprises: The camera is controlled to collect images of a plurality of first objects placed on the target table to obtain the target image.
8. The instruction generation method according to any one of claims 1 to 5, characterized in that: The action includes at least one of the following: grabbing, placing, moving or positioning.
9. A robot control method, characterized in that: include: Acquire a target instruction, the target instruction comprising: an action and an object to be processed, the target instruction being generated by the instruction generation method according to any one of claims 1 to 8; The robot is controlled according to the target instruction to perform the action on the object to be processed.
10. An electronic device, characterized in that: include: A processor, a memory, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the method according to any one of claims 1 to 9 is implemented.
11. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, which are used to implement the method according to any one of claims 1 to 9 when executed by a processor.