Task processing method, device, electronic equipment and robot for robot

Abstract

The present invention provides a task processing method, device, electronic device, and robot for a robot, relating to technical fields such as artificial intelligence, trusted artificial intelligence, intelligent robots, and intelligent equipment. The method comprises: in response to receiving task information input by a user, obtaining a first control panel image and multimodal operation information of an operated object indicated by the task information, the multimodal operation information including textual operation information; invoking a large language model to generate an operation sequence based on the task information and the textual operation information; generating an instruction sequence based on the operation sequence, the first control panel image, and the multimodal operation information; and controlling the robot to operate each operation area according to the instruction sequence to process the task corresponding to the task information. This invention can solve the technical problem of robots having difficulty accurately executing tasks, resulting in a limited user experience, ensuring that the robot can accurately manipulate the operation area, thereby achieving the technical effect of improving the user experience.

Description

Technical Field

[0001] The present invention relates to the technical fields of artificial intelligence, trusted artificial intelligence, intelligent robots, intelligent equipment, etc., and relates to a task processing method, device, electronic equipment and robot for robots. Background Art

[0002] With the rapid development of artificial intelligence, robots are increasingly being used in a wide range of fields, including industrial and agricultural production, construction, logistics, and daily life. Embodied intelligence is developing even faster. Robots with embodied intelligence can interact with their environment based on user task information, enabling the robot to operate equipment. The devices that robots can operate in everyday life often include a variety of control panels, often with multiple operating logics.

[0003] In the process of realizing the concept of the present invention, the inventors found that there are at least the following problems in the related technology: robots usually learn tasks by learning pre-constructed end-to-end instances, but for flexible and changeable task information, it is difficult for robots to understand or accurately perform the tasks corresponding to the task information, which affects the user experience. Summary of the Invention

[0004] In view of this, the present invention provides a task processing method, device, electronic equipment and robot for a robot.

[0005] One aspect of the present invention provides a task processing method for a robot, comprising: in response to receiving task information input by a user, obtaining a first control panel image and multimodal operation information of an operated object indicated by the task information, wherein the multimodal operation information includes text operation information for operating the operated object; calling a large language model to generate an operation sequence based on the task information and the text operation information, wherein the operation sequence includes at least one operation area; generating an instruction sequence based on the operation sequence, the first control panel image and the multimodal operation information, wherein the instruction sequence includes control instructions corresponding to each operation area; and controlling the robot to operate each operation area according to the instruction sequence to process the task corresponding to the task information.

[0006] According to an embodiment of the present invention, a large language model is called to generate an operation sequence based on task information and text operation information, including: obtaining prompt information, wherein the prompt information includes an example operation sequence, at least one first vacant field corresponding to the operated object, and a second vacant field corresponding to the task information; using the text operation information and the task information to fill the first vacant field and the second vacant field, respectively, to obtain updated prompt information; and inputting the updated prompt information into the large language model to output an operation sequence consistent with the format of the example operation sequence.

[0007] According to an embodiment of the present invention, the first missing field and the second missing field are filled in respectively using text operation information and task information to obtain updated prompt information, including: determining the target field value that matches each first missing field from the text operation information according to the field name of the first missing field in the prompt information; and filling the target field value into the first missing field and filling the task information into the second missing field to obtain updated prompt information.

[0008] According to an embodiment of the present invention, the multimodal operation information also includes a pre-collected second control panel image, and the first control panel image is collected by the robot in response to receiving input task information; an instruction sequence is generated based on the operation sequence, the first control panel image and the multimodal operation information, including: determining a position transformation relationship based on the first control panel image and the second control panel image, wherein the position transformation relationship represents a position transformation relationship between the coordinate system of the second control panel image and the coordinate system of the first control panel image; determining the depth information of each operation area based on the first control panel image; and generating a control instruction based on the depth information, the position transformation relationship and the text operation information.

[0009] According to an embodiment of the present invention, determining a position transformation relationship based on a first control panel image and a second control panel image includes: performing feature registration on the first control panel image and the second control panel image to obtain at least one feature point pair; and determining the position transformation relationship based on the two-dimensional position information of each of the two feature points in the feature point pair.

[0010] According to an embodiment of the present invention, the text operation information includes the two-dimensional position information of each operation area in the second control panel image; based on the depth information, the position transformation relationship and the text operation information, a control instruction is generated, including: for each operation area, based on the two-dimensional position information and the position transformation relationship of the operation area in the second control panel image, determining the two-dimensional position information of the operation area in the robot coordinate system; based on the depth information and the two-dimensional position information of the operation area in the robot coordinate system, determining the three-dimensional position information of the operation area; and generating a control instruction based on the three-dimensional position information.

[0011] According to an embodiment of the present invention, the method further includes: calling a region detection model to perform region detection on the second control panel image to obtain first identification information of at least one operation area; calling a fine-tuned visual language model to generate text operation information based on the first identification information and the second control panel image; and combining the text operation information and the second control panel image into multimodal operation information and storing it in a database, so that in response to receiving input task information, the multimodal operation information is obtained from the database.

[0012] According to an embodiment of the present invention, the visual language model is obtained by fine-tuning by the following method: calling the region detection model to perform region detection on the sample control panel image to obtain second identification information of at least one sample operation area; calling the multimodal large model to generate region description information based on the sample control panel image identified with the second identification information, wherein the region description information is used to describe the operation function of the sample operation area; calling the multimodal large model to generate sample text operation information based on the region description information, the second identification information and the sample control panel image; and using multiple sample text operation information and multiple sample control panel images to fine-tune the visual language model to obtain a fine-tuned visual language model.

[0013] Another aspect of the present invention provides a task processing device for a robot, comprising: an acquisition module for acquiring, in response to receiving task information input by a user, a first control panel image and multimodal operation information of an operated object indicated by the task information, wherein the multimodal operation information includes text operation information for operating the operated object; a calling module for calling a large language model to generate an operation sequence based on the task information and the text operation information, wherein the operation sequence includes at least one operation area; a generation module for generating an instruction sequence based on the operation sequence, the first control panel image and the multimodal operation information, wherein the instruction sequence includes control instructions corresponding to each operation area; and a control module for controlling the robot to operate each operation area according to the instruction sequence to process the task corresponding to the task information.

[0014] Another aspect of the present invention provides an electronic device, comprising: one or more processors; and a memory for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the above method.

[0015] Another aspect of the present invention provides a computer-readable storage medium storing computer-executable instructions, which are used to implement the above method when executed.

[0016] Another aspect of the present invention provides a computer program product, which includes computer executable instructions. When the instructions are executed, they are used to implement the above method.

[0017] Another aspect of the present invention provides a robot for implementing the above method.

[0018] In an embodiment of the present invention, a large language model is invoked to generate an operation sequence based on task information and textual operation information. By inputting the textual operation information used to operate the operated object and the user's task information into the large language model, the large language model can understand the operational logic of the first control panel image and flexibly generate an operation sequence corresponding to the task information, thereby ensuring that the robot can effectively understand the task information. Furthermore, an instruction sequence is generated based on the operation sequence, the first control panel image, and the multimodal operation information, wherein the instruction sequence includes control instructions corresponding to each operation area. The robot is controlled to operate each operation area according to the instruction sequence, ensuring that the robot can accurately manipulate the operation area to accurately complete the task. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The above and other objects, features and advantages of the present invention will become more apparent through the following description of the embodiments of the present invention with reference to the accompanying drawings, in which:

[0020] Figure 1 An exemplary system architecture is shown to which the method and apparatus for processing a task for a robot of the present invention can be applied.

[0021] Figure 2 A flowchart of a task processing method for a robot according to an embodiment of the present invention is shown.

[0022] Figure 3 A schematic diagram of a scenario for generating an operation sequence according to an embodiment of the present invention is shown.

[0023] Figure 4 A flow chart of generating control instructions according to an embodiment of the present invention is shown.

[0024] Figure 5 A schematic diagram of a scenario for generating a control instruction according to an embodiment of the present invention is shown.

[0025] Figure 6 A schematic diagram of a scenario for generating an instruction sequence according to an embodiment of the present invention is shown.

[0026] Figure 7 A block diagram of a task processing device for a robot according to an embodiment of the present invention is shown.

[0027] Figure 8 A block diagram of an electronic device suitable for implementing task processing for a robot according to an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0028] Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the present invention. In the following detailed description, for ease of explanation, many specific details are set forth to provide a comprehensive understanding of embodiments of the present invention. However, it is apparent that one or more embodiments may also be implemented without these specific details. In addition, in the following description, descriptions of known structures and technologies are omitted to avoid unnecessarily confusing the concept of the present invention.

[0029] The terms used herein are only for describing specific embodiments and are not intended to limit the present invention. The terms "comprise", "include", etc. used herein indicate the presence of the features, steps, operations and / or components, but do not exclude the presence or addition of one or more other features, steps, operations or components.

[0030] All terms used herein (including technical and scientific terms) have the meanings commonly understood by those skilled in the art unless otherwise defined. It should be noted that the terms used herein should be interpreted as having a meaning consistent with the context of this specification and should not be interpreted in an idealized or overly rigid manner.

[0031] When expressions such as "at least one of A, B, and C, etc." are used, they should generally be interpreted in accordance with the meaning commonly understood by those skilled in the art (for example, "a system having at least one of A, B, and C" should include but is not limited to a system having A alone, B alone, C alone, A and B, A and C, B and C, and / or A, B, C, etc.).

[0032] In the embodiments of the present invention, the collection, updating, analysis, processing, use, transmission, provision, disclosure, and storage of all data involved (including, but not limited to, user personal information) comply with relevant laws and regulations, are used for legitimate purposes, and do not violate public order and good morals. In particular, necessary measures are taken to prevent unauthorized access to user personal information data and safeguard the security of user personal information, network security, and national security.

[0033] In the embodiment of the present invention, the user's authorization or consent is obtained before obtaining or collecting the user's personal information.

[0034] In daily life, most devices that require operation have corresponding control panels. Embodied intelligence robots can thus operate these devices by operating the control panels. For example, when getting a cup of coffee in the morning, one needs to select the preferred type of coffee on the coffee machine's control panel. When taking the elevator, one needs to select a different floor according to their needs. When working, one needs to use a mouse and keyboard according to their needs. When returning home, one needs to use the air conditioner remote control to set the appropriate mode. When doing laundry in the evening, one needs to operate the washing machine's control panel.

[0035] However, existing technologies for embodied intelligent robot task learning can be achieved through imitation learning or by using large language models to generate robot action sequences. However, these approaches treat a single robot task as an end-to-end skill. While the robot can complete the learned task, it cannot flexibly complete other related tasks based on the user's task information. In other words, existing robots cannot understand the operational logic of tasks and, consequently, cannot flexibly execute multiple tasks on a variety of operation panels based on diverse task information.

[0036] To this end, embodiments of the present invention provide a task processing method for a robot. By inputting textual operation information for manipulating an object and the user's task information into a large language model, the large language model is able to understand the operational logic of a first control panel image and flexibly generate an operation sequence corresponding to the task information, thereby ensuring that the robot can fully understand the task information. Subsequently, a command sequence is generated based on the operation sequence, the first control panel image, and the multimodal operation information. This command sequence controls the robot to operate each operation area, ensuring that the robot can accurately manipulate the operation area and accurately complete the task.

[0037] Figure 1 The following shows an exemplary system architecture to which the task processing method and apparatus for robots of the present invention can be applied. Figure 1 The examples shown are merely examples of system architectures to which the embodiments of the present invention may be applied, to help those skilled in the art understand the technical content of the present invention, but do not mean that the embodiments of the present invention cannot be used in other devices, systems, environments or scenarios.

[0038] like Figure 1 As shown, the system architecture 100 according to this embodiment may include a user 101 , a robot 102 , and an operated object 103 .

[0039] The operated object 103 can be a device with a control panel, such as a coffee machine, elevator, washing machine, printer, or paper shredder. The control panel can include physical controls such as buttons, keys, knobs, levers, springs, and sliders. The control panel can also include a display screen containing a variety of virtual controls. Both these physical and virtual controls can be the robot's operating area. For example, the robot can perform operations such as pressing, turning, pushing, pulling, touching, and sliding on the operating area to complete the task corresponding to the task information.

[0040] The user 101 can interact with the robot 102 in various ways to obtain task information. For example, the user 101 can interact with the robot 102 through touch screen, voice, keyboard, gestures, etc.

[0041] The robot 102 may be equipped with various sensors to convert the user 101's input into text-based task information. For example, the robot may use a camera to capture visual images and extract content from them to obtain task information; or a microphone to receive sound signals and convert them into task information.

[0042] It should be noted that the task processing method for a robot provided in the embodiments of the present invention can generally be executed by the robot 102. Accordingly, the task processing device for a robot provided in the embodiments of the present invention can generally be disposed in the robot 102. In addition, the task processing method for a robot provided in the embodiments of the present invention can also be implemented by a server, and the task processing device for a robot can also be disposed in the server.

[0043] For example, after user 101 interacts with robot 102 to obtain task information, robot 102 transmits the task information to a server. In response to receiving the input task information, the server obtains a first control panel image and multimodal operation information of the operated object indicated by the task information, where the multimodal operation information includes text operation information for operating the operated object. The server then invokes a large language model to generate an operation sequence based on the task information and the text operation information, where the operation sequence includes at least one operation area. The server then generates an instruction sequence based on the operation sequence, the first control panel image, and the multimodal operation information, where the instruction sequence includes control instructions corresponding to each operation area. The server then sends the instruction sequence to the robot to control the robot to operate each operation area according to the instruction sequence to process the task corresponding to the task information.

[0044] The server can be a server that provides various services. For example, the server can analyze and process the received data, and feed back the processing results (such as web pages, information, or data obtained or generated according to user requests) to the robot.

[0045] It should be understood that Figure 1 The number of robots and operated objects in the figure is only for illustration. Any number of robots and operated objects may be provided according to the implementation requirements.

[0046] Figure 2 FIG. 1 is a flow chart of a task processing method for a robot according to an embodiment of the present invention. Figure 2 As shown, the method 200 includes operations S210 to S240.

[0047] In operation S210 , in response to receiving task information input by a user, a first control panel image and multimodal operation information of an operated object indicated by the task information are acquired, wherein the multimodal operation information includes text operation information for operating the operated object.

[0048] Task information is used to represent the user's task requirements. For example, task information may include the object being operated, allowing the operated object to be directly determined from the task information. For example, task information could be "Help me make a cup of coffee in the coffee machine" or "Help me wash my clothes in the washing machine," and the operated objects could be "coffee machine" or "washing machine."

[0049] Alternatively, the task information may not include the object being operated on. In this case, the object being operated on can be determined by understanding the content of the task information. For example, the task information could be "Please make me a cup of coffee" or "Please do my laundry." By understanding the content of the task information, "coffee machine" or "washing machine" can be determined as the object being operated on. In one embodiment, the content understanding task can be performed by invoking large language models (LLMs) to determine the object being operated on from the task information. These LLMs can be the same as or different from the LLMs used to generate the operation sequence.

[0050] The multimodal operation information may be predetermined and stored in a database and may be directly retrieved from the database when needed. The multimodal operation information includes at least text operation information in a text mode. The text operation information is used to indicate the operation logic for operating the control panel of the operated object. For example, if the operated object is a rice cooker, the text operation information may be "Press and hold the power button to adjust the cooking mode."

[0051] The first control panel image may be captured by the robot. The first control panel image refers to an image of the control panel including the operated object. To ensure clarity of the text and operation area on the control panel, the first control panel image may be captured from a predetermined viewing angle, such as a view directly facing the control panel.

[0052] In operation S220 , the large language model is called to generate an operation sequence according to the task information and the text operation information, wherein the operation sequence includes at least one operation area.

[0053] In an embodiment of the present invention, large language models (LLMs) can understand task information and textual operation information and, based on the understood textual operation information, generate an operation sequence for processing the task corresponding to the task information. For example, after understanding the task information and textual operation information, the large language model can break down the task information and learn to achieve the corresponding task through multiple operations on the control panel. Each operation on the control panel corresponds to an operation area, which can be a physical or virtual control on the control panel, or the area where the control is located.

[0054] For example, taking the task information of "help me make a cup of hot American coffee" as an example, the operation sequence may include three operation areas A, B, and C, which correspond to the position areas of the controls for selecting the type of coffee, temperature, and making coffee, respectively.

[0055] In operation S230 , an instruction sequence is generated according to the operation sequence, the first control panel image, and the multimodal operation information, wherein the instruction sequence includes a control instruction corresponding to each operation area.

[0056] In the implementation of the present invention, the entire task processing process includes: understanding the task and processing the task. The operation sequence generated by LLMs completes the task arrangement based on the understanding of the task. However, for a specific robot, it is also necessary to control the robot to operate at least one operation area to realize the processing task.

[0057] Specifically, the first control panel image visually presents the control panel of the operated object. After determining the operation sequence, each operation area in the operation sequence needs to be located in the first control panel image to control the robot to operate the operation area at the correct location. Considering that multimodal operation information is used to indicate how to operate the operated object, when locating the operation area, the multimodal operation information and the first control panel image can be used to locate the operation area. Based on the located position information, control instructions for operating each operation area are generated. Multiple control instructions constitute an instruction sequence.

[0058] In one embodiment, at least one operation area in the operation sequence has an operation order, and at least one control instruction in the generated instruction sequence is arranged in this operation order. For example, taking the task information "Make me a hot Americano," the operation sequence may include three operation areas A, B, and C, and the generated instruction sequence may include control instructions A1, B1, and C1, which are used to control the robot to press controls for selecting coffee type, temperature, and brewing, respectively.

[0059] In operation S240 , the robot is controlled to operate each operation area according to the instruction sequence to process the task corresponding to the task information.

[0060] For example, the robot can be controlled to operate each operating area according to each control instruction in the instruction sequence. After completing all control instructions in the instruction sequence, the task is completed. In one embodiment, the robot is typically a robot with embodied intelligence and has a built-in control chip or processor for processing the above tasks.

[0061] In an embodiment of the present invention, a large language model is invoked to generate an operation sequence based on task information and textual operation information. By inputting the textual operation information used to operate the operated object and the user's task information into the large language model, the large language model can understand the operational logic of the first control panel image and flexibly generate an operation sequence corresponding to the task information, thereby ensuring that the robot can effectively understand the task information. Furthermore, an instruction sequence is generated based on the operation sequence, the first control panel image, and the multimodal operation information, wherein the instruction sequence includes control instructions corresponding to each operation area. The robot is controlled to operate each operation area according to the instruction sequence, ensuring that the robot can accurately manipulate the operation area to accurately complete the task.

[0062] According to an embodiment of the present invention, a large language model is called to generate an operation sequence based on task information and text operation information, including: obtaining prompt information, wherein the prompt information includes an example operation sequence, at least one first vacant field corresponding to the operated object, and a second vacant field corresponding to the task information; using the text operation information and the task information to fill the first vacant field and the second vacant field, respectively, to obtain updated prompt information; and inputting the updated prompt information into the large language model to output an operation sequence consistent with the format of the example operation sequence.

[0063] The prompt information is pre-built and stored. When the large language model is called, the prompt information can be obtained from a predetermined address. It is understood that the example operation sequence, the first vacant field, and the second vacant field are also predetermined and are used to guide the large language model to generate an operation sequence in a predetermined format.

[0064] It should be noted that the first and second of the first and second vacant fields are only used to distinguish the corresponding relationship between the operated object and the task information.

[0065] In one embodiment, the operation sequence includes not only the operation areas, but also the operation duration for each operation area and / or the waiting time after operating each operation area. Therefore, each control instruction in the instruction sequence may also include a time parameter for controlling the robot to operate the operation area according to the operation duration and / or wait for the waiting time after operating the operation area.

[0066] In this embodiment, for a control instruction that includes a waiting time, when controlling the robot to perform an operation based on the control instruction, the robot can process the task in an operation that is more consistent with the control panel logic, so as to ensure accurate task processing and improve the user experience.

[0067] For example, a sample sequence of operations could be: <1> 0. <4> 0. <4> 0. <3> 20. <1> 0. Each line represents an operation, where the numbers in <> correspond to different operation areas in the device operation function, and the number after <> represents the waiting time after the operation, in seconds. The above operation instructions wait 20 seconds after the fourth step before proceeding to the fifth step.

[0068] The first and second missing fields represent fields whose values ​​are empty. There can be multiple first missing fields. For example, the first missing field can include at least one of the following: device name (template['device_name']), device function (template['device_function']), device operation description (template['description']), device operation function modules (modules), all possible pages (pages), the current operation page (this_page), the description of the current operation page (page_description), etc.

[0069] The field value of the first missing field can be obtained from the text operation information in the multimodal operation information. For example, for multiple first missing fields, for each first missing field, a field value matching each first missing field can be determined from the text operation information, and the field value is filled into the corresponding first missing field.

[0070] The second blank field may be one, such as task, and the task information may be directly filled as the field value of the second blank field.

[0071] In one embodiment, the field values ​​of the first missing field and the second missing field in the prompt information can be marked by special front identifier and back identifier so that the field values ​​can be filled between the front identifier and the back identifier using text operation information and task information.

[0072] After the prompt information is filled in using the text operation information and task information, the updated prompt information can be obtained. After the prompt information is input into the large language model, the operation sequence can be output.

[0073] In an embodiment of the present invention, prompt information is obtained and the first and second vacant fields are filled with text operation information and task information, respectively, to obtain updated prompt information. The updated prompt information is input into a large language model, and an operation sequence consistent with the format of an example operation sequence is output. This enables the large language model, with the assistance of the text operation information, to generate an operation sequence that conforms to the operation logic of the operated object and is consistent with the format of the example operation sequence, and flexibly adapts to different operated objects and task information, thereby improving the flexibility of the task and the accuracy of task execution.

[0074] For example, the prompt message may be: "There are the following device user manuals: Device name: {template['device_name']}, Device function: {template['device_function']}, Device operation instructions: {template['description']}, Device operation function modules: {modules}, All possible pages: {pages}, Current operation page: {this_page}, Description of current operation page: {page_description}, Please refer to the above instructions and generate an operation sequence for device operation based on the user's operation instructions. The sequence is output in the following format. Please do not generate any other additional text: Example operation sequence (as above). The user's task information is: {task}. Please generate an operation sequence. The part of the curly braces that is missing is obtained from the corresponding text operation information. In this prompt message, the leading and trailing identifiers are { and} respectively."

[0075] Figure 3 FIG. 1 shows a schematic diagram of a scenario for generating an operation sequence according to an embodiment of the present invention. Figure 3As shown, in embodiment 300, the acquired prompt information 301 includes a first missing field 311, a second missing field 312, and an example operation sequence 313. After the first missing field 311 is filled with text operation information T1 and the second missing field 312 is filled with task information T2, the example operation sequence 313, the filled first missing field, and the filled second missing field constitute updated prompt information 304. After the updated prompt information 304 is input into the large language model M1, an operation sequence T3 having the same format as the example operation sequence 313 is obtained.

[0076] For example, the sequence of operations could be <2> 0. <3> 0. <1> 0. <1> 3. <2> 0 means operating the second button of the rice cooker (such as the power button) to turn on the rice cooker; operating the third button (such as the mode button) to select the mode; operating the first button (such as the time button) twice (each operation increases half an hour), setting the timer to one hour, and waiting for 3 seconds to complete the timer; then operating the second button to confirm the start of cooking.

[0077] According to an embodiment of the present invention, the first missing field and the second missing field are filled in respectively using text operation information and task information to obtain updated prompt information, including: determining the target field value that matches each first missing field from the text operation information according to the field name of the first missing field in the prompt information; filling the target field value into the first missing field and filling the task information into the second missing field to obtain updated prompt information.

[0078] In an embodiment of the present invention, both the prompt information and the text operation information may include a field name, whereby the target field value matching the first missing field may be determined from the text operation information by the field name. The field name of the first missing field may be the aforementioned device_name, etc.

[0079] For example, by calculating the similarity between field names, a field name similar to the field name of the first missing field can be determined from the text operation information, and the field value of the field name can be determined to be the target field value that matches the first missing field.

[0080] In one embodiment, the field name-field value in the text operation information has a specific format, such as the specific format: "field name": "field value". The field name can be determined by identifying the specific format, and the similarity between the field name and the field name of the first missing field in the prompt information can be calculated.

[0081] For the second vacant field, the task information can be directly filled into the default value of the vacant field without the need for a matching operation. After filling the first vacant field and the second vacant field, updated prompt information can be obtained.

[0082] In an embodiment of the present invention, by determining the target field value that matches each first missing field from the text operation information based on the field name of the first missing field in the prompt information, and filling the first missing field with the target field value and filling the second missing field with the task information, the correspondence between the first missing field and the target field value in the updated prompt information can be ensured to avoid the subsequent generation of an erroneous operation sequence based on the erroneous matching relationship.

[0083] In one embodiment, the text operation information may be: {"image_dir":"0_Color.png","points":[{"label":"1","points":[75.92028985507247,63.956521739130444]},{"label":"2","points":[160.84782608695653,64.10144927536233]},{"label":"3 ","points":[245.7753623188406,63.521739130434796]},{"label":"4","points":[75.630434782 6087,123.37681159420289]},{"label":"5","points":[160.84782608695653,123.37681159420289]},{"label":"6","points":[246.21014492753625,123.81159420289855]}],"modules":{"1":"<Button> Make Espresso","2":"<Button> Make Hot Latte","3":"<Button> Make Hot Mocha","4":"<Button> Make Hot Americano","5":"<Button> Make Hot Cappuccino","6":"<Button> Make Hot Chocolate Milk"},"description":"When using this device, you can make different types of coffee by clicking different buttons on the control panel of the device. The use of this coffee machine includes three pages: the start page: for selecting the type of coffee to be made, the detailed settings page: for setting the sweetness, and the confirmation page: for confirming the making. The current page is: the start page "next_template":{"1":"0_1_template.json","2":"0_2_template.json","3":"0_3_template.json","4":"0_4_template.json","5":"0_5_template.json","6":"0_6_template.json"},"pages":["Start page","Detailed settings page","Confirmation page"],"this_page":"Start page","this_page_description":"Used to select the type of coffee you want to make"}.

[0084] The meanings of the field names are as follows: "image_dir": the address of the second control panel image in the multimodal operation information; "points": a list of all possible operation areas, where each element is a dictionary containing two pieces of information: "label" is the label of the operation area, and "points" is the pixel coordinates of the operation area on the second control panel image. "modules": a text description of each operation area. The description starts with the type of a specific operation area and is marked with '<>'. The types of operation areas include: (1) buttons / touch screen keys; (2) knobs; (3) levers / shrapnels / sliders; (4) others, four categories. "description": a detailed description of the functions and operation logic of the control panel and the operation target. "device_name": the name of the operation target corresponding to the control panel "device_function": the function of the operation target. "next_template": if there is a page change in the control panel, the next template corresponding to a certain operation needs to be formulated in sequence. For example, after a rice cooker selects the cooking function, it will switch to the next control panel logic to select the softness of the rice. At this time, the physical control panel may not change, but the entire operation logic has changed, so a new template needs to be specified. "pages": used to define all possible logic panels of the current control panel. "this_page": used to specify the logic panel to which the current control panel belongs. "this_page_description": used to describe what function the current logic panel is used to implement.

[0085] In a specific embodiment, the text operation information may include a template for each page. Combining the meaning of the field name and the text operation information, it can be determined that the text operation information belongs to the template of the "start page", which provides six buttons for selecting different coffee types.

[0086] Figure 4 A flow chart of generating control instructions according to an embodiment of the present invention is shown.

[0087] like Figure 4 As shown, the embodiment 400 of generating a control instruction includes operations S431 to S433.

[0088] In operation S431 , a position transformation relationship is determined based on the first control panel image and the second control panel image, wherein the position transformation relationship represents a position transformation relationship between a coordinate system of the second control panel image and a coordinate system of the first control panel image.

[0089] According to an embodiment of the present invention, the multimodal operation information also includes a pre-captured second control panel image, where the first control panel image is captured by the robot in response to receiving input task information. In this embodiment, the second control panel image can be captured by a device other than the robot, or by the robot before capturing the first control panel image. For example, the second control panel image can be captured by the robot when generating the multimodal operation information.

[0090] For example, if the second control panel image is captured by a device other than the robot, the coordinate systems of the first and second control panel images will differ, and the coordinates of the same operating area will differ in the first and second control panel images. If both the first and second control panel images are captured by the robot, the coordinate systems of the two images may also differ due to different shooting positions.

[0091] Since the first control panel image and the second control panel image both include the same control panel, the position transformation relationship between the two coordinate systems can be determined through the same control panel.

[0092] In one embodiment, as in the above points field, its field value includes xy two-dimensional position, so the position change relationship in the embodiment of the present invention can be a position change relationship under xy two-dimensional coordinates.

[0093] In operation S432, depth information of each operation area is determined based on the first control panel image.

[0094] According to an embodiment of the present invention, a depth prediction model can be used to perform depth prediction on each pixel of the first control panel image to obtain depth information of each pixel in the first control panel image. The depth information of each operation area includes: the depth information of each pixel in the operation area.

[0095] For example, the depth prediction model can be a network such as Convolutional Neural Networks (CNN), deep learning, U-net, etc.

[0096] In operation S433 , a control instruction is generated according to the depth information, the position transformation relationship, and the text operation information.

[0097] Based on the above statements, it can be seen that the text operation information can include the two-dimensional position information of each operation area, and the depth information can be used as the third dimension. The three-dimensional position information of the operation area is determined in combination with the two-dimensional position information. Thus, the coordinates of the operation area in the robot's coordinate system can be obtained, and then the control instructions for controlling the robot to operate the operation area can be generated.

[0098] In this embodiment of the present invention, by determining the position transformation relationship between the first and second control panel images, the two-dimensional coordinate offset of the control panel caused by the coordinate system difference can be determined. By determining the depth information of each operation area using the first control panel image, the third-dimensional coordinate based on the current captured image can be determined. As a result, the control instructions generated based on the depth information, position transformation relationship, and text operation information provide accurate position information in the robot coordinate system, ensuring the accuracy of subsequent task processing.

[0099] According to an embodiment of the present invention, determining a position transformation relationship based on a first control panel image and a second control panel image includes: performing feature registration on the first control panel image and the second control panel image to obtain at least one feature point pair; and determining the position transformation relationship based on the two-dimensional position information of each of the two feature points in the feature point pair.

[0100] For example, a feature detection algorithm can be used to perform feature detection on the first control panel image to obtain at least one feature point of the first control panel image and a descriptor for each feature point. Similarly, a feature detection algorithm can be used to perform feature detection on the second control panel image to obtain at least one feature point of the second control panel image and a descriptor for each feature point. Subsequently, a feature matching algorithm can be used to determine, based on the descriptors of the feature points, feature points that match each feature point in the first control panel image from the at least one feature point of the second control panel image, thereby forming feature point pairs. In one embodiment, there can generally be multiple feature point pairs between the first control panel image and the second control panel image.

[0101] Feature detection algorithms can include the Scale-Invariant Feature Transform (SIFT) algorithm, the Speeded Up Robust Features (SURF) algorithm, or the Oriented Fast and Rotated BRIEF (ORB) algorithm. Feature matching algorithms can include the Fast Approximate Nearest Neighbor Search Library (FLANN) algorithm, the brute force matcher (BFMatcher), and so on.

[0102] In an embodiment of the present invention, each feature point in a feature point pair is a pixel in the first control panel image / the second control panel image. Therefore, the two-dimensional position information of the feature point in the second control panel image can be obtained based on the points field in the text operation information. When the robot captures the first control panel image, a pose detector can be provided within the robot to determine the two-dimensional position information of each feature point when capturing the first control panel image.

[0103] Using the two feature points in each feature point pair, the position transformation relationship between the coordinate system of the second control panel image and the coordinate system of the first control panel image can be calculated. In one embodiment, the position transformation relationship can be represented by a homography matrix.

[0104] For example, a random sample consensus algorithm (RANSAC) is used to calculate the homography matrix from the second control panel image to the first control panel image using multiple pairs of feature points. For example, the following formula (1) can be used:

[0105] (1)

[0106] Among them, for the two feature points in the feature point pair, x and y are the horizontal and vertical coordinates of the feature points on the first control panel image (two-dimensional position information in the first control panel image), z is 1, x is 1, and t 、y t are the horizontal and vertical coordinates of the feature points on the second control panel image (two-dimensional position information in the second control panel image), z t is 1, is the homography matrix, h 11 ~ h 33 are the 9 parameter values ​​of the homography matrix obtained by solving.

[0107] In an embodiment of the present invention, at least one feature point pair is obtained by performing feature registration on the first control panel image and the second control panel image, and a position transformation relationship is determined based on the two-dimensional position information of each of the two feature points in the feature point pair. This allows for determining the accurate position transformation relationship between the second control panel image and the currently captured first control panel image, which helps to subsequently locate the accurate operating area and control the robot to accurately perform tasks.

[0108] According to an embodiment of the present invention, the text operation information includes the two-dimensional position information of each operation area in the second control panel image; based on the depth information, the position transformation relationship and the text operation information, a control instruction is generated, including: for each operation area, based on the two-dimensional position information and the position transformation relationship of the operation area in the second control panel image, determining the two-dimensional position information of the operation area in the robot coordinate system; based on the depth information and the two-dimensional position information of the operation area in the robot coordinate system, determining the three-dimensional position information of the operation area; and generating a control instruction based on the three-dimensional position information.

[0109] It can be understood that, since the first control panel image is taken by a robot, the coordinate system of the first control panel image is also referred to as the robot coordinate system.

[0110] For example, after determining the two-dimensional position transformation relationship, the field value of the points field in the text operation information can be used as the two-dimensional position information of the operation area. Thus, for each operation area, the known two-dimensional position information in the second control panel image can be converted into two-dimensional position information in the robot coordinate system based on the homography matrix, thereby determining the accurate two-dimensional position without the need for complex positioning processing of the second control panel image. On this basis, the depth information is used as the coordinate of the third dimension, combined with the two-dimensional position information of the robot coordinate system determined above, the three-dimensional position information of the operation area in the real three-dimensional space, such as the three-dimensional coordinates, can be obtained. After determining the three-dimensional position information, a control instruction can be generated based on the robot's control logic so that the robot can perform corresponding operations on the operation area at the three-dimensional position information according to the control instruction.

[0111] In an embodiment of the present invention, the three-dimensional position information of the operating area is determined by depth information and the two-dimensional position information of the operating area in the robot coordinate system; based on the three-dimensional position information, a control instruction is generated, which can simply and accurately locate the operating area and control the robot to accurately process the task according to the control instruction.

[0112] Figure 5 A schematic diagram of a scenario for generating a control instruction according to an embodiment of the present invention is shown.

[0113] like Figure 5As shown, in this scenario 500, multimodal operation information 502 includes a second control panel image P2 and text operation information T1. Depth prediction is performed on the first control panel image P1 to obtain depth information 503. A position transformation relationship 504 can be determined based on the first control panel image P1 and the second control panel image P2. Based on the position transformation relationship 504 and the two-dimensional position information 505 of each operation area in the text operation information T1 in the second control panel image, two-dimensional position information 506 of each operation area in the first control panel image is determined. Based on the depth information 503 and the two-dimensional position information 506 of each operation area in the first control panel image, three-dimensional position information 507 of each operation area can be determined, and then a control instruction 508 corresponding to each operation area is generated based on the three-dimensional position information 507.

[0114] According to an embodiment of the present invention, the method further includes: calling a region detection model to perform region detection on the second control panel image to obtain first identification information of at least one operation area; calling a fine-tuned visual language model to generate text operation information based on the first identification information and the second control panel image; and combining the text operation information and the second control panel image into multimodal operation information and storing it in a database, so that in response to receiving input task information, the multimodal operation information is obtained from the database.

[0115] In an embodiment of the present invention, for each operated object, multimodal operation information only needs to be generated once based on the second control panel image. When a task needs to be processed, in response to receiving task information input by the user, the multimodal operation information of the operated object can be directly obtained from the database based on the identification information of the operated object.

[0116] The region detection model can be pre-trained and can be a general model for performing region detection on the second control panel image of various operated objects. For example, the region detection model can be a YOLO (You Only Look Once) model, such as YOLO v11.

[0117] When training the region detection model, acquire as many images as possible, including those of the control panel's operated objects. After screening and labeling, sample data is obtained for training. In this sample data, the operating area of ​​each operated object is labeled. These areas include keys, buttons, levers, sliders, and other control panel features. The sample data is then used to train the initial region detection model until the training requirements are met and the region detection model is obtained.

[0118] After training, the second control panel image can be input into the region detection model, which outputs first identification information including at least one operation region. This first identification information can be a detection frame and / or the center position of the detection frame. It is understood that if the second control panel image includes multiple operation regions, the region detection model can simultaneously output the first identification information for each of the multiple operation regions.

[0119] The vision-language model (VLM) used in the present invention is a fine-tuned VLM. The input of the vision-language model can be information in text mode and image mode, such as first identification information in text mode and second control panel image in image mode.

[0120] The fine-tuned visual language model is used to perform content understanding and text generation tasks. The content understanding task can include understanding the control panel's operating logic, the function of each operating area, and the operating logic in the second control panel image. The text generation task can include generating textual descriptions of the operating areas, two-dimensional location information, and the type of the operating areas.

[0121] For example, for an operated object of the coffee machine type, after the second control panel image and the first identification information of the positioning operation area are input into the fine-tuned visual language model, text operation information as in the above embodiment can be generated.

[0122] In one embodiment, each operated object corresponds to a piece of multimodal operation information. After generating textual operation information for each operated object, the textual operation information and the second control panel image are combined into a data pair and stored in a database as the multimodal operation information. Thus, in response to and upon receipt of input task information, the multimodal operation information can be retrieved from the database based on the identification information of the operated object indicated in the task information.

[0123] In an embodiment of the present invention, by calling a region detection model, region detection is performed on the second control panel image to obtain first identification information of at least one operation area; calling a fine-tuned visual language model to generate text operation information based on the first identification information and the second control panel image; the text operation information and the second control panel image are combined into multimodal operation information and stored in a database, so that in the process of processing a task, there is no need to perform region detection on the second control panel image, and the multimodal operation information can be directly obtained, thereby ensuring that the task processing operation is simple and convenient.

[0124] According to an embodiment of the present invention, the visual language model is obtained by fine-tuning by the following method: calling the region detection model to perform region detection on the sample control panel image to obtain second identification information of at least one sample operation area; calling the multimodal large model to generate region description information based on the sample control panel image identified with the second identification information, wherein the region description information is used to describe the operation function of the sample operation area; calling the multimodal large model to generate sample text operation information based on the region description information, the second identification information and the sample control panel image; using multiple sample text operation information and multiple sample control panel images, the visual language model is fine-tuned to obtain a fine-tuned visual language model.

[0125] The sample control panel images can be images of control panels of various devices collected through the invention's channels. Using the trained region detection model, region detection can be performed on multiple sample control panel images to obtain second identification information for at least one sample operation region in each sample control panel image. The method for determining the second identification information is similar to that for obtaining the first identification information and will not be further described here.

[0126] A sample control panel image marked with the second identification information is input into the multimodal large model, and the multimodal large model is instructed by prompt words to generate a functional description of each operation area, thereby enabling the multimodal large model to output area description information of each operation area.

[0127] For example, the region description information can be information in the description field. For example, using the coffee machine described above, the region description information could be: When using this device, you can make different types of coffee by clicking different buttons on the control panel. The coffee machine's usage consists of three screens: the start page for selecting the type of coffee to be made, the detailed settings page for setting the sweetness, and the confirmation page for confirming the coffee. The current page is the start page.

[0128] After obtaining the region description information, if any region description information does not match the functional description of the control panel operation area, it can be readjusted through multiple interactions with the multimodal macro model until the correct region description information is generated. Alternatively, for a sample control panel image, region description information that does not match the functional description can be filtered out. The region description information, the second identification information, and the sample control panel image are then re-input into the multimodal macro model, and a prompt word is used to instruct the multimodal macro model to output sample text operation information in the same format as the above-mentioned text operation information.

[0129] It is understandable that the sample text operation information generated by the multimodal large model is also the sample data used to fine-tune the visual language model. In this embodiment, a small number of sample control panel images can be used to enable the multimodal large model to output sufficient sample data.

[0130] For multiple sample control panel images, multiple sample text operation information can be obtained. Specifically, the multiple sample text operation information can be used as examples to fine-tune the visual language model based on the multiple sample text operation information and the multiple sample control panel images, so that the fine-tuned visual language model can output correct text operation information.

[0131] In an embodiment of the present invention, the region detection model is called to perform region detection, and the multimodal large model is called to generate region description information; and sample text operation information is generated based on the region description information, the second identification information and the sample control panel image. A large number of sample text operation information with relatively accurate sample text operation information can be generated, thereby improving the accuracy of the visual language model after subsequent fine-tuning, thereby improving the accuracy of the generated text operation information and improving the accuracy of the robot's task processing.

[0132] Figure 6 FIG. 1 shows a schematic diagram of a scenario for generating an instruction sequence according to an embodiment of the present invention. Figure 6 As shown, in embodiment 600 of generating an instruction sequence, for a new operated object, second identification information 602 can be obtained by capturing a second control panel image P2 and performing region detection on the second control panel image P2 using a region detection model. The second control panel image P2 and second identification information 602 are input into the fine-tuned visual language model M2 to generate text operation information T1, which is then stored in a database together with the second control panel image P2. Thus, during the application phase, the robot can, in response to receiving task information T2 input by the user, retrieve multimodal operation information from the database, including text operation information T1 and second control panel image P2. The large language model M1 is then invoked to generate an operation sequence T3 based on the text operation information T1 and task information T2. ​​Furthermore, by performing feature registration (S607) on the first control panel image P1 and the second control panel image P2, a position transformation relationship can be obtained. Combining the operation sequence T3, the two-dimensional position information in the text operation information T1, and the depth information of the first control panel image P1, an instruction sequence 608 can be determined.

[0133] In the embodiments of the present invention, the corresponding operation arrangement and the positioning of the operation area of ​​each operation can be flexibly output for different control panels and different task information of users, so as to guide the robot to complete the operation of the control panel in steps, thereby realizing the processing of task information. For multiple task information and multiple control panels, it can ensure that the robot can well understand and accurately process the tasks corresponding to the task information, and the user experience is better.

[0134] Figure 7 FIG. 1 shows a block diagram of a task processing device for a robot according to an embodiment of the present invention. Figure 7 As shown, the task processing device 700 for a robot includes an acquisition module 710 , a calling module 720 , a generation module 730 and a control module 740 .

[0135] The acquisition module 710 is configured to, in response to receiving task information input by a user, acquire a first control panel image and multimodal operation information of an operated object indicated by the task information, wherein the multimodal operation information includes text operation information for operating the operated object.

[0136] The calling module 720 is used to call the large language model and generate an operation sequence according to the task information and the text operation information, wherein the operation sequence includes at least one operation area.

[0137] The generating module 730 is configured to generate an instruction sequence according to the operation sequence, the first control panel image, and the multimodal operation information, wherein the instruction sequence includes a control instruction corresponding to each operation area.

[0138] The control module 740 is used to control the robot to operate each operation area according to the instruction sequence to process the task corresponding to the task information.

[0139] According to an embodiment of the present invention, the calling module 720 includes: an acquisition submodule, an update submodule, and an output submodule. The acquisition submodule is used to obtain prompt information, wherein the prompt information includes an example operation sequence, at least one first vacant field corresponding to the operated object, and a second vacant field corresponding to the task information. The update submodule is used to fill the first vacant field and the second vacant field with the text operation information and the task information, respectively, to obtain updated prompt information. The output submodule is used to input the updated prompt information into the large language model and output an operation sequence consistent with the format of the example operation sequence.

[0140] According to an embodiment of the present invention, the update submodule includes a matching unit and a filling unit. The matching unit is configured to determine a target field value that matches each first missing field in the prompt information from the text operation information based on the field name of the first missing field. The filling unit is configured to fill the first missing field with the target field value and the second missing field with the task information to obtain the updated prompt information.

[0141] According to an embodiment of the present invention, the multimodal operation information also includes a pre-collected second control panel image, and the first control panel image is collected by the robot in response to receiving input task information. The generation module 730 includes: a position relationship determination submodule, a depth determination submodule, and a generation submodule. The position relationship determination submodule is used to determine the position transformation relationship based on the first control panel image and the second control panel image, wherein the position transformation relationship represents the position transformation relationship between the coordinate system of the second control panel image and the coordinate system of the first control panel image. The depth determination submodule is used to determine the depth information of each operation area based on the first control panel image. The generation submodule is used to generate control instructions based on the depth information, the position transformation relationship, and the text operation information.

[0142] According to an embodiment of the present invention, the position relationship determination submodule includes: a registration unit configured to perform feature registration on the first control panel image and the second control panel image to obtain at least one feature point pair; and a determination unit configured to determine a position transformation relationship based on the two-dimensional position information of each of the two feature points in the feature point pair.

[0143] According to an embodiment of the present invention, the text operation information includes two-dimensional position information of each operation area in the second control panel image.

[0144] The generation submodule includes a first determination unit, a second determination unit, and an instruction generation unit. For each operating area, the first determination unit is configured to determine the two-dimensional position information of the operating area in the robot coordinate system based on the two-dimensional position information and position transformation relationship of the operating area in the second control panel image. The second determination unit is configured to determine the three-dimensional position information of the operating area based on the depth information and the two-dimensional position information of the operating area in the robot coordinate system. The generation unit is configured to generate control instructions based on the three-dimensional position information.

[0145] According to an embodiment of the present invention, the task processing device 700 for a robot further includes: a region detection module, an operation information generation module, and a storage module. The region detection module is configured to invoke a region detection model to perform region detection on the second control panel image to obtain first identification information of at least one operation region. The operation information generation module is configured to invoke a fine-tuned visual language model to generate text operation information based on the first identification information and the second control panel image. The storage module is configured to combine the text operation information and the second control panel image into multimodal operation information and store it in a database, so that the multimodal operation information can be retrieved from the database in response to receiving input task information.

[0146] According to an embodiment of the present invention, the visual language model is obtained by fine-tuning in the following manner: calling a region detection model to perform region detection on a sample control panel image to obtain second identification information of at least one sample operation area; calling a multimodal large model to generate region description information based on the sample control panel image identified with the second identification information, wherein the region description information is used to describe the operation function of the sample operation area; calling a multimodal large model to generate sample text operation information based on the region description information, the second identification information and the sample control panel image; and using multiple sample text operation information and multiple sample control panel images to fine-tune the visual language model to obtain a fine-tuned visual language model.

[0147] Any number of the modules, submodules, units, and subunits according to embodiments of the present invention, or at least part of the functionality of any number of these units, can be implemented in a single module. Any one or more of the modules, submodules, units, and subunits according to embodiments of the present invention can be split into multiple modules for implementation. Any one or more of the modules, submodules, units, and subunits according to embodiments of the present invention can be at least partially implemented as a hardware circuit, such as a field programmable gate array (FPGA), a programmable logic array (PLA), a system on a chip, a system on a substrate, a system on a package, an application-specific integrated circuit (ASIC), or can be implemented in hardware or firmware using any other reasonable method of circuit integration or packaging, or can be implemented in any one of the three implementation methods of software, hardware, and firmware, or any appropriate combination of any of these. Alternatively, one or more of the modules, submodules, units, and subunits according to embodiments of the present invention can be at least partially implemented as a computer program module that, when executed, can perform the corresponding functionality.

[0148] For example, any number of the acquisition module 710, the call module 720, the generation module 730, and the control module 740 may be combined into a single module / unit / sub-unit, or any one of these modules / units / sub-units may be split into multiple modules / units / sub-units. Alternatively, at least part of the functionality of one or more of these modules / units / sub-units may be combined with at least part of the functionality of other modules / units / sub-units and implemented in a single module / unit / sub-unit. According to an embodiment of the present invention, at least one of the acquisition module 710, the call module 720, the generation module 730, and the control module 740 may be at least partially implemented as a hardware circuit, such as a field programmable gate array (FPGA), a programmable logic array (PLA), a system on a chip, a system on a substrate, a system on a package, an application specific integrated circuit (ASIC), or may be implemented in hardware or firmware by any other reasonable means of circuit integration or packaging, or may be implemented in any one of software, hardware, and firmware, or any appropriate combination of any of these. Alternatively, at least one of the acquisition module 710 , the calling module 720 , the generation module 730 and the control module 740 may be at least partially implemented as a computer program module, and when the computer program module is executed, the corresponding function may be performed.

[0149] It should be noted that the apparatus part in the embodiment of the present invention corresponds to the method part in the embodiment of the present invention. The description of the apparatus part specifically refers to the method part and will not be repeated here.

[0150] Figure 8 A block diagram of an electronic device suitable for implementing task processing for a robot according to an embodiment of the present invention is shown.

[0151] like Figure 8 As shown, an electronic device 800 according to an embodiment of the present invention includes a processor 801, which can perform various appropriate actions and processes based on a program stored in a read-only memory (ROM) 802 or a program loaded from a storage unit 808 into a random access memory (RAM) 803. The processor 801 may include, for example, a general-purpose microprocessor (e.g., a CPU), an instruction set processor and / or a related chipset and / or a special-purpose microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor 801 may also include onboard memory for caching purposes. The processor 801 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of the present invention.

[0152] Various programs and data required for the operation of the electronic device 800 are stored in the RAM 803. The processor 801, ROM 802, and RAM 803 are connected to each other via a bus 804. The processor 801 executes the programs in the ROM 802 and / or RAM 803 to perform various operations according to the method flow of the embodiment of the present invention. It should be noted that the programs may also be stored in one or more memories other than the ROM 802 and RAM 803. The processor 801 may also execute the programs stored in the one or more memories to perform various operations according to the method flow of the embodiment of the present invention.

[0153] According to an embodiment of the present invention, electronic device 800 may further include an input / output (I / O) interface 805, which is also connected to bus 804. Electronic device 800 may also include one or more of the following components connected to I / O interface 805: an input section 806 including a keyboard, mouse, etc.; an output section 807 including devices such as a cathode ray tube (CRT), liquid crystal display (LCD), and speakers; a storage section 808 including a hard disk; and a communication section 809 including a network interface card such as a LAN card or modem. Communication section 809 performs communication processing via a network such as the Internet. A drive 810 is also connected to I / O interface 805 as needed. Removable media 811, such as a magnetic disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed in drive 810 as needed, so that computer programs read from the removable media can be installed into storage section 808 as needed.

[0154] According to an embodiment of the present invention, the method flow according to an embodiment of the present invention can be implemented as a computer software program. For example, an embodiment of the present invention includes a computer program product, which includes a computer program carried on a computer-readable storage medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 809, and / or installed from the removable medium 811. When the computer program is executed by the processor 801, the above-mentioned functions defined in the system of the embodiment of the present invention are executed. According to an embodiment of the present invention, the system, device, apparatus, module, unit, etc. described above can be implemented by a computer program module.

[0155] The present invention also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments, or may exist independently and not incorporated into the device / apparatus / system. The computer-readable storage medium carries one or more programs, which, when executed, implement the method according to the embodiments of the present invention.

[0156] According to embodiments of the present invention, a computer-readable storage medium may be a non-volatile computer-readable storage medium. Examples include, but are not limited to, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present invention, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0157] For example, according to an embodiment of the present invention, the computer-readable storage medium may include the ROM 802 and / or the RAM 803 described above and / or one or more memories other than the ROM 802 and the RAM 803 .

[0158] An embodiment of the present invention also includes a computer program product, which includes a computer program, which contains program code for executing the method provided by the embodiment of the present invention. When the computer program product is run on an electronic device, the program code is used to enable the electronic device to implement the task processing method for a robot provided by the embodiment of the present invention.

[0159] When the computer program is executed by the processor 801, the above functions defined in the system / device of the embodiment of the present invention are performed. According to the embodiment of the present invention, the system, device, module, unit, etc. described above can be implemented by a computer program module.

[0160] In one embodiment, the computer program may be stored on a tangible storage medium such as an optical storage device or a magnetic storage device. In another embodiment, the computer program may be transmitted and distributed in the form of a signal on a network medium, downloaded and installed via the communication portion 809, and / or installed from a removable medium 811. The program code contained in the computer program may be transmitted using any appropriate network medium, including but not limited to wireless, wired, or any suitable combination thereof.

[0161] According to an embodiment of the present invention, the program code for executing the computer program provided by the embodiment of the present invention can be written in any combination of one or more programming languages. Specifically, these computer programs can be implemented using high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, languages ​​such as Java, C++, Python, "C" or similar programming languages. The program code can be executed entirely on the user computing device, partially on the user device, partially on a remote computing device, or entirely on a remote computing device or server. In the case of a remote computing device, the remote computing device can be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computing device (for example, using an Internet service provider to connect via the Internet).

[0162] The flowcharts and block diagrams in the accompanying drawings illustrate the possible architectures, functions, and operations of the systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each box in the flowchart or block diagram may represent a module, program segment, or portion of code, which contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the boxes may occur in an order different from that marked in the accompanying drawings. For example, two boxes shown in succession may actually be executed substantially in parallel, or they may sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flowchart, as well as the combination of boxes in the block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or may be implemented using a combination of dedicated hardware and computer instructions. It will be understood by those skilled in the art that the features described in the various embodiments of the present invention may be combined and / or coupled in various ways, even if such combinations or couplings are not explicitly described in the present invention. In particular, the features described in the various embodiments of the present invention may be combined and / or coupled in various ways without departing from the spirit and teachings of the present invention. All such combinations and / or combinations fall within the scope of the present invention.

[0163] The above describes embodiments of the present invention. However, these embodiments are for illustrative purposes only and are not intended to limit the scope of the present invention. Although each embodiment has been described separately above, this does not mean that the measures in each embodiment cannot be advantageously used in combination. Without departing from the scope of the present invention, those skilled in the art may make various substitutions and modifications, which should all fall within the scope of the present invention.

Claims

1. A task processing method for a robot, characterized in that: The method comprises: In response to receiving task information input by a user, acquiring a first control panel image and multimodal operation information of an operated object indicated by the task information, wherein the multimodal operation information includes text operation information for operating the operated object, and the text operation information is used to indicate operation logic for operating the control panel of the operated object; Invoking a large language model to generate an operation sequence based on the task information and the text operation information, wherein the operation sequence includes at least one operation area having an operation order, and each operation area corresponds to an operation on the control panel; generating an instruction sequence according to the operation sequence, the first control panel image, and the multimodal operation information, wherein the instruction sequence includes a control instruction corresponding to each of the operation areas; and The robot is controlled to operate each of the operation areas according to the instruction sequence to process the task corresponding to the task information.

2. The method according to claim 1, characterized in that The calling of the large language model to generate an operation sequence according to the task information and the text operation information includes: Acquiring prompt information, wherein the prompt information includes an example operation sequence, at least one first vacant field corresponding to the operated object, and a second vacant field corresponding to the task information; Filling the first vacant field and the second vacant field with the text operation information and the task information respectively to obtain updated prompt information; and The updated prompt information is input into the large language model, and the operation sequence consistent with the format of the example operation sequence is output.

3. The method according to claim 2, characterized in that The step of filling the first vacant field and the second vacant field with the text operation information and the task information to obtain updated prompt information includes: Determining, from the text operation information, a target field value matching each of the first missing fields according to the field name of the first missing field in the prompt information; and The target field value is filled into the first vacant field, and the task information is filled into the second vacant field to obtain the updated prompt information.

4. The method according to claim 1, wherein The multimodal operation information further includes a pre-collected second control panel image, wherein the first control panel image is collected by the robot in response to receiving the input task information; Generating an instruction sequence according to the operation sequence, the first control panel image, and the multimodal operation information includes: determining a position transformation relationship according to the first control panel image and the second control panel image, wherein the position transformation relationship represents a position transformation relationship between a coordinate system of the second control panel image and a coordinate system of the first control panel image; determining depth information of each of the operation areas according to the first control panel image; and The control instruction is generated according to the depth information, the position transformation relationship and the text operation information.

5. The method according to claim 4, characterized in that The determining of the position transformation relationship according to the first control panel image and the second control panel image includes: Performing feature registration on the first control panel image and the second control panel image to obtain at least one feature point pair; and The position transformation relationship is determined according to the two-dimensional position information of each of the two feature points in the feature point pair.

6. The method according to claim 4, characterized in that The text operation information includes two-dimensional position information of each operation area in the second control panel image; and generating the control instruction according to the depth information, the position transformation relationship, and the text operation information includes: For each of the operating areas, determining the two-dimensional position information of the operating area in the robot coordinate system according to the two-dimensional position information of the operating area in the second control panel image and the position transformation relationship; Determining three-dimensional position information of the operating area according to the depth information and the two-dimensional position information of the operating area in the robot coordinate system; and The control instruction is generated according to the three-dimensional position information.

7. The method according to any one of claims 4 to 6, characterized in that The method further comprises: Invoking a region detection model to perform region detection on the second control panel image to obtain first identification information of at least one operation region; Invoking the fine-tuned visual language model to generate the text operation information according to the first identification information and the second control panel image; and The text operation information and the second control panel image are combined into the multimodal operation information and stored in a database, so that the multimodal operation information is acquired from the database in response to receiving the input task information.

8. The method according to claim 7, characterized in that The visual language model is fine-tuned by the following method: Calling the region detection model to perform region detection on the sample control panel image to obtain second identification information of at least one sample operation region; Invoking the multimodal large model to generate region description information based on the sample control panel image marked with the second identification information, wherein the region description information is used to describe the operation function of the sample operation region; calling the multimodal large model to generate sample text operation information according to the area description information, the second identification information, and the sample control panel image; and The visual language model is fine-tuned using the plurality of sample text operation information and the plurality of sample control panel images to obtain the fine-tuned visual language model.

9. A task processing device for a robot, characterized in that: The device comprises: an acquisition module, configured to, in response to receiving task information input by a user, acquire a first control panel image of an operated object and multimodal operation information indicated by the task information, wherein the multimodal operation information includes text operation information for operating the operated object, and the text operation information is used to indicate operation logic for operating the control panel of the operated object; a calling module, configured to call a large language model and generate an operation sequence based on the task information and the text operation information, wherein the operation sequence includes at least one operation area having an operation order, and each operation area corresponds to an operation on the control panel; a generating module, configured to generate an instruction sequence according to the operation sequence, the first control panel image, and the multimodal operation information, wherein the instruction sequence includes a control instruction corresponding to each of the operation areas; and A control module is used to control the robot to operate each of the operating areas according to the instruction sequence to process the task corresponding to the task information.

10. An electronic device comprising: one or more processors; a memory for storing one or more programs, It is characterized in that when the one or more programs are executed by the one or more processors, the one or more processors are enabled to implement the method according to any one of claims 1 to 8.

11. A computer-readable storage medium having executable instructions stored thereon, characterized in that: When the instruction is executed by a processor, the processor implements the method according to any one of claims 1 to 8.

12. A computer program product comprising a computer program, characterized in that The computer program implements the method according to any one of claims 1 to 8 when executed by a processor.

13. A robot, characterized in that: Used to implement the method according to any one of claims 1 to 8.

Images