Three-dimensional scene graph updating method and robot control equipment

Abstract

The invention provides a three-dimensional scene graph updating method and robot control equipment, and relates to the technical field of robot control. According to the method and the device, instruction analysis is performed on the natural language operation instruction issued by the user by calling the large language model, so that the high-level language action instruction corresponding to the target robot needing to pass through the target observation point in the target scene is obtained, and in the process that the target robot moves according to the high-level language action instruction in the target scene, the high-level language action instruction of the target robot is obtained. The method comprises the following steps: acquiring an actual scene image shot by a target robot at a target observation point for a target storage structure, and then according to the acquired actual scene image, updating information of a placed article of a current target three-dimensional scene graph of a target scene at the target storage structure; the dynamic maintenance effect of the working scene information of the robot is achieved in a local scene information updating mode, and the change of the working scene of the robot can be dynamically sensed with low loss.

Description

Technical Field

[0001] The present application relates to the field of robot control technology, and in particular to a three-dimensional scene graph updating method and a robot control device. Background Art

[0002] With the continuous development of science and technology, robotics has been widely valued by all walks of life due to its great research and application value. Among them, users use natural language to drive robots to perform complex tasks through large language models (LLM), which is an important research direction in the field of robot control technology today. In the process of using large language models to drive robots to perform complex tasks, large language models usually parse the natural language instructions issued by users and generate complex instruction sequences for robot agents that they can understand and execute, so as to drive robots to perform complex tasks expected by users by driving them to execute complex instruction sequences.

[0003] At present, when a large language model is used to control a robot to perform complex tasks in a work scenario, the understanding and perception of the work scenario information depends on a pre-built three-dimensional scene graph, so as to use the three-dimensional scene graph of the work scenario to provide rich environmental cognition for the robot's task execution operation. However, it is worth noting that this pre-built three-dimensional scene graph is a static scene graph, which is difficult to reflect the dynamic scene changes of the work scenario, and it is very easy to cause the complex instruction sequence generated by the large language model for the robot to be incompatible with the real work scenario. The corresponding robot will not be able to perform the complex tasks expected by the user, unless a lot of time and energy are spent on periodically rescanning the work scene and rebuilding the three-dimensional scene graph to achieve the expected complex task execution effect. Summary of the invention

[0004] In view of this, the purpose of the present application is to provide a three-dimensional scene graph updating method and a robot control device, which can update the local scene information of the three-dimensional scene graph of the robot's working scene while using a large language model to drive the robot to perform the desired task, so as to achieve the dynamic maintenance effect of the robot's working scene information, without spending a lot of time and energy to periodically rescan the work scene and rebuild the three-dimensional scene graph, so as to effectively reduce the time cost loss and computing resource loss required for the robot's working scene change cognitive operation, and enable the corresponding large language model to dynamically perceive the robot's working scene information.

[0005] In order to achieve the above purpose, the technical solution adopted in the embodiment of the present application is as follows:

[0006] In a first aspect, the present application provides a three-dimensional scene graph updating method, the method comprising:

[0007] Obtaining a natural language operation instruction for a target robot, and calling a large language model to parse the natural language operation instruction, to obtain a high-level language action instruction for the target robot to pass through a target observation point in a target scene, wherein the target observation point is an optimal observation point of a target placement structure pointed to by the natural language operation instruction;

[0008] Driving the target robot to move in the target scene according to the high-level language action instructions, and acquiring an actual scene image captured by the target robot at the target observation point for the target placement structure;

[0009] The information of the placed items at the target placement structure is updated on the current target three-dimensional scene graph of the target scene according to the actual scene image.

[0010] In an optional implementation, the step of calling a large language model to parse the natural language operation instruction to obtain a high-level language action instruction for the target robot to pass through a target observation point in a target scene includes:

[0011] Calling the large language model to perform task decomposition on the natural language job instruction in the target scenario to obtain a plurality of continuous subtasks corresponding to the natural language job instruction;

[0012] For a first subtask among the multiple consecutive subtasks that matches the target placement structure, based on the position information of the target placement structure and the target observation point in the target scene, a pre-stored high-level language movement control function is called to generate a first high-level language subtask instruction for moving to the target placement structure via the target observation point;

[0013] For each second subtask except the first subtask in the plurality of consecutive subtasks, based on the association relationship between the actual subtask content of the second subtask and the content of the target three-dimensional scene graph, calling an adapted high-level language movement control function to generate a corresponding second high-level language subtask instruction;

[0014] The first high-level language subtask instruction and all second high-level language subtask instructions are sequenced and aggregated to obtain the high-level language action instruction.

[0015] In an optional implementation, the step of updating the information of the placed items at the target placement structure of the current target three-dimensional scene graph of the target scene according to the actual scene image includes:

[0016] Acquire a reference scene image of the target placement structure adapted to the target three-dimensional scene graph at the target observation point;

[0017] Calculating the image similarity between the reference scene image and the actual scene image;

[0018] Detecting whether the image similarity is less than a preset similarity threshold;

[0019] When it is detected that the image similarity is less than a preset similarity threshold, the information of the placed objects located at the target placement structure in the target three-dimensional scene graph is updated according to the actual scene image.

[0020] In an optional implementation, the step of calculating the image similarity between the reference scene image and the actual scene image includes:

[0021] Mapping the reference scene image and the actual scene image into an embedding space respectively, to obtain a first embedding vector corresponding to the reference scene image and a second embedding vector corresponding to the actual scene image;

[0022] The cosine similarity between the first embedding vector and the second embedding vector is calculated, and the calculated cosine similarity is used as the image similarity.

[0023] In an optional implementation, the step of updating the information of the placed objects located at the target placement structure in the target three-dimensional scene graph according to the actual scene image includes:

[0024] Performing object recognition on the actual scene image to determine all candidate objects in the actual scene image that are located close to the target storage structure;

[0025] According to the structural information of the target placement structure, performing placement rationality analysis on all the determined candidate objects, so as to screen out all the target objects actually placed in the target placement structure from all the candidate objects;

[0026] For each target object, based on the actual scene image, object attribute information adapted to the target three-dimensional scene graph format is generated, and information correction is performed on the generated object attribute information to obtain actual object information of the target object;

[0027] According to the actual item information of all target items, the information of the placed items located at the target placement structure in the target three-dimensional scene graph is replaced.

[0028] In an optional embodiment, the method further comprises:

[0029] For each placement structure in the target scene, determining all traversable observation points of the target robot with respect to the placement structure in an original three-dimensional scene graph of the target scene;

[0030] Obtaining original scene images of the placement structure observed by the target robot at each passable observation point in the original three-dimensional scene graph, and obtaining information of all placed items of the placement structure at the original three-dimensional scene graph;

[0031] According to all the information of the placed items corresponding to the placement structure and the original scene images of all the passable observation points, the best observation point of the placement structure is determined among all the passable observation points.

[0032] In an optional embodiment, for each placement structure in the target scene, the step of determining all traversable observation points of the target robot regarding the placement structure in the original three-dimensional scene graph of the target scene includes:

[0033] In the original three-dimensional scene graph, based on the safe moving distance of the target robot, an observation area at least covering the placement structure is delineated;

[0034] A plurality of initial observation points are evenly generated in the observation area, and invalid observation points that the target robot cannot reach are eliminated from the plurality of initial observation points to obtain all passable observation points of the target robot on the storage structure.

[0035] In an optional embodiment, the step of determining the best observation point of the storage structure from among all passable observation points based on all information of placed items corresponding to the storage structure and respective original scene images of all passable observation points comprises:

[0036] Calculating the cosine similarity between the original scene images of all the passable observation points and the information of all the placed objects;

[0037] Calculate the observation value of each of the passable observation points for the placement structure according to the observation distances of each of the passable observation points to the placement structure in the original three-dimensional scene graph and the cosine similarities corresponding to each of the passable observation points;

[0038] From all the passable observation points, select the passable observation point with the largest corresponding observation value as the best observation point of the placement structure.

[0039] In an optional implementation, for a single storage structure, the observation value of the i-th passable observation point corresponding to the storage structure is calculated using the following formula:

[0040]

[0041] Among them, Val iIt is used to indicate the observation value of the i-th passable observation point for the placement structure, d max It is used to represent the maximum observation distance among the observation distances of all the passable observation points corresponding to the placement structure, d i It is used to represent the observation distance of the i-th passable observation point corresponding to the placement structure, CS i It is used to represent the cosine similarity of the i-th traversable observation point corresponding to the placement structure.

[0042] In a second aspect, the present application provides a robot control device, comprising a processor and a memory, wherein the memory stores a computer program that can be executed by the processor, and the processor can execute the computer program to drive a target robot to cooperate in implementing the three-dimensional scene graph updating method described in any one of the aforementioned embodiments.

[0043] In this case, the beneficial effects of the embodiments of the present application may include the following:

[0044] The present application calls a large language model to parse the natural language operation instructions issued by the user, obtains the high-level language action instructions of the corresponding target robot that needs to pass through the target observation point in the target scene, and obtains the actual scene image taken by the target robot at the target observation point for the target storage structure during the movement of the target robot in accordance with the high-level language action instructions in the target scene, and then updates the information of the placed items at the target storage structure of the current target three-dimensional scene graph of the target scene according to the actual scene image obtained, so that in the process of using the large language model to drive the robot to perform the desired task, the three-dimensional scene graph of the robot's working scene is locally updated to achieve the dynamic maintenance effect of the robot's working scene information, without consuming a lot of time and energy to periodically rescan the working scene and rebuild the three-dimensional scene graph, so as to effectively reduce the time cost loss and computing resource loss required for the robot's working scene change cognitive operation, and enable the corresponding large language model to dynamically perceive the robot's working scene information.

[0045] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, preferred embodiments are specifically cited below and described in detail with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0047] Figure 1 A schematic diagram of the composition of a robot control device provided in an embodiment of the present application;

[0048] Figure 2 One of the flowcharts of the three-dimensional scene graph updating method provided in the embodiment of the present application;

[0049] Figure 3 for Figure 2 A schematic flow chart of the sub-steps included in step S210;

[0050] Figure 4 for Figure 2 A schematic flow chart of the sub-steps included in step S230;

[0051] Figure 5 The second flowchart of the three-dimensional scene graph updating method provided in the embodiment of the present application.

[0052] Icons: 10 - robot control device; 11 - memory; 12 - processor; 13 - communication unit. DETAILED DESCRIPTION

[0053] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. The components of the embodiments of the present application described and shown in the drawings here can be arranged and designed in various different configurations.

[0054] Therefore, the following detailed description of the embodiments of the present application provided in the accompanying drawings is not intended to limit the scope of the present application for which protection is sought, but merely represents selected embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without creative work are within the scope of protection of the present application.

[0055] It should be noted that similar reference numerals and letters denote similar items in the following drawings, and therefore, once an item is defined in one drawing, further definition and explanation thereof is not required in subsequent drawings.

[0056] In the description of the present application, it should be understood that the terms "center", "up", "down", "left", "right", "vertical", "horizontal", "inside", "outside", etc. indicate orientations or positional relationships based on the orientations or positional relationships shown in the accompanying drawings, or are the orientations or positional relationships in which the product of the application is conventionally placed when in use, or are the orientations or positional relationships conventionally understood by those skilled in the art. They are only for the convenience of describing the present application and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore should not be understood as a limitation on the present application.

[0057] In the description of this application, it should also be noted that, unless otherwise clearly specified and limited, the terms "set", "install", "connect", and "connect" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be indirectly connected through an intermediate medium, or it can be the internal communication of two elements. For ordinary technicians in this field, the specific meanings of the above terms in this application can be understood according to specific circumstances.

[0058] In addition, in the description of the present application, it is understood that the relational terms such as the term "first" and "second" are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply that there is any such actual relationship or order between these entities or operations. Moreover, the term "include", "comprise" or any other variant thereof is intended to cover non-exclusive inclusion, so that the process, method, article or equipment including a series of elements includes not only those elements, but also includes other elements that are not clearly listed, or also includes elements inherent to such process, method, article or equipment. In the absence of more restrictions, the elements defined by the sentence "comprise one..." do not exclude the existence of other identical elements in the process, method, article or equipment including the elements. For those of ordinary skill in the art, the specific meanings of the above terms in the present application can be understood by specific circumstances.

[0059] In conjunction with the accompanying drawings, some embodiments of the present application are described in detail below. In the absence of conflict, the following embodiments and features in the embodiments can be combined with each other.

[0060] Please refer to Figure 1 , Figure 1: is a schematic diagram of the composition of the robot control device 10 provided in the embodiment of the present application. In the embodiment of the present application, the robot control device 10 can be connected to the controlled robot for communication, and respond to the natural language operation instructions issued by the user based on the large language model loaded by itself, and control the controlled robot to perform the desired task represented by the natural language operation instructions in the current working scene. Among them, the robot control device 10 can be a computer device independent of the controlled robot, and the computer device can be, but not limited to, a personal computer, a laptop computer, a tablet computer, a server, etc.; the robot control device 10 can also be integrated with the controlled robot.

[0061] In the embodiment of the present application, the robot control device 10 may include a memory 11, a processor 12, and a communication unit 13. The memory 11, the processor 12, and the communication unit 13 are electrically connected to each other directly or indirectly to achieve data transmission or interaction. For example, the memory 11, the processor 12, and the communication unit 13 may be electrically connected to each other via one or more communication buses or signal lines.

[0062] In this embodiment, the memory 11 may be, but not limited to, a random access memory (RAM), a read only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable read-only memory (EEPROM), etc. The memory 11 is used to store a computer program, and the processor 12 may execute the computer program accordingly after receiving an execution instruction.

[0063] At the same time, the memory 11 can also be used to store a large language model, which can be loaded with a three-dimensional scene graph of the robot working scene where the controlled robot is currently located (for example, the original three-dimensional scene graph initially constructed for the robot working scene), the physical hardware information of the controlled robot, and multiple high-level language mobile control functions related to the robot task execution operation. Among them, the physical hardware information includes the specific body structure information of the controlled robot (for example, joint type, mobile wheel type, body height information, etc.); the high-level language mobile control function is a program code described in a high-level language (for example, python language, Java language or PHP language, etc.) for realizing a certain mobile control function of the controlled robot (for example, grabbing action function, putting down action function, forward action function, etc.), and the multiple high-level language mobile control functions may include getUserPosition() function, getPosition(str) function, goto(pos) function, take() function, put() function, moveAhead(length) function, moveBack(length) function, turnRight(angel) function and turnLeft(angle) function, etc., Among them, the getUserPosition() function is used to obtain the current robot position of the controlled robot, the getPosition(str) function is used to obtain the actual position of the str object, the goto(pos) function is used to drive the controlled robot to move to the position corresponding to pos, the take() function is used to drive the controlled robot to grab the object at the current position, the put() function is used to drive the controlled robot to put down the object being grabbed at the current position, the moveAhead(length) function is used to drive the controlled robot to move forward a distance of length, the moveBack(length) function is used to drive the controlled robot to move back a distance of length, the turnRight(angel) function is used to drive the controlled robot to turn right the angle of angel, and the turnLeft(angle) function is used to drive the controlled robot to turn left the angle of angel.

[0064] In this embodiment, the processor 12 may be an integrated circuit chip with signal processing capability. The processor 12 may be a general-purpose processor, including a central processing unit (CPU), a graphics processing unit (GPU), a network processor (NP), a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), or at least one of other programmable logic devices, discrete gate or transistor logic devices, and discrete hardware components. A general-purpose processor may be a microprocessor or the processor may be any conventional processor, etc., which may implement or execute the disclosed methods, steps, and logic block diagrams in the embodiments of the present application.

[0065] In this embodiment, the communication unit 13 is used to establish a communication connection between the robot control device 10 and other electronic devices through a network, and to send and receive data through the network, wherein the network includes a wired communication network and a wireless communication network. For example, the robot control device 10 can obtain a natural language operation instruction issued by a user through a terminal device through the communication unit 13, wherein the natural language operation instruction can be issued to the terminal device in a voice input mode or a text input mode, wherein the terminal device can be, but not limited to, a smart phone, a personal computer, a tablet computer, etc.

[0066] In this embodiment, the robot control device 10 may pre-store a specific computer program related to the three-dimensional scene graph updating function in the memory 11, and by driving the processor 12 to execute the specific computer program accordingly, while calling the large language model to drive the controlled robot to perform the desired task, the three-dimensional scene graph of the robot's working scene is locally updated to achieve the dynamic maintenance effect of the robot's working scene information, without spending a lot of time and energy to periodically rescan the work scene and rebuild the three-dimensional scene graph, so as to effectively reduce the time cost loss and computing resource loss required for the robot's working scene change cognitive operation, and facilitate low-loss to ensure that the large language model dynamically perceives the robot's working scene changes.

[0067] Understandably, Figure 1 The block diagram shown is only a schematic diagram of a composition of the robot control device 10. The robot control device 10 may also include Figure 1 More or fewer components as shown, or with Figure 1 Different configurations shown. Figure 1 Each component shown in the figure can be implemented by hardware, software or a combination thereof.

[0068] In the present application, in order to ensure that the robot control device 10 can update the local scene information of the three-dimensional scene graph of the robot's working scene while using a large language model to drive the robot to perform the desired task, so as to achieve dynamic maintenance of the robot's working scene information and facilitate low-loss dynamic perception of changes in the robot's working scene, an embodiment of the present application provides a three-dimensional scene graph updating method to achieve the aforementioned purpose. The three-dimensional scene graph updating method provided by the present application is described in detail below.

[0069] Please refer to Figure 2 , Figure 2 is one of the flow charts of the three-dimensional scene graph updating method provided in the embodiment of the present application. In the embodiment of the present application, Figure 2 The illustrated three-dimensional scene graph updating method may include steps S210 to S230.

[0070] Step S210, obtain natural language operation instructions for the target robot, and call the large language model to parse the natural language operation instructions to obtain high-level language action instructions for the target robot to pass through the target observation point in the target scene.

[0071] In this embodiment, the target robot is a controlled robot that can be currently controlled by the robot control device 10; after obtaining the natural language operation instructions issued by the user to the target robot, the robot control device 10 can perform language recognition on the natural language operation instructions to obtain the target operation instruction content in text form, and then input the target operation instruction content into the large language model to utilize the text recognition ability and robot motion planning ability of the large language model to construct a high-level language action instruction that matches the natural language operation instruction based on the current target three-dimensional scene graph of the target scene (that is, the robot working scene where the target robot is located) (that is, the actual three-dimensional scene graph of the target scene before receiving the natural language operation instruction). Among them, the high-level language action instruction relates to the target storage structure pointed to by the natural language operation instruction, the target storage structure is statically set in the target scene, and any storage structure in the target scene is used to place movable items, wherein the storage structure can be but not limited to: table, cabinet, dressing table, washbasin, etc.; the high-level language action instruction can drive the target robot to perform the desired task represented by the natural language operation instruction through the target observation point (i.e., the best observation point of the target storage structure) in the target scene, so that the target robot takes a scene image of the target storage structure when moving to the target observation point, and obtains an actual scene image that can represent the actual scene condition around the target storage structure. In one implementation of this embodiment, the target storage structure is in the middle position of the actual scene image.

[0072] Optionally, see Figure 3 , Figure 3 yes Figure 2 In the embodiment of the present application, the step S210 may include sub-steps S211 to S214 to ensure that the corresponding generated high-level language action instructions can drive the target robot to perform the user's desired task through the target observation point in the target scene.

[0073] Sub-step S211, calling the large language model to perform task decomposition on the natural language job instructions in the target scenario, and obtaining multiple continuous sub-tasks corresponding to the natural language job instructions.

[0074] In this embodiment, the large language model can determine the multiple robot decomposition actions that need to be continuously implemented in the target scene based on the target three-dimensional scene graph of the target scene, so that each robot decomposition action corresponds to a continuous subtask. For example, for the natural language operation instruction "Go to the study to help me get the water bottle on the desk", it can be disassembled into continuous subtasks "move to the study", "move to the desk", "grab the water bottle on the desk", "move outside the study", "move to the user" and "put down the grabbed water bottle" that are distributed in sequence.

[0075] Sub-step S212, for the first subtask matching the target storage structure among the multiple consecutive subtasks, based on the position information of the target storage structure and the target observation point in the target scene, a pre-stored high-level language movement control function is called to generate a first high-level language subtask instruction for going to the target storage structure via the target observation point.

[0076] In this embodiment, the large language model can perform task content recognition on the first subtask to extract a first high-level language movement control function adapted to the first subtask from a plurality of pre-stored high-level language movement control functions, and then call the first high-level language movement control function to generate a matching first high-level language subtask instruction according to the position information of the target storage structure and the target observation point in the target scene. For example, for the first subtask "move to the desk", the matching first high-level language subtask instruction can be described as "goto (target observation point)" and "goto (desk)".

[0077] Sub-step S213, for each second subtask except the first subtask in the multiple consecutive subtasks, based on the actual subtask content of the second subtask and the content association relationship of the target three-dimensional scene graph, calling the adapted high-level language movement control function to generate a corresponding second high-level language subtask instruction.

[0078] In this embodiment, the content association relationship can be used to represent the mapping relationship between the actual subtask content corresponding to the second subtask and the various objects existing in the target three-dimensional scene graph; for each second subtask, the large language model can perform task content identification on the second subtask to extract a second high-level language movement control function that is adapted to the second subtask from a plurality of pre-stored high-level language movement control functions, and then call the second high-level language movement control function to generate a matching second high-level language subtask instruction based on the content association relationship corresponding to the second subtask.

[0079] Sub-step S214, performing instruction sorting and aggregation on the first high-level language sub-task instruction and all second high-level language sub-task instructions to obtain a high-level language action instruction.

[0080] In this embodiment, the large language model can perform instruction sorting and instruction aggregation processing on the first high-level language subtask instructions and all second high-level language subtasks obtained according to the subtask sorting status of the first subtask and all second subtasks at the natural language job instructions, so as to obtain high-level language action instructions adapted to the natural language job instructions.

[0081] Therefore, the present application can ensure that the corresponding generated high-level language action instructions can drive the target robot to perform the user's desired tasks through the target observation point in the target scene by executing the above sub-steps S211 to S214.

[0082] Step S220, driving the target robot to move in the target scene according to the high-level language action instruction, and obtaining the actual scene image captured by the target robot at the target observation point for the target storage structure.

[0083] In one implementation of this embodiment, for the first high-level language subtask instruction included in the high-level language action instruction, a high-level language instruction content "collect(sceneinfo)" can be added at the target observation point to instruct the target robot to take scene images of the target storage structure at the target observation point through the high-level language instruction content, thereby effectively obtaining the actual local scene information (i.e., the actual scene image) corresponding to the target storage structure in the robot's working scene. At this time, the first high-level language subtask instruction can be expressed as "goto(target observation point)", "collect(scene info)" and "goto(desk)" in sequence.

[0084] Step S230: updating the information of the placed objects at the target placement structure of the current target three-dimensional scene graph of the target scene according to the actual scene image.

[0085] In this embodiment, after acquiring the actual scene image of the target storage structure at the target observation point, the robot control device 10 can extract the actual item information (including the item type, item placement position, item functional attribute, item physical attribute, item shape, etc. of the corresponding target item) of each of the target items actually placed in the target storage structure from the actual scene image according to the characteristic that the large language model tends to convert the three-dimensional scene graph into a topological structure format composed of nodes and edges for recording, and then use the extracted actual item information of each of the target items to replace the information of the placed items at the target storage structure of the target three-dimensional scene graph to complete the local scene information update operation of the three-dimensional scene graph of the robot working scene, so as to achieve the dynamic maintenance effect of the robot working scene information, without consuming a lot of time and energy to periodically rescan the working scene and rebuild the three-dimensional scene graph, which can effectively reduce the time cost loss and computing resource loss required for the robot working scene change cognitive operation, so that the corresponding large language model can dynamically perceive the robot working scene change with low loss.

[0086] In one implementation of the present embodiment, when the robot control device 10 performs a local scene information update operation for the robot working scene, it is not necessary to consider whether the scene condition corresponding to the target placement structure in the target three-dimensional scene graph has changed, and directly uses the actual scene image captured by the target robot at the target observation point for the target placement structure to update the information of the placed objects in the target three-dimensional scene graph at the target placement structure.

[0087] Optionally, see Figure 4 , Figure 4 yes Figure 2 In another implementation of this embodiment, in order to avoid performing meaningless local scene information update operations and ensure that the robot control device 10 performs local scene information update operations only when the scene around the target storage structure changes drastically, the step S230 may include sub-steps S231 to S234.

[0088] Sub-step S231, obtaining a reference scene image of the target object structure at the target observation point adapted to the target three-dimensional scene graph.

[0089] In this embodiment, the reference scene image is a valid scene image obtained by observing the target object structure at the target observation point in the target three-dimensional scene graph.

[0090] Sub-step S232, calculating the image similarity between the reference scene image and the actual scene image.

[0091] The step of calculating the image similarity between the reference scene image and the actual scene image may include:

[0092] Mapping the reference scene image and the actual scene image into an embedding space respectively, to obtain a first embedding vector corresponding to the reference scene image and a second embedding vector corresponding to the actual scene image;

[0093] The cosine similarity between the first embedding vector and the second embedding vector is calculated, and the calculated cosine similarity is used as the image similarity.

[0094] Sub-step S233, detecting whether the image similarity is less than a preset similarity threshold.

[0095] In this embodiment, when it is detected that the image similarity between the reference scene image and the actual scene image is less than the preset similarity threshold, it indicates that the scene around the target storage structure has undergone drastic changes. At this time, the local scene information update operation can be performed by executing sub-step S234; and when it is detected that the image similarity between the reference scene image and the actual scene image is greater than or equal to the preset similarity threshold, it indicates that the scene around the target storage structure has not changed much, and there is no need to perform a local scene information update operation on the target three-dimensional scene graph.

[0096] Sub-step S234, when it is detected that the image similarity is less than a preset similarity threshold, the information of the placed objects located at the target placement structure in the target three-dimensional scene graph is updated according to the actual scene image.

[0097] In this embodiment, the step of "updating the information of the placed objects located at the target placement structure in the target three-dimensional scene graph according to the actual scene image" in the sub-step S234 may include sub-steps A to D.

[0098] Sub-step A: performing object recognition on the actual scene image to determine all candidate objects in the actual scene image that are located close to the target storage structure.

[0099] Sub-step B: performing placement rationality analysis on all determined candidate objects according to the structural information of the target placement structure, so as to screen out all target objects actually placed in the target placement structure from all candidate objects.

[0100] The placement rationality analysis operation for any candidate item may include, but is not limited to: item location rationality analysis operation, item semantic rationality analysis operation, item quantity rationality analysis operation, and the like.

[0101] Sub-step C: for each target object, generate object attribute information adapted to the target three-dimensional scene graph format based on the actual scene image, and perform information correction on the generated object attribute information to obtain the actual object information of the target object.

[0102] Among them, the item attribute information may include item type information, item placement location information, item function attribute information, item physical attribute information, item shape information, etc. of the corresponding target item; the actual scene image can be detected using a visual language model designed based on prompts to obtain semantic text information of the actual scene image, and then the item attribute information can be verified and corrected using the semantic text information to obtain the actual item information of the corresponding target item.

[0103] Sub-step D: According to the actual item information of all target items, information of the items placed at the target placement structure in the target three-dimensional scene graph is replaced.

[0104] Therefore, the present application can avoid executing meaningless local scene information update operations by executing the above sub-steps S231 to S234, and ensure that the robot control device 10 only executes local scene information update operations when the scene around the target storage structure changes drastically.

[0105] The present application can ensure that the robot control device 10 can update the local scene information of the three-dimensional scene graph of the robot's working scene while using the large language model to drive the robot to perform the desired task by executing the above steps S210 to S230, thereby achieving a dynamic maintenance effect on the robot's working scene information and facilitating low-loss dynamic perception of changes in the robot's working scene.

[0106] Optionally, see Figure 5 , Figure 5 This is the second flow chart of the three-dimensional scene graph updating method provided in the embodiment of the present application. Figure 2 Compared with the 3D scene graph update method shown in Figure 5 The three-dimensional scene graph updating method shown may further include steps S240 to S260 to effectively determine the best observation points corresponding to each existing storage structure of the target robot in the target scene.

[0107] Step S240: for each storage structure in the target scene, determine all traversable observation points of the target robot with respect to the storage structure in the original three-dimensional scene graph of the target scene.

[0108] In this embodiment, for each placement structure in the target scene, the step of "determining all traversable observation points of the target robot about the placement structure in the original three-dimensional scene graph of the target scene" in step S240 may include:

[0109] In the original three-dimensional scene graph, based on the safe moving distance of the target robot, an observation area at least covering the placement structure is delineated;

[0110] A plurality of initial observation points are evenly generated in the observation area, and invalid observation points that the target robot cannot reach are eliminated from the plurality of initial observation points to obtain all passable observation points of the target robot on the storage structure.

[0111] Among them, the observation area can be a circular area, the center position of the observation area overlaps with the calibration point position of the corresponding placement structure (for example, the geometric center position), and the observation radius of the observation area can be twice the safe moving distance; the invalid observation points that the target robot cannot reach can be determined based on the distribution position information of static objects in the original three-dimensional scene graph (for example, the placement structure and the wall structure around the location object structure).

[0112] Step S250, obtaining the original scene image of the placement structure observed by the target robot at each passable observation point in the original three-dimensional scene graph, and obtaining information of all placed items of the placement structure at the original three-dimensional scene graph.

[0113] In this embodiment, for any traversable observation point of a certain placement structure in the original three-dimensional scene graph, the image shooting direction of the placement structure observed from the traversable observation point can be determined according to the calibration point position of the placement structure and the actual observation point position of the traversable observation point, and then based on the determined image shooting direction, the original scene image of the placement structure is extracted in the original three-dimensional scene graph, wherein the placement structure is located in the middle position of the corresponding original scene image.

[0114] For any placement structure, all information about items placed in the placement structure in the original three-dimensional scene graph includes information about item type, item placement location, item functional attributes, item physical attributes, item shape, etc. of the corresponding items.

[0115] Step S260, determining the best observation point of the storage structure from among all the passable observation points according to all the information of the placed items corresponding to the storage structure and the original scene images of all the passable observation points.

[0116] In this embodiment, for any storage structure, the step of "determining the best observation point of the storage structure from all passable observation points according to all placed item information corresponding to the storage structure and the original scene images of all passable observation points" may include:

[0117] Calculating the cosine similarity between the original scene images of all the passable observation points and the information of all the placed objects;

[0118] Calculate the observation value of each of the passable observation points for the placement structure according to the observation distances of each of the passable observation points to the placement structure in the original three-dimensional scene graph and the cosine similarities corresponding to each of the passable observation points;

[0119] From all the passable observation points, select the passable observation point with the largest corresponding observation value as the best observation point of the placement structure.

[0120] Among them, for a single storage structure, the observation value of the i-th passable observation point corresponding to the storage structure is calculated using the following formula:

[0121]

[0122] Among them, Val i It is used to indicate the observation value of the i-th passable observation point for the placement structure, d max It is used to represent the maximum observation distance among the observation distances of all the passable observation points corresponding to the placement structure, d i It is used to represent the observation distance of the i-th passable observation point corresponding to the placement structure, CS i It is used to represent the cosine similarity of the i-th traversable observation point corresponding to the placement structure.

[0123] Therefore, the present application can effectively determine the optimal observation points corresponding to each existing storage structure of the target robot in the target scene by executing the above steps S240 to S260.

[0124] In the embodiments provided in the present application, it should be understood that the disclosed devices and methods can also be implemented in other ways. The device embodiments described above are merely schematic, for example, the flowcharts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the devices, methods and computer program products according to the embodiments of the present application. In this regard, each box in the flowchart or block diagram can represent a module, a program segment or a part of a code, and a module, a program segment or a part of a code contains one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or the flowchart, and the combination of boxes in the block diagram and / or the flowchart can be implemented with a dedicated hardware-based system that performs a specified function or action, or can be implemented with a combination of dedicated hardware and computer instructions.

[0125] In addition, each functional module in each embodiment of the present application can be integrated together to form an independent part, or each module can exist separately, or two or more modules can be integrated to form an independent part. If the various functions provided by the present application are implemented in the form of software functional modules and sold or used as independent products, they can be stored in a storage medium. Based on such an understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or the part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including several instructions for the robot control device 10 to perform all or part of the steps of the method recorded in each embodiment of the present application. The aforementioned readable storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or CD-ROM and other media that can store program codes.

[0126] The above are only various implementations of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A three-dimensional scene graph updating method, characterized in that: The method comprises: Obtaining a natural language operation instruction for a target robot, and calling a large language model to parse the natural language operation instruction, to obtain a high-level language action instruction for the target robot to pass through a target observation point in a target scene, wherein the target observation point is an optimal observation point of a target placement structure pointed to by the natural language operation instruction; Driving the target robot to move in the target scene according to the high-level language action instructions, and acquiring an actual scene image captured by the target robot at the target observation point for the target placement structure; The information of the placed items at the target placement structure is updated on the current target three-dimensional scene graph of the target scene according to the actual scene image.

2. The method according to claim 1, characterized in that The step of calling the large language model to parse the natural language operation instruction to obtain the high-level language action instruction of the target robot that needs to pass through the target observation point in the target scene includes: Calling the large language model to perform task decomposition on the natural language job instruction in the target scenario to obtain a plurality of continuous subtasks corresponding to the natural language job instruction; For a first subtask among the multiple consecutive subtasks that matches the target placement structure, based on the position information of the target placement structure and the target observation point in the target scene, a pre-stored high-level language movement control function is called to generate a first high-level language subtask instruction for moving to the target placement structure via the target observation point; For each second subtask except the first subtask in the plurality of consecutive subtasks, based on the association relationship between the actual subtask content of the second subtask and the content of the target three-dimensional scene graph, calling an adapted high-level language movement control function to generate a corresponding second high-level language subtask instruction; The first high-level language subtask instruction and all second high-level language subtask instructions are sequenced and aggregated to obtain the high-level language action instruction.

3. The method according to claim 1, characterized in that The step of updating the information of the placed items at the target placement structure of the current target three-dimensional scene graph of the target scene according to the actual scene image comprises: Acquire a reference scene image of the target placement structure adapted to the target three-dimensional scene graph at the target observation point; Calculating the image similarity between the reference scene image and the actual scene image; Detecting whether the image similarity is less than a preset similarity threshold; When it is detected that the image similarity is less than a preset similarity threshold, the information of the placed objects located at the target placement structure in the target three-dimensional scene graph is updated according to the actual scene image.

4. The method according to claim 3, characterized in that The step of calculating the image similarity between the reference scene image and the actual scene image comprises: Mapping the reference scene image and the actual scene image into an embedding space respectively, to obtain a first embedding vector corresponding to the reference scene image and a second embedding vector corresponding to the actual scene image; The cosine similarity between the first embedding vector and the second embedding vector is calculated, and the calculated cosine similarity is used as the image similarity.

5. The method according to claim 3, characterized in that: The step of updating the information of the placed objects located at the target placement structure in the target three-dimensional scene graph according to the actual scene image comprises: Performing object recognition on the actual scene image to determine all candidate objects in the actual scene image that are located close to the target storage structure; According to the structural information of the target placement structure, performing placement rationality analysis on all the determined candidate objects, so as to screen out all the target objects actually placed in the target placement structure from all the candidate objects; For each target object, based on the actual scene image, object attribute information adapted to the target three-dimensional scene graph format is generated, and information correction is performed on the generated object attribute information to obtain actual object information of the target object; According to the actual item information of all target items, the information of the placed items located at the target placement structure in the target three-dimensional scene graph is replaced.

6. The method according to any one of claims 1 to 5, characterized in that: The method further comprises: For each placement structure in the target scene, determining all traversable observation points of the target robot with respect to the placement structure in an original three-dimensional scene graph of the target scene; Obtaining original scene images of the placement structure observed by the target robot at each passable observation point in the original three-dimensional scene graph, and obtaining information of all placed items of the placement structure at the original three-dimensional scene graph; According to all the information of the placed items corresponding to the placement structure and the original scene images of all the passable observation points, the best observation point of the placement structure is determined among all the passable observation points.

7. The method according to claim 6, characterized in that For each placement structure in the target scene, the step of determining all traversable observation points of the target robot with respect to the placement structure in the original three-dimensional scene graph of the target scene includes: In the original three-dimensional scene graph, based on the safe moving distance of the target robot, an observation area at least covering the placement structure is delineated; A plurality of initial observation points are evenly generated in the observation area, and invalid observation points that the target robot cannot reach are eliminated from the plurality of initial observation points to obtain all passable observation points of the target robot on the storage structure.

8. The method according to claim 6, characterized in that The step of determining the best observation point of the storage structure from among all passable observation points according to all the information of the placed items corresponding to the storage structure and the original scene images of all the passable observation points comprises: Calculating the cosine similarity between the original scene images of all the passable observation points and the information of all the placed objects; Calculate the observation value of each of the passable observation points for the placement structure according to the observation distances of each of the passable observation points to the placement structure in the original three-dimensional scene graph and the cosine similarities corresponding to each of the passable observation points; From all the passable observation points, select the passable observation point with the largest corresponding observation value as the best observation point of the placement structure.

9. The method according to claim 8, characterized in that For a single storage structure, the observation value of the i-th passable observation point corresponding to the storage structure is calculated using the following formula: Among them, Val i It is used to indicate the observation value of the i-th passable observation point for the placement structure, d max It is used to represent the maximum observation distance among the observation distances of all the passable observation points corresponding to the placement structure, d i It is used to represent the observation distance of the i-th passable observation point corresponding to the placement structure, CS i It is used to represent the cosine similarity of the i-th traversable observation point corresponding to the placement structure.

10. A robot control device, characterized in that: It includes a processor and a memory, the memory stores a computer program that can be executed by the processor, and the processor can execute the computer program to drive the target robot to cooperate in implementing the three-dimensional scene graph updating method described in any one of claims 1-9.

Images