Three-dimensional scene graph update method and robot control device

Abstract

The application provides a three-dimensional scene graph updating method and a robot control device, and relates to the technical field of robot control. The application analyzes natural language operation instructions issued by a user by calling a large language model to obtain high-level language action instructions for a target robot to pass through a target observation point in a target scene. During the movement of the target robot in the target scene according to the high-level language action instructions, actual scene images captured by the target robot at the target observation point for a target storage structure are obtained. Then, the actual scene images are used to update the information of the placed objects of the target three-dimensional scene graph of the target scene at the target storage structure, so that the dynamic maintenance effect of the robot working scene information is realized through local scene information updating, and the dynamic perception of the robot working scene changes is facilitated with low loss.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of robot control, in particular to a three-dimensional scene graph updating method and a robot control device. BACKGROUND

[0002] With the continuous development of science and technology, robot technology has been widely valued by various industries due to its great research value and application value. One important research direction in the field of robot control technology today is that users use natural language to drive robots to perform complex tasks through large language models (LLM). In the process of using a large language model to drive a robot to perform a complex task, the large language model usually analyzes the natural language instructions issued by the user and generates a complex instruction sequence that the robot agent can understand and execute, so as to drive the robot to perform the complex instruction sequence and execute the complex task expected by the user.

[0003] Currently, when using a large language model to control a robot to perform a complex task in a work scene, the understanding and perception of the work scene information depend on a three-dimensional scene graph constructed in advance, so as to provide rich environmental cognition for robot task execution operations using the three-dimensional scene graph of the work scene. However, it is worth noting that this three-dimensional scene graph constructed in advance is a static scene graph and cannot reflect the dynamic scene changes of the work scene, which easily leads to the fact that the complex instruction sequence generated by the large language model for the robot cannot be effectively compatible with the real work scene, and the corresponding robot cannot perform the complex task expected by the user, unless a large amount of time and effort is spent to periodically rescan the work scene and rebuild the three-dimensional scene graph to achieve the desired complex task execution effect. SUMMARY

[0004] Therefore, the purpose of the present application is to provide a three-dimensional scene graph updating method and a robot control device, which can update the three-dimensional scene graph of the robot work scene in the process of using a large language model to drive a robot to perform a desired task, so as to achieve the effect of dynamically maintaining the robot work scene information without the need to periodically rescan the work scene and rebuild the three-dimensional scene graph, thereby effectively reducing the time cost and computing resource loss of the robot work scene change cognition operation, and enabling the corresponding large language model to dynamically perceive the robot work scene information.

[0005] To achieve the above purpose, the technical solutions adopted by the embodiments of the present application are as follows:

[0006] In a first aspect, the present application provides a three-dimensional scene graph updating method, which comprises:

[0007] acquire a natural language task instruction for a target robot, and call a large language model to perform instruction analysis on the natural language task instruction to obtain a high-level language action instruction for the target robot to pass through a target observation point in a target scene, wherein the target observation point is an optimal observation point of a target storage structure pointed to by the natural language task instruction;

[0008] drive the target robot to move in the target scene according to the high-level language action instruction, and acquire an actual scene image captured by the target robot at the target observation point for the target storage structure;

[0009] update the information of the placed object of the target three-dimensional scene map of the target scene at the target storage structure according to the actual scene image.

[0010] In an optional implementation, the step of calling the large language model to perform instruction analysis on the natural language task instruction to obtain the high-level language action instruction for the target robot to pass through the target observation point in the target scene includes:

[0011] calling the large language model to perform task decomposition on the natural language task instruction in the target scene to obtain a plurality of continuous sub-tasks corresponding to the natural language task instruction;

[0012] for a first sub-task in the plurality of continuous sub-tasks that matches the target storage structure, based on position information of the target storage structure and the target observation point in the target scene respectively, calling a pre-stored high-level language movement control function to generate a first high-level language sub-task instruction for passing through the target observation point to the target storage structure;

[0013] for each second sub-task in the plurality of continuous sub-tasks other than the first sub-task, based on an association relationship between the actual sub-task content of the second sub-task and the content of the target three-dimensional scene map, calling an adapted high-level language movement control function to generate a corresponding second high-level language sub-task instruction;

[0014] performing instruction sorting and aggregation on the first high-level language sub-task instruction and all second high-level language sub-task instructions to obtain the high-level language action instruction.

[0015] In an optional implementation, the step of updating the information of the target three-dimensional scene map of the target scene at the target storage structure according to the actual scene image includes:

[0016] acquiring a reference scene image of the target storage structure at the target observation point adapted to the target three-dimensional scene map;

[0017] calculating an image similarity between the reference scene image and the actual scene image;

[0018] detecting whether the image similarity is less than a preset similarity threshold;

[0019] in a case where it is detected that the image similarity is less than the preset similarity threshold, updating, according to the actual scene image, the information of the placed object located at the target placement structure in the target three-dimensional scene graph.

[0020] In an optional implementation, the step of calculating the image similarity between the reference scene image and the actual scene image comprises:

[0021] mapping the reference scene image and the actual scene image into an embedding space respectively to obtain a first embedding vector corresponding to the reference scene image and a second embedding vector corresponding to the actual scene image;

[0022] calculating a cosine similarity between the first embedding vector and the second embedding vector, and taking the calculated cosine similarity as the image similarity.

[0023] In an optional implementation, the step of updating, according to the actual scene image, the information of the placed object located at the target placement structure in the target three-dimensional scene graph comprises:

[0024] performing object recognition on the actual scene image to determine all candidate objects in the actual scene image that are located close to the target placement structure;

[0025] performing placement rationality analysis on all the determined candidate objects according to the structure information of the target placement structure to filter out all target objects that are actually placed at the target placement structure from the candidate objects;

[0026] for each target object, generating object attribute information that is adapted to the format of the target three-dimensional scene graph based on the actual scene image, and performing information correction on the generated object attribute information to obtain actual object information of the target object;

[0027] replacing, according to the actual object information of all target objects, the information of the placed object located at the target placement structure in the target three-dimensional scene graph.

[0028] In an optional implementation, the method further comprises:

[0029] for each placement structure in the target scene, determining all observable points of the target robot with respect to the placement structure in the original three-dimensional scene graph of the target scene;

[0030] obtaining an original scene image of the storage structure observed by the target robot at each of the passable observation points in the original three-dimensional scene graph, and obtaining all stored item information of the storage structure at the original three-dimensional scene graph;

[0031] determining, from all the passable observation points, a best observation point of the storage structure according to all the stored item information corresponding to the storage structure and the original scene image of each of the passable observation points.

[0032] In an optional implementation, for each storage structure in the target scene, the step of determining, in the original three-dimensional scene graph of the target scene, all the passable observation points of the target robot with respect to the storage structure, comprises:

[0033] circumscribing an observation area covering at least the storage structure in the original three-dimensional scene graph based on a safe moving distance of the target robot;

[0034] generating a plurality of initial observation points in the observation area uniformly, and eliminating invalid observation points that cannot be reached by the target robot from the plurality of initial observation points to obtain all the passable observation points of the target robot with respect to the storage structure.

[0035] In an optional implementation, the step of determining, from all the passable observation points, a best observation point of the storage structure according to all the stored item information corresponding to the storage structure and the original scene image of each of the passable observation points, comprises:

[0036] calculating a cosine similarity between the original scene image of each of the passable observation points and the all the stored item information respectively;

[0037] calculating an observation value of each of the passable observation points with respect to the storage structure according to an observation distance of each of the passable observation points to the storage structure in the original three-dimensional scene graph and the cosine similarity corresponding to each of the passable observation points respectively;

[0038] selecting, from all the passable observation points, a passable observation point corresponding to the largest observation value as the best observation point of the storage structure.

[0039] In an optional implementation, for a single storage structure, an observation value of an i-th passable observation point corresponding to the storage structure is calculated by using the following formula:

[0040]

[0041] wherein, Val id represents the observation value of the i-th passable observation point for the storage structure, max d represents the maximum observation distance in the observation distances of all passable observation points corresponding to the storage structure, i CS represents the observation distance of the i-th passable observation point corresponding to the storage structure. i CS represents the cosine similarity of the i-th passable observation point corresponding to the storage structure.

[0042] In a second aspect, the present application provides a robot control device, comprising a processor and a memory, the memory stores a computer program capable of being executed by the processor, and the processor can execute the computer program to drive a target robot to cooperate to realize the three-dimensional scene graph updating method in any one of the preceding embodiments.

[0043] In this case, the beneficial effects of the embodiments of the present application can include the following:

[0044] The present application parses the natural language job instruction issued by the user by calling the large language model, obtains the high-level language action instruction required by the target robot to pass through the target observation point in the target scene, and obtains the actual scene image photographed by the target robot at the target observation point for the target storage structure in the process of the target robot moving in the target scene according to the high-level language action instruction. Then, according to the actual scene image obtained, the information of the placed object of the target three-dimensional scene graph of the target scene at the target storage structure is updated, so that in the process of driving the robot to perform the expected task by using the large language model, the three-dimensional scene graph of the robot working scene is updated locally to realize the dynamic maintenance effect of the robot working scene information, without consuming a lot of time and effort to periodically rescan the job scene and reconstruct the three-dimensional scene graph, so as to effectively reduce the time cost and calculation resource loss required for the change recognition operation of the robot working scene, and make the corresponding large language model can dynamically perceive the robot working scene information.

[0045] In order to make the above-mentioned purposes, features and advantages of the present application more obvious and easy to understand, the following preferred embodiments are described in detail below, and the accompanying drawings are described as follows. BRIEF DESCRIPTION OF DRAWINGS

[0046] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following will briefly introduce the drawings needed in the embodiments. It should be understood that the following drawings only show some embodiments of the present application, and therefore should not be regarded as a limitation on the scope. For those skilled in the art, other related drawings can also be obtained without creative labor on the basis of these drawings.

[0047] Figure 1 A schematic diagram of a robot control device according to an embodiment of the application;

[0048] Figure 2 A schematic diagram of a method of updating a three-dimensional scene graph according to an embodiment of the application;

[0049] Figure 3 A schematic diagram of a method of updating a three-dimensional scene graph according to an embodiment of the application; Figure 2 A schematic diagram of sub-steps comprised in step S210 in

[0050] Figure 4 A schematic diagram of sub-steps comprised in step S230 in Figure 2 A schematic diagram of sub-steps comprised in step S230 in

[0051] Figure 5 A schematic diagram of a method of updating a three-dimensional scene graph according to an embodiment of the application;

[0052] Legend: 10 - robot control device; 11 - memory; 12 - processor; 13 - communication unit. DETAILED DESCRIPTION

[0053] In order to make the purposes, technical solutions, and advantages of the embodiments of the present application clearer, the following will be combined with the accompanying drawings for the embodiments of the present application to make a clear and complete description of the technical solutions of the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments of the present application. The components of the embodiments of the present application described and shown in the accompanying drawings can be arranged and designed in various different configurations.

[0054] Therefore, the following detailed description of the embodiments of the present application provided in the accompanying drawings is not intended to limit the scope of the claimed application, but only represents selected embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work are within the scope of protection of the present application.

[0055] It should be noted that: similar reference numerals and letters represent similar items in the following drawings, therefore, once an item is defined in one drawing, it does not need to be further defined and explained in subsequent drawings.

[0056] In the description of the application, it needs to be understood that the terms "center", "upper", "lower", "left", "right", "vertical", "horizontal", "inner", "outer" and the like indicate the orientation or positional relationship based on the orientation or positional relationship shown in the drawings, or the orientation or positional relationship commonly understood by those skilled in the art, or the orientation or positional relationship commonly understood by those skilled in the art, or the orientation or positional relationship commonly understood by those skilled in the art, only for the convenience of describing the application and simplifying the description, and not indicating or implying that the device or element referred to must have a particular orientation, be constructed and operated in a particular orientation, and therefore cannot be understood as a limitation of the application.

[0057] In the description of the application, it also needs to be explained that, unless otherwise explicitly specified and limited, the terms "set", "install", "connect", "connect" should be understood broadly, for example, it can be fixedly connected, or it can be detachably connected, or integrally connected; it can be mechanically connected, or it can be electrically connected; it can be directly connected, or it can be indirectly connected through an intermediate medium, or it can be the communication inside two elements. For those skilled in the art, the specific meaning of the above terms in the application can be understood according to the specific circumstances.

[0058] In addition, in the description of the application, it can be understood that the relationship terms such as "first" and "second" and the like are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between the entities or operations. Moreover, the terms "include", "contain" or any other variant thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements not only includes those elements, but also includes other elements not explicitly listed, or includes elements inherent to such process, method, article or device. Without more limitation, the element defined by the sentence "including a…" does not exclude the presence of other identical elements in the process, method, article or device including the element. For those skilled in the art, the specific meaning of the above terms in the application can be understood according to the specific circumstances.

[0059] Some embodiments of the application will be described in detail below with reference to the accompanying drawings. The following examples and features in the examples can be combined with each other without conflict.

[0060] Please refer to Figure 1 , Figure 1is a constituent schematic diagram of a robot control device 10 provided by an embodiment of the present application. In the embodiment of the present application, the robot control device 10 can be in communication connection with a controlled robot, and respond to a natural language job instruction issued by a user based on a large language model loaded by itself, to control the controlled robot to execute a desired task represented by the natural language job instruction in a current working scenario. Wherein, the robot control device 10 can be a computer device independent of the controlled robot, and the computer device can be, but is not limited to, a personal computer, a notebook computer, a tablet computer, a server, etc.; the robot control device 10 can also be integrated with the controlled robot.

[0061] In the embodiment of the present application, the robot control device 10 can include a memory 11, a processor 12 and a communication unit 13. Wherein, the memory 11, the processor 12 and the communication unit 13 are directly or indirectly electrically connected to each other to realize the transmission or interaction of data. For example, the memory 11, the processor 12 and the communication unit 13 can be electrically connected to each other through one or more communication buses or signal lines.

[0062] In the embodiment, the memory 11 can be, but is not limited to, a random access memory (RAM), a read only memory (ROM), a programmable read only memory (PROM), an erasable programmable read only memory (EPROM), an electrically erasable programmable read only memory (EEPROM) and the like. Wherein, the memory 11 is used to store a computer program, and the processor 12 can execute the computer program accordingly after receiving an execution instruction.

[0063] Meanwhile, the memory 11 can also be used to store a large language model, which can be loaded with a three-dimensional scene graph of a robot work scene where the controlled robot is currently located (for example, an original three-dimensional scene graph initially constructed for the robot work scene), physical hardware information of the controlled robot, and a plurality of high-level language movement control functions related to robot task execution operations. Among them, the physical hardware information includes specific body structure information of the controlled robot (for example, joint type, moving wheel type, body height information, etc.); the high-level language movement control function is a program code described in a high-level language (for example, python language, Java language or PHP language, etc.) for realizing a certain movement control function (for example, a grabbing action function, a putting down action function, a forward action function, etc.) of the controlled robot, and a plurality of the high-level language movement control functions can include getUserPosition() function, getPosition(str) function, goto(pos) function, take() function, put() function, moveAhead(length) function, moveBack(length) function, turnRight(angel) function and turnLeft(angle) function, etc., wherein the getUserPosition() function is used to obtain the current robot position of the controlled robot, the getPosition(str) function is used to obtain the actual position of the str object, the goto(pos) function is used to drive the controlled robot to move to the position corresponding to pos, the take() function is used to drive the controlled robot to grab the object at the current position, the put() function is used to drive the controlled robot to put down the object being grabbed at the current position, the moveAhead(length) function is used to drive the controlled robot to move ahead by a distance of length, the moveBack(length) function is used to drive the controlled robot to move back by a distance of length, the turnRight(angel) function is used to drive the controlled robot to turn right by an angle of angel, and the turnLeft(angle) function is used to drive the controlled robot to turn left by an angle of angel.

[0064] In this embodiment, the processor 12 can be an integrated circuit chip with signal processing capability. The processor 12 can be a general-purpose processor, including at least one of a central processing unit (CPU), a graphics processing unit (GPU), and a network processor (NP), a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component, or the like. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc., which can implement or execute the disclosed methods, steps, and logic block diagrams in the embodiments of the present application.

[0065] In this embodiment, the communication unit 13 is configured to establish a communication connection between the robot control device 10 and other electronic devices through a network, and transmits and receives data through the network, wherein the network includes a wired communication network and a wireless communication network. For example, the robot control device 10 can obtain a natural language operation instruction issued by a user through a terminal device through the communication unit 13, wherein the natural language operation instruction can be issued to the terminal device in a voice input manner or a text input manner, and the terminal device can be, but is not limited to, a smart phone, a personal computer, a tablet computer, etc.

[0066] In this embodiment, the robot control device 10 can pre-store a specific computer program related to the three-dimensional scene graph updating function at the memory 11, and through driving the processor 12 to execute the specific computer program correspondingly, in the process of calling a large language model to drive a controlled robot to execute a desired task, the three-dimensional scene graph of a robot working scene is updated with local scene information, so as to realize the dynamic maintenance effect of the robot working scene information, without consuming a lot of time and effort to periodically rescan the operation scene and reconstruct the three-dimensional scene graph, so as to effectively reduce the time cost and computing resource loss required for the robot working scene change recognition operation, and facilitate low-loss dynamic perception of the robot working scene change by the large language model.

[0067] It can be understood that, Figure 1 The block diagram shown is only a composition schematic diagram of the robot control device 10, and the robot control device 10 can further include more or fewer components than those shown in Figure 1 or have a different configuration than that shown in Figure 1 . Figure 1 The components shown in the above embodiments can be implemented in hardware, software, or a combination thereof.

[0068] In the present application, in order to ensure that the robot control device 10 can update the local scene information of the three-dimensional scene graph of the robot working scene in the process of driving the robot with a large language model to perform a desired task, so as to achieve the effect of dynamically maintaining the robot working scene information, and facilitate the dynamic perception of the robot working scene change with low loss, the present application embodiment provides a three-dimensional scene graph updating method to achieve the foregoing purpose, which will be described in detail below.

[0069] Please refer to Figure 2 , Figure 2 is one of the flowcharts of the three-dimensional scene graph updating method provided by the present application embodiment. In the present application embodiment, Figure 2 The three-dimensional scene graph updating method shown in the present application embodiment can include steps S210-S230.

[0070] Step S210, obtaining a natural language job instruction for a target robot, and calling a large language model to parse the natural language job instruction to obtain a high-level language action instruction for the target robot to pass through a target observation point in a target scene.

[0071] In the present embodiment, the target robot is a controlled robot that can be currently controlled by the robot control device 10; after obtaining the natural language job instruction issued by the user for the target robot, the robot control device 10 can obtain the target job instruction content in the form of text by language recognition on the natural language job instruction, and then input the target job instruction content into the large language model to utilize the text recognition capability and robot motion planning capability of the large language model, and construct a high-level language action instruction matching the natural language job instruction based on the current target three-dimensional scene graph of the target scene (i.e. the actual three-dimensional scene graph of the target scene before receiving the natural language job instruction). The high-level language action instruction is related to a target placement structure pointed by the natural language job instruction, the target placement structure is statically set in the target scene, and any placement structure in the target scene is used to place movable articles, wherein the placement structure can be but not limited to: a table, a cabinet, a dressing table, a washstand, etc.; the high-level language action instruction can drive the target robot to execute the desired task represented by the natural language job instruction by passing through the target observation point (i.e. the best observation point of the target placement structure) in the target scene, so that the target robot can take a scene image of the target placement structure when moving to the target observation point, and obtain an actual scene image representing the actual scene condition around the target placement structure. In an embodiment of the present embodiment, the target placement structure is located at the middle position of the actual scene image.

[0072] Optionally, please refer to Figure 3 , Figure 3 is Figure 2 The flowchart of the sub-steps included in step S210 in FIG. 2. In the embodiment of the present application, step S210 can include sub-step S211 to sub-step S214 to ensure that the corresponding generated high-level language action instruction can drive the target robot to perform the user desired task in the target scene while passing through the target observation point.

[0073] Sub-step S211, calling a large language model to task decompose the natural language job instruction in the target scene to obtain a plurality of continuous sub-tasks corresponding to the natural language job instruction.

[0074] In the embodiment, the large language model can determine a plurality of robot decomposition actions that the natural language job instruction needs to be continuously implemented in the target scene based on the target three-dimensional scene graph of the target scene, so that each robot decomposition action corresponds to a continuous sub-task. For example, the natural language job instruction "go to the study to help me take the water bottle on the desk back" can be decomposed into continuous sub-tasks "move into the study", "move to the desk", "grab the water bottle on the desk", "move outside the study", "move to the user", and "put down the grabbed water bottle".

[0075] Sub-step S212, for a first sub-task in the plurality of continuous sub-tasks that matches the target storage structure, based on the position information of the target storage structure and the target observation point in the target scene respectively, calling a pre-stored high-level language movement control function to generate a first high-level language sub-task instruction to go to the target storage structure through the target observation point.

[0076] In the embodiment, the large language model can extract a first high-level language movement control function adapted to the first sub-task from a plurality of pre-stored high-level language movement control functions by performing task content recognition on the first sub-task, and then call the first high-level language movement control function to generate a matched first high-level language sub-task instruction according to the position information of the target storage structure and the target observation point in the target scene. For example, the matched first high-level language sub-task instruction for the first sub-task "move to the desk" can be described as "goto(target observation point)" and "goto(desk)".

[0077] Sub-step S213, for each second sub-task in the plurality of continuous sub-tasks other than the first sub-task, based on the content association relationship between the actual sub-task content of the second sub-task and the target three-dimensional scene graph, calling an adapted high-level language movement control function to generate a corresponding second high-level language sub-task instruction.

[0078] In this embodiment, the content association relationship can be used to represent the mapping relationship between the actual subtask content corresponding to the second subtask and each object existing in the target three-dimensional scene graph; for each second subtask, the large language model can perform task content recognition on the second subtask to extract a second high-level language mobile control function adapted to the second subtask from the pre-stored plurality of high-level language mobile control functions, and then call the second high-level language mobile control function to generate a matched second high-level language subtask instruction according to the content association relationship corresponding to the second subtask.

[0079] Substep S214, the first high-level language subtask instruction and all second high-level language subtask instructions are sorted and aggregated to obtain a high-level language action instruction.

[0080] In this embodiment, the large language model can perform instruction sorting and aggregation processing on the obtained first high-level language subtask instruction and all second high-level language subtasks according to the subtask sorting condition of the first subtask and all second subtasks at the natural language task instruction, to obtain a high-level language action instruction adapted to the natural language task instruction.

[0081] Therefore, the present application can ensure that the generated high-level language action instruction can drive the target robot to perform the user desired task at the target observation point in the target scene by executing the above-mentioned substeps S211-S214.

[0082] Step S220, driving the target robot to move in the target scene according to the high-level language action instruction, and obtaining an actual scene image captured by the target robot at the target observation point for the target object structure.

[0083] In one implementation of this embodiment, for the first high-level language subtask instruction included in the high-level language action instruction, a high-level language instruction content "collect(sceneinfo)" can be added at the target observation point to instruct the target robot to capture a scene image for the target object structure at the target observation point, so as to effectively obtain the actual local scene information (i.e., the actual scene image) corresponding to the target object structure in the robot working scene. At this time, the first high-level language subtask instruction can be represented in turn as "goto(target observation point)", "collect(scene info)", and "goto(desk)".

[0084] Step S230, updating the information of the placed object at the target object structure in the current target three-dimensional scene graph of the target scene according to the actual scene image.

[0085] In this embodiment, after the robot control device 10 obtains the actual scene image of the target placement structure at the target observation point, it can extract all the actual placement information of the target items placed by the target placement structure from the actual scene image according to the characteristics of the large language model that tends to convert a three-dimensional scene graph into a topological structure format (for example, JSON format) recorded by nodes and edges, and then replace the placed item information of the target three-dimensional scene graph at the target placement structure with the extracted actual placement information of all target items, to complete the local scene information updating operation of the three-dimensional scene graph of the robot work scene, realize the dynamic maintenance effect of the robot work scene information, and effectively reduce the time cost and computing resource loss of the robot work scene change cognition operation, so that the corresponding large language model can dynamically perceive the changes of the robot work scene with low loss.

[0086] In one implementation of this embodiment, when the robot control device 10 performs the local scene information updating operation on the robot work scene, it can directly update the placed item information of the target three-dimensional scene graph at the target placement structure with the actual scene image taken by the target robot at the target observation point for the target placement structure, without considering whether the scene condition corresponding to the target placement structure in the target three-dimensional scene graph has changed.

[0087] Optionally, please refer to Figure 4 , Figure 4 is Figure 2 the flowchart of the sub-steps included in step S230. In another implementation of this embodiment, to avoid performing meaningless local scene information updating operation and ensure that the robot control device 10 only performs local scene information updating operation when the scene around the target placement structure changes dramatically, step S230 can include sub-step S231 to sub-step S234.

[0088] Sub-step S231: Obtain the reference scene image of the target placement structure at the target observation point and adapted to the target three-dimensional scene graph.

[0089] In this embodiment, the reference scene image is an effective scene image obtained by observing the target placement structure at the target observation point in the target three-dimensional scene graph.

[0090] Sub-step S232, calculate the image similarity between the reference scene image and the actual scene image.

[0091] The step of calculating the image similarity between the reference scene image and the actual scene image can include:

[0092] Map the reference scene image and the actual scene image into an embedding space respectively to obtain a first embedding vector corresponding to the reference scene image and a second embedding vector corresponding to the actual scene image.

[0093] Calculate the cosine similarity between the first embedding vector and the second embedding vector, and take the calculated cosine similarity as the image similarity.

[0094] Sub-step S233, detect whether the image similarity is less than a preset similarity threshold.

[0095] In this embodiment, when it is detected that the image similarity between the reference scene image and the actual scene image is less than the preset similarity threshold, it indicates that the surrounding scene of the target storage structure has a dramatic change phenomenon, and at this time, the local scene information update operation can be performed by executing sub-step S234; when it is detected that the image similarity between the reference scene image and the actual scene image is greater than or equal to the preset similarity threshold, it indicates that the surrounding scene of the target storage structure has no change, and at this time, the local scene information update operation does not need to be performed on the target three-dimensional scene graph.

[0096] Sub-step S234, in the case where it is detected that the image similarity is less than the preset similarity threshold, update the information of the placed object located at the target storage structure in the target three-dimensional scene graph according to the actual scene image.

[0097] In this embodiment, the step of "updating the information of the placed object located at the target storage structure in the target three-dimensional scene graph according to the actual scene image" in the sub-step S234 can include sub-step A to sub-step D.

[0098] Sub-step A: perform object recognition on the actual scene image to determine all candidate objects in the actual scene image that are located close to the target storage structure.

[0099] Sub-step B: perform storage rationality analysis on all the determined candidate objects according to the structure information of the target storage structure to screen all target objects actually placed at the target storage structure from the candidate objects.

[0100] The placement rationality analysis operation for any candidate item can include, but is not limited to, an item position rationality analysis operation, an item semantic rationality analysis operation, and an item quantity rationality analysis operation, etc.

[0101] Sub-step C: For each target item, generating item attribute information adapted to the target three-dimensional scene graph format based on the actual scene image, and performing information correction on the generated item attribute information to obtain actual item information of the target item.

[0102] The item attribute information can include item type information, item placement position information, item function attribute information, item physical attribute information, and item shape information of the corresponding target item. The actual scene image can be detected using a visual language model based on a prompt design to obtain semantic text information of the actual scene image, and then the semantic text information is used for information verification processing and information correction processing of the item attribute information to obtain actual item information of the corresponding target item.

[0103] Sub-step D: Replacing the placed item information located at the target item placement structure in the target three-dimensional scene graph according to the actual item information of all target items.

[0104] Therefore, by performing the above-mentioned sub-step S231 to sub-step S234, the meaningless local scene information update operation can be avoided, and it is ensured that the robot control device 10 only performs the local scene information update operation when the scene around the target item placement structure changes dramatically.

[0105] By performing the above-mentioned step S210 to step S230, it is ensured that the robot control device 10 can update the three-dimensional scene graph of the robot working scene locally during the process of driving the robot to perform the expected task using the large language model, so as to realize the dynamic maintenance effect of the robot working scene information, and facilitate the dynamic perception of the robot working scene change with low loss.

[0106] Optionally, please refer to Figure 5 , Figure 5 is a flowchart of a three-dimensional scene graph updating method provided by an embodiment of the present application. In the embodiment of the present application, compared with the three-dimensional scene graph updating method shown in Figure 2 , the three-dimensional scene graph updating method shown in Figure 5 may further include steps S240 to S260 to effectively determine the best observation point of the target robot for each existing item placement structure in the target scene.

[0107] Step S240, for each storage structure in the target scene, determining all passable observation points of the target robot about the storage structure in the original three-dimensional scene graph of the target scene.

[0108] In this embodiment, the step "determining all passable observation points of the target robot about the storage structure in the original three-dimensional scene graph of the target scene" in the step S240 can include:

[0109] In the original three-dimensional scene graph, based on the safe moving distance of the target robot, an observation area covering at least the storage structure is circled;

[0110] A plurality of initial observation points are generated uniformly in the observation area, and invalid observation points that the target robot cannot reach are removed from the plurality of initial observation points, to obtain all passable observation points of the target robot about the storage structure.

[0111] Wherein, the observation area can be a circular area, the center position of the observation area and the calibration point position (for example, the geometric center position) of the corresponding storage structure overlap with each other, and the observation radius of the observation area can be twice the safe moving distance; the invalid observation points that the target robot cannot reach can be determined according to the distribution position information of static things (for example, storage structures and wall structures around the storage structures) in the original three-dimensional scene graph.

[0112] Step S250, obtaining original scene images of the target robot observing the storage structure at each passable observation point in the original three-dimensional scene graph, and obtaining all placed item information of the storage structure in the original three-dimensional scene graph.

[0113] In this embodiment, for any passable observation point of a storage structure in the original three-dimensional scene graph, the image shooting direction of the storage structure from the passable observation point can be determined according to the calibration point position of the storage structure and the actual observation point position of the passable observation point, and then the original scene image about the storage structure is extracted in the original three-dimensional scene graph based on the determined image shooting direction, wherein the storage structure is in the middle position of the corresponding original scene image.

[0114] For any one storage structure, all placed item information of the storage structure in the original three-dimensional scene graph includes item type, item placement position, item function attribute, item physical attribute, item shape and other information of the corresponding placed item.

[0115] Step S260, determining the best observation point of the storage structure in all the passable observation points according to all the stored article information corresponding to the storage structure and all the original scene images of all the passable observation points respectively.

[0116] In the embodiment, for any one storage structure, the step of "determining the best observation point of the storage structure in all the passable observation points according to all the stored article information corresponding to the storage structure and all the original scene images of all the passable observation points respectively" can include:

[0117] calculating the cosine similarity between the original scene image of each of the passable observation points and all the stored article information respectively;

[0118] calculating the observation value of each of the passable observation points for the storage structure according to the observation distance of each of the passable observation points to the storage structure in the original three-dimensional scene image and the cosine similarity corresponding to each of the passable observation points respectively;

[0119] selecting the passable observation point corresponding to the maximum observation value from all the passable observation points as the best observation point of the storage structure.

[0120] Wherein, for a single storage structure, the observation value of the i-th passable observation point corresponding to the storage structure is calculated by the following formula:

[0121]

[0122] Wherein, Val i is used to represent the observation value of the i-th passable observation point for the storage structure, d max is used to represent the maximum observation distance in the observation distances of all the passable observation points corresponding to the storage structure, d i is used to represent the observation distance of the i-th passable observation point corresponding to the storage structure, CS i is used to represent the cosine similarity of the i-th passable observation point corresponding to the storage structure.

[0123] Therefore, the present application can effectively determine the best observation point of each existing storage structure corresponding to the target robot in the target scene by executing the above steps S240-S260.

[0124] In the embodiments of the present disclosure, it should be understood that the disclosed apparatus and method can also be implemented in other manners. The embodiments described above are merely exemplary. For example, the flowcharts and block diagrams in the accompanying drawings illustrate the possible implementation architectures, functions and operation of the apparatus, method and computer program product according to the embodiments of the present disclosure. In this regard, each block in the flowcharts or block diagrams can represent a module, a segment or a portion of code which comprises one or more executable instructions for implementing the specified logic function. It should also be noted that in some alternative implementations, the functions noted in the blocks can occur in a different order than that noted in the accompanying drawings. For example, two blocks noted in succession can actually be executed substantially concurrently or in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts and combinations thereof can be implemented by a dedicated hardware-based system, or a combination of dedicated hardware and computer instructions.

[0125] In addition, the various functional modules in the embodiments of the present disclosure can be integrated together to form a separate part, or can exist independently, or two or more modules can be integrated to form a separate part. The various functions provided by the present disclosure, if realized in the form of software functional modules and sold or used as independent products, can be stored in a storage medium. Based on this understanding, the technical solutions of the present disclosure can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a plurality of instructions for causing the robot control device 10 to execute all or part of the steps of the method described in the various embodiments of the present disclosure. The aforementioned readable storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk or an optical disk, and various media that can store program codes.

[0126] The above is merely some embodiments of the present disclosure, but the protection scope of the present disclosure is not limited thereto, and any person skilled in the art can easily think of changes or replacements within the technical scope disclosed by the present disclosure, which should be covered by the protection scope of the present disclosure. Therefore, the protection scope of the present disclosure should be subject to the protection scope of the claims.

Claims

1. A three-dimensional scene graph updating method, characterized in that: The method comprises: Obtaining natural language operation instructions for a target robot, and using a large language model to parse the natural language operation instructions to obtain high-level language action instructions for the target robot to pass through a target observation point in a target scenario, where the target observation point is the optimal observation point of the target placement structure pointed to by the natural language operation instruction; driving the target robot to move in the target scene according to the high-level language action instructions, and acquiring an actual scene image captured by the target robot at the target observation point with respect to the target placement structure; The information of the placed items at the target placement structure is updated in the current target three-dimensional scene graph of the target scene according to the actual scene image.

2. The method according to claim 1, characterized in that The step of calling the large language model to parse the natural language operation instruction to obtain the high-level language action instruction for the target robot to pass through the target observation point in the target scene includes: Calling the large language model to perform task decomposition on the natural language job instruction in the target scenario to obtain a plurality of continuous subtasks corresponding to the natural language job instruction; For a first subtask among the multiple consecutive subtasks that matches the target placement structure, based on the position information of the target placement structure and the target observation point in the target scene, a pre-stored high-level language movement control function is called to generate a first high-level language subtask instruction for moving to the target placement structure via the target observation point; For each second subtask other than the first subtask in the plurality of consecutive subtasks, based on an association relationship between actual subtask content of the second subtask and content of the target three-dimensional scene graph, calling an adapted high-level language movement control function to generate a corresponding second high-level language subtask instruction; The first high-level language subtask instruction and all second high-level language subtask instructions are sequenced and aggregated to obtain the high-level language action instruction.

3. The method according to claim 1, characterized in that The step of updating the information of the placed items at the target placement structure of the current target three-dimensional scene graph of the target scene according to the actual scene image comprises: Acquire a reference scene image of the target placement structure adapted to the target three-dimensional scene graph at the target observation point; Calculating image similarity between the reference scene image and the actual scene image; Detecting whether the image similarity is less than a preset similarity threshold; When it is detected that the image similarity is less than a preset similarity threshold, the information of the placed objects located at the target placement structure in the target three-dimensional scene graph is updated according to the actual scene image.

4. The method according to claim 3, characterized in that The step of calculating the image similarity between the reference scene image and the actual scene image comprises: Mapping the reference scene image and the actual scene image into embedding spaces respectively to obtain a first embedding vector corresponding to the reference scene image and a second embedding vector corresponding to the actual scene image; Calculate the cosine similarity between the first embedding vector and the second embedding vector, and use the calculated cosine similarity as the image similarity.

5. The method according to claim 3, characterized in that The step of updating the information of the placed items located at the target placement structure in the target three-dimensional scene graph according to the actual scene image comprises: Performing object recognition on the actual scene image to determine all candidate objects in the actual scene image that are located close to the target storage structure; performing a placement rationality analysis on all determined candidate items based on the structural information of the target placement structure, so as to screen out all target items actually placed in the target placement structure from all candidate items; For each target item, generating item attribute information adapted to the target three-dimensional scene graph format based on the actual scene image, and performing information correction on the generated item attribute information to obtain actual item information of the target item; According to the actual item information of all target items, information of the placed items located at the target placement structure in the target three-dimensional scene graph is replaced.

6. The method according to any one of claims 1 to 5, characterized in that The method further comprises: For each placement structure in the target scene, determining all traversable observation points of the target robot with respect to the placement structure in an original three-dimensional scene graph of the target scene; Obtaining original scene images of the placement structure observed by the target robot at each traversable observation point in the original three-dimensional scene graph, and obtaining information of all placed items of the placement structure at the original three-dimensional scene graph; According to all the information of the placed items corresponding to the storage structure and the original scene images of all the passable observation points, the best observation point of the storage structure is determined among all the passable observation points.

7. The method according to claim 6, characterized in that For each placement structure in the target scene, the step of determining all traversable observation points of the target robot with respect to the placement structure in the original three-dimensional scene graph of the target scene includes: In the original three-dimensional scene graph, based on the safe movement distance of the target robot, an observation area that at least covers the placement structure is delineated; A plurality of initial observation points are evenly generated in the observation area, and invalid observation points that the target robot cannot reach are eliminated from the plurality of initial observation points to obtain all traversable observation points of the target robot on the placement structure.

8. The method according to claim 6, characterized in that The step of determining the best observation point of the storage structure from among all passable observation points based on information of all placed items corresponding to the storage structure and the original scene images of all passable observation points includes: Calculating the cosine similarity between the original scene images of all the passable observation points and the information of all the placed objects; Calculate the observation value of each of the passable observation points with respect to the placement structure based on the observation distance of each of the passable observation points to the placement structure in the original three-dimensional scene graph and the cosine similarity corresponding to each of the passable observation points; From all the passable observation points, the passable observation point with the greatest corresponding observation value is selected as the best observation point of the placement structure.

9. The method according to claim 8, characterized in that For a single storage structure, the observation value of the i-th passable observation point corresponding to the storage structure is calculated using the following formula: Among them, Val i It is used to indicate the observation value of the i-th passable observation point for the placement structure, d max It is used to represent the maximum observation distance among all the observation distances of the passable observation points corresponding to the placement structure, d i It is used to represent the observation distance of the i-th passable observation point corresponding to the placement structure, CS i It is used to represent the cosine similarity of the i-th passable observation point corresponding to the placement structure.

10. A robot control device, characterized in that: It includes a processor and a memory, the memory stores a computer program that can be executed by the processor, and the processor can execute the computer program to drive the target robot to cooperate with the three-dimensional scene graph updating method described in any one of claims 1 to 9.

Images