Target object searching method and system based on node information, terminal and medium

Abstract

The invention relates to the field of robot navigation, and particularly discloses a target object searching method and system based on node information, a terminal and a medium, and the method comprises the steps: constructing a building, floor, room and object layered scene map from top to bottom; extracting object information in the room nodes by using a large language model, and taking the extracted information as node information of the room nodes; folding the layered scene map to obtain a simplified scene map, and adding corresponding node information to each room node in the simplified scene map; in response to a navigation task instruction, inputting the simplified scene map added with the node information and the navigation task instruction into the large language model, so that the large language model is positioned to a room node matched with the navigation task; and unfolding the positioned room node, and finding a target object from the current room node by using a large language model. The efficiency and precision of target object searching are improved, and the operation efficiency of the robot is improved.

Description

Technical Field

[0001] The present invention relates to the field of robot navigation, and in particular to a target object search method, system, terminal and medium based on node information. Background Art

[0002] After receiving a navigation task instruction, the robot first searches for the target object related to the navigation instruction within the constructed scene graph. It then performs task planning, constructing a sequence of actions to complete the navigation task. Finally, the robot's chassis and manipulators, including its mobile operating devices, implement the navigation task. To improve the robot's search efficiency for target objects within the scene graph, the scene graph is often designed as a hierarchical graph structured "building-floor-room-object," based on the geometric spatial location of objects and rooms. Leveraging the powerful interpretation and reasoning capabilities of large language models, the hierarchical structure of the scene graph is used to locate the target objects required for the navigation task, enabling intelligent navigation.

[0003] However, related technologies use a large language model (LLM) during target search based on a simplified scene graph and a constructed navigation task prompt word template to infer the room to which the navigation task belongs. The inferred room nodes are then expanded sequentially until the target object is found. A simplified scene graph is a folded hierarchical scene graph that only contains buildings, floors, and rooms. This approach, which predicts room nodes based solely on the simplified scene graph and navigation task to find the target object, lacks sufficient room node prediction accuracy and requires multiple interactions of expanding and collapsing room nodes, resulting in low efficiency and impacting the robot's operational efficiency. Summary of the Invention

[0004] To solve the above problems, the present invention provides a target object search method, system, terminal and medium based on node information, which improve the efficiency and accuracy of target object search and improve the operating efficiency of the robot.

[0005] In a first aspect, the technical solution of the present invention provides a target object search method based on node information, comprising the following steps: Constructing a hierarchical scene graph, wherein the node types of the hierarchical scene graph are buildings, floors, rooms, and objects from top to bottom; Use the large language model to extract the object information in the room node, and use the extracted information as the node information of the room node; Folding the layered scene graph to obtain a simplified scene graph, wherein the simplified scene graph only includes buildings, floors, and rooms; and then attaching corresponding node information to each room node in the simplified scene graph; In response to navigation task instructions, the simplified scene graph with attached node information and navigation task instructions are input into the large language model, so that the large language model locates the room node that matches the navigation task; Expand the located room node and use the large language model to find the target object in the current room node.

[0006] In an optional embodiment, a large language model is used to extract object information in a room node, and the extracted information is used as node information of the room node, specifically including: Construct node information extraction prompt words; Traverse each room node in the hierarchical scene graph, obtain the properties of the room node and the set of all lower-level nodes it contains; The attributes of the room node, the set of all lower-level nodes it contains, and the prompt words are input into the large language model, so that the large language model can be refined to the node information of the room node.

[0007] In an optional embodiment, positioning the large language model at a room node that matches the navigation task specifically includes: The large language model outputs a set of room nodes with the highest probability of matching the navigation task, and the room nodes in this set are sorted from high to low according to the probability of matching.

[0008] In an optional embodiment, the located room node is expanded and the target object is found in the current room node using the large language model, specifically including: Select the room node with the highest matching probability from the room node set, expand the room node, and use the large language model to search for the target object in the room node; If the target object is found, the target object search procedure is terminated; If the target object is not found, the room node with the second highest matching probability is selected from the room node set, and the target object is searched from this room node using the large language model. This process is repeated until the target object is found or all room nodes in the set are exhausted. If the target object is still not found after traversing all room nodes in the collection, an error message will be displayed.

[0009] In an optional embodiment, the method further comprises the following steps: Construct a mapping relationship between the navigation task instructions and the room nodes where the target object is located and store it in the target area; In response to the current navigation task instruction, searching for a navigation task instruction whose semantic similarity with the current navigation task instruction exceeds a threshold from the target area; If found, it is located at the room node according to the mapping relationship and recorded as the memory room node; Check whether the room node set contains the memory room node; If included, add the memory weight factor to the matching probability of the memory room node as the new matching probability, and then re-sort all room nodes in the set according to the matching probability; If not included, the memory weight factor of the memory room node is added to the set as its matching probability, and all room nodes in the set are reordered according to the matching probability.

[0010] In an optional embodiment, the memory weight factor is calculated by the following formula,

[0011] In the formula, is the current memory weight factor, is the initial weight base, is the decay coefficient, is the time length from the last time the memory is verified to be valid to the current time.

[0012] In an optional embodiment, the method further comprises the following steps: According to the hit record of the room node in the recent several times, the navigation task instruction corresponding to the hit of the room node is obtained; The large language model is used to extract memory hit information from the obtained several corresponding navigation task instructions; In the simplified scene graph, the corresponding memory hit information is attached to each room node; In response to the navigation task instruction, the simplified scene graph with attached node information and memory hit information, and the navigation task instruction are input to the large language model, so that the large language model is positioned to the room node matching the navigation task.

[0013] In a second aspect, the technical solution of the present application provides a target object search system based on node information, comprising: A hierarchical scene graph construction module is used to construct a hierarchical scene graph, and the node types of the hierarchical scene graph are buildings, floors, rooms and objects from top to bottom; A node information extraction module is used to extract object information in the room node using a large language model, and the extracted information is used as the node information of the room node; A scene graph simplification module is used to fold the hierarchical scene graph to obtain a simplified scene graph, and the simplified scene graph only contains buildings, floors and rooms; then the corresponding node information is attached to each room node in the simplified scene graph; A target object search module is used to input the simplified scene graph with attached node information and the navigation task instruction to the large language model in response to the navigation task instruction, so that the large language model is positioned to the room node matching the navigation task; the positioned room node is unfolded, and the large language model is used to find the target object from the current room node.

[0014] In a third aspect, the technical solution of the present application provides a terminal, comprising: A memory, used for storing a target object search program based on node information; A processor is configured to implement the steps of any of the above-mentioned target object search methods based on node information when executing the target object search program based on node information.

[0015] In a fourth aspect, the technical solution of the present invention provides a computer-readable storage medium, on which a target object search program based on node information is stored. When the target object search program based on node information is executed by a processor, the steps of the target object search method based on node information as described in any one of the above items are implemented.

[0016] It can be seen from the above technical solutions that the present application has the following advantages: using a large speech model to actively extract object information from room nodes, appending the extracted information as node information to a simplified scene graph, and then performing target search based on the simplified scene graph with attached node information. The present invention appends semantic node information generated by a large language model to each room node, enabling the large language model to more accurately understand the content features of each room, and perform intelligent reasoning based on the semantic features of the scene, thereby more accurately locating the room node that is most likely to contain the target object, avoiding the inefficient interactive process of multiple expansion and folding of room nodes in traditional methods, reducing repeated search operations due to inaccurate predictions, and overall improving the execution efficiency of robot navigation tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solution of the present application, the following is a brief introduction to the drawings required for the description. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0018] Figure 1 A flowchart of a target object search method based on node information provided by an embodiment of the present invention.

[0019] Figure 2 A schematic diagram of the process architecture of another target object search method based on node information provided by an embodiment of the present invention.

[0020] Figure 3 A schematic diagram of a layered scene graph.

[0021] Figure 4 A schematic block diagram of the structure of a target object search system based on node information provided by an embodiment of the present invention.

[0022] Figure 5 A schematic diagram of the structure of a terminal provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0023] In order to make the application objectives, features, and advantages of this application more obvious and easy to understand, the technical solutions protected by this application will be clearly and completely described below using specific embodiments and drawings. Obviously, the embodiments described below are only part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0024] Unless otherwise defined, all technical and scientific terms used in this application have the same meaning as those commonly understood by those skilled in the art to which the present invention pertains. The terms used in this application and in the specification of the present invention are for the purpose of describing specific embodiments only and are not intended to limit the present invention.

[0025] Figure 1 A flowchart of a target object search method based on node information provided by an embodiment of the present invention. Figure 1 The execution entity may be a node information-based target object search system. The node information-based target object search method provided in embodiments of the present invention is executed by a computer device. Accordingly, the node information-based target object search system runs on the computer device. The order of the steps in this flowchart may be changed, and some steps may be omitted, depending on different needs.

[0026] like Figure 1 As shown, the method includes the following steps.

[0027] S1, constructing a hierarchical scene graph, wherein the node types of the hierarchical scene graph are buildings, floors, rooms, and objects from top to bottom.

[0028] This step constructs a multi-level spatial map, clarifies the subordinate relationships of each element in the scene, and forms a four-level structural system of "building-floor-room-object".

[0029] S2, use the large language model to refine the object information in the room node, and use the refined information as the node information of the room node.

[0030] This step utilizes a large language model (LLM) to extract object-related information from room nodes and use it as node attribute data. Specifically, a node information extraction prompt is constructed to guide the LLM to focus on key information. The LLM then traverses all room nodes, extracting their attributes and the set of object nodes they contain. This data is then input into the LLM to generate semantically defined node information. This allows the LLM to proactively extract semantic features of objects within the room, rather than relying solely on geometric location information. Leveraging the LLM's semantic understanding capabilities, descriptive data containing object information is generated for each room node, enhancing the scene graph's semantic expressiveness and improving search accuracy.

[0031] S3, folding the layered scene graph to obtain a simplified scene graph, wherein the simplified scene graph only includes buildings, floors, and rooms; and then adding corresponding node information to each room node in the simplified scene graph.

[0032] This step collapses the hierarchical scene graph, retaining only the three-level structure of "building-floor-room," hiding object nodes to form a streamlined graph, reducing data volume and improving computational efficiency. At the same time, it retains key semantic information (room node information) to ensure that the LLM can still perform reasoning based on semantic features.

[0033] S4, in response to the navigation task instruction, inputs the simplified scene graph with attached node information and the navigation task instruction into the large language model, so that the large language model locates the room node that matches the navigation task.

[0034] This step prioritizes locating the room most likely to contain the target object, reducing invalid searches.

[0035] S5, expand the located room node and use the large language model to find the target object from the current room node.

[0036] Furthermore, as a refinement and extension of the specific implementation of the above embodiment, in order to fully illustrate the specific implementation process of this embodiment, another target object search method based on node information is provided. Figure 2This is a schematic diagram of the proposed method's process architecture. First, a hierarchical scene graph data structure is constructed based on the scene's geometric structure and the relative positions of objects. The graph data for the robot's scene is represented from top to bottom as a "building-floor-room-object" topology. Second, leveraging the reasoning capabilities of the large language model, the large language model extracts subnode information from each room node in the scene graph to describe the scene information below that room node. Then, based on a limited number of tokens in the large language model, the scene graph is collapsed to construct a simplified scene graph. The collapsed scene data is represented by the node information extracted by the large language model. Finally, when the user issues a task instruction, the large language model uses the node information in the simplified scene graph to quickly locate the room node that is highly consistent with the task instruction. Starting from this room node, the large language model then sequentially expands the subnodes under that node from top to bottom. The large language model then confirms whether the environmental information in the subnodes meets the task instruction requirements. Thus, by leveraging the node information in the simplified hierarchical scene graph, the large language model's scene reasoning is assisted, quickly locating the target object, improving the large language model's scene search capabilities, and enhancing the performance of scene graph navigation.

[0037] The method specifically includes the following steps.

[0038] SS1, build a hierarchical scene graph.

[0039] During the robot's driving process, the onboard visual camera and IMU perceive scene information, construct a graph node for each object in the scene, and construct the relative position relationship between objects as a graph edge connecting the nodes. Based on the geometric structure of the scene and the relative position relationship between objects, the graph is divided into a hierarchical structure of "building-floor-room-object". Figure 3 A schematic diagram of a layered scene graph.

[0040] SS2, extracts the node information of each room node.

[0041] As the scale of the scene increases and the number of objects in the scene increases, the amount of data in the hierarchical scene graph will increase dramatically. Considering the limited number of tokens in the large language model, when the large language model searches for the location of the target object in the hierarchical scene graph according to the task instruction, it is impractical to input all the data of the hierarchical scene graph into the large language model. At the same time, for specific navigation task instructions, the scene graph nodes involved in the task are limited. Only the part of the scene graph data related to the task instruction needs to be input into the large language model to locate the location of the target object. Therefore, a method for adding node information to the scene graph is designed. Based on the large language model, the sub-nodes of the lower layer of the room node of the hierarchical scene graph are refined, and the node information is used to display the subordinate scene information of the room node. When the large language model searches for a scene for a specific task instruction, the lower layer nodes of the hierarchical scene graph are first folded to obtain a simplified scene graph, and the node information is attached to the corresponding node to represent the lower layer sub-node attributes of the node, which is used to guide the large language model to perform scene reasoning and improve the efficiency of target search. Specifically, it includes the following steps.

[0042] SS21 uses a large language model to extract object information in room nodes and uses the extracted information as the node information of the room nodes.

[0043] SS211, constructing prompt words for node information extraction; SS212, traverse each room node in the hierarchical scene graph, obtain the attributes of the room node and the set of all lower-level nodes it contains; SS213 inputs the attributes of the room node, the set of all lower-level nodes it contains, and the prompt words into the large language model, so that the large language model can be refined to the node information of the room node.

[0044] The prompt for extracting node information from a large language model is shown below.

[0045] Agent role: You are an information extraction agent that can summarize the scene content of the node in one sentence based on the information provided in the scene graph.

[0046] Scene Graph: A 3D scene graph described in JSON format.

[0047] Output Response Format: "Node information": Use one sentence to extract the scene information under this node.

[0048] Example: {'role': 'user', 'content': { 'Scene Graph' : {"nodes": [{"floorB": {"room": [{"objects":[{"7":"bottle"},{"18": "bowl"}, {"21": "bowl"}, {"55": "pottedplant"}, {"56": "potted plant"}, {"57": "potted plant"}, {"67": "diningtable"}, {"68": "dining table"},{"77": "microwave"}, {"78": "oven"}, {"79": "oven"}, {"80": "oven"}, {"83": "oven"}, {"88": "sink"}, {"93": "sink"}, {"99": "refrigerator"}, {"117": "vase"}], "20": "kitchen"}]}}]} }}, {'role': 'assistant', 'content': { Node information: 'The room is filled with a large number of tableware and kitchen utensils.' }} Among them, the role defines the role positioning of the large language model and is an information extraction agent; the scene graph is a JSON-formatted text data that describes the current scene information and consists of objects, rooms, floors and their corresponding attributes; the example lists input and output references.

[0049] Based on large language model Refining room nodes Lower-level scene data, node information Defined as: , (1) Where: Room Node Attributes; Room Node The set of all lower-level nodes contained in ; The defined prompt is used to guide the large language model to extract node information; Generate functions for large language model nodes.

[0050] The attributes of the room node can be the structured description information of the room node in the hierarchical scene graph, including basic identification information, spatial attributes, semantic attributes, and dynamic information (such as update timestamps and relationships with other nodes).

[0051] SS22, fold the layered scene graph to obtain a simplified scene graph, which only contains buildings, floors and rooms; then add corresponding node information to each room node in the simplified scene graph.

[0052] The following is a scene graph without node information: {"nodes": [ {"floorA": {"room": [ {"1": "kitchen"}, {"2": "home_office"}, ...]}}, {"floorB": {"room": [ {"11": "playroom"}, {"12": "bedroom"}, ...]}}, … ]} The following is a scene graph containing node information: {"nodes": [ {"floorA": {"room": [ {"1": "kitchen", "node_information": "There are a lot of fruits here."}, {"2": "home_office", "node_information": "You can work here."}, ...]}}, {"floorB": {"room": [ {"11": "playroom", "node_information": "This is a good place to play."}, {"12": "bedroom", "node_information": "A lot of porcelain is stored here."}, ...]}} It should be noted that the hierarchical scene graph is updated in real time, and then the node information of the room nodes is updated in real time, and the latest node information is appended to the streamlined scene graph to improve the accuracy of target search.

[0053] SS3, target search using large language models.

[0054] During target search, the scene graph, the robot's current position, task instructions and other information are used as input, and a large language model is used to determine the room where the target object is located in the scene graph. The simplified scene graph with additional node information is input into the large language model, which not only retains the main structure of the scene graph but also reduces the amount of scene data input to the large language model. Based on the node information, the large language model quickly locates the node that is highly consistent with the task instruction, and expands the subnodes under the node from top to bottom starting from the node. The large language model again confirms whether the environmental information in the subnode meets the requirements of the task instruction. When the target object is found, the target search is completed. It specifically includes the following steps.

[0055] SS31, in response to the navigation task instruction, inputs the simplified scene graph with attached node information and the navigation task instruction into the large language model, so that the large language model locates the room node that matches the navigation task.

[0056] In some optional implementations, a large language model may be used to locate a room node that best matches the navigation task, and then the target object may be searched for in the room node.

[0057] In some optional implementations, the large language model outputs a set of several room nodes with the highest probability of matching the navigation task, with the room nodes in this set sorted from highest to lowest probability of matching. For example, a set of three room nodes {room node A, room node B, room node C} may be output, with the probability of matching decreasing in descending order.

[0058] SS32, expands the located room node and uses the large language model to find the target object from the current room node.

[0059] If only one most suitable room node is located, expand the room node and use the large language model to find the target object from the current room node. If the target object cannot be found, an error prompt will be issued.

[0060] If multiple room nodes are located, the room node with the highest probability of matching is selected from the room node set, the room node is expanded, and the target object is searched from the room node using the large language model; if the target object is found, the target object search program is terminated; if the target object is not found, the room node with the second highest probability of matching is selected from the room node set, and the target object is searched from the room node using the large language model, and so on, until the target object is found or all room nodes in the set are traversed; if all room nodes in the set are traversed and the target object is still not found, an error prompt is given.

[0061] In some optional implementations, to further improve the efficiency of target object search, after finding the target object for each navigation task instruction, a mapping relationship is constructed between the navigation task instruction and the room node where the target object is located and stored in the target area. When a new navigation task instruction is received, in response to the current navigation task instruction, a navigation task instruction whose semantic similarity with the current navigation task instruction exceeds a threshold is searched from the target area; if found, the room node is located according to the mapping relationship and recorded as a memory room node; it is detected whether the room node set contains the memory room node; if it does, the memory weight factor of the memory room node is added to the new fit probability, and then all room nodes in the set are re-sorted according to the fit probability; if it does not, the memory weight factor of the memory room node is added to the set as its fit probability, and all room nodes in the set are re-sorted according to the fit probability. This process takes into account the memory room node, combines real-time reasoning with historical data, avoids blind reliance on the instantaneous output of the LLM, and improves the efficiency of target object search.

[0062] For example, the navigation task instruction is "Help me find a ball", the initial LLM output is: {"playroom": 0.8, "bedroom": 0.3}, the memory record is: {"bedroom": ["ball"]} (the ball was found here before), and the memory weight factor is 0.6. Then, after adjustment, {"bedroom": 0.9, "playroom": 0.8} is obtained. After adjustment, the probability of the bedroom is greater than that of the living room, so the bedroom is searched first.

[0063] In an optional embodiment, the memory weight factor is calculated by the following formula,

[0064] Where, is the current memory weight factor, is the initial weight base, is the attenuation coefficient, It is the duration from the last time the memory was verified to the current moment. The time point of the last verification is also the timestamp of the last time the target object was successfully found in the room.

[0065] Among them, the initial weight base It can be adjusted according to the historical hit frequency, such as high-frequency hits =1.2, low frequency =0.8. Hit frequency refers to the ratio of the number of times the target is successfully found in the room to the total number of times the room is searched. In some optional implementations, historical hit records are attached to room nodes. Specifically, the navigation task instructions corresponding to hitting the room node are obtained based on the most recent hit records for the room node. A large language model is used to extract memory hit information from the obtained navigation task instructions. The corresponding memory hit information is attached to each room node in the simplified scene graph. In response to the navigation task instruction, the simplified scene graph with attached node information, memory hit information, and navigation task instruction are input into the large language model, allowing the large language model to locate the room node that matches the navigation task. For example, if the memory hit information is "found a ball item three times," the LLM can consider historical data during the initial probability ranking, reducing the need for subsequent adjustments and further improving search efficiency.

[0066] After identifying the target object and its corresponding room through target search, the large language model generates an action sequence to complete the task. The pose of the target for each action sequence is then obtained from the scene graph. Navigation to the target location is performed using the MoveBase package. For grasping actions, the robot arm and object recognition perform the corresponding grasping operations.

[0067] An embodiment of a target object search method based on node information is described in detail above. Based on the target object search method based on node information described in the above embodiment, an embodiment of the present invention also provides a target object search system based on node information corresponding to the method.

[0068] Figure 4 This is a block diagram of the structure of a node information-based target object search system according to an embodiment of the present invention. In this embodiment, the node information-based target object search system 400 can be divided into multiple functional modules according to their functions. A module, as used herein, refers to a series of computer program segments that can be executed by at least one processor and perform a fixed function, and is stored in a memory.

[0069] The hierarchical scene graph construction module 410 is used to construct a hierarchical scene graph, wherein the node types of the hierarchical scene graph are buildings, floors, rooms, and objects from top to bottom.

[0070] The node information extraction module 420 is configured to extract object information in the room node using a large language model, and use the extracted information as the node information of the room node.

[0071] The scene graph simplification module 430 is used to fold the layered scene graph to obtain a simplified scene graph, which only contains buildings, floors and rooms; and then add corresponding node information to each room node in the simplified scene graph.

[0072] The target object search module 440 is used to respond to navigation task instructions, input the simplified scene graph with attached node information and navigation task instructions into the large language model, so that the large language model can locate the room node that matches the navigation task; expand the located room node, and use the large language model to find the target object in the current room node.

[0073] In some optional implementations, the target object search module 440 is further configured to output a set of room nodes with the highest probability of matching the navigation task using a large language model, and to sort the room nodes in the set from high to low according to the probability of matching.

[0074] In some optional embodiments, the target object search module 440 is also used to construct a mapping relationship between the navigation task instruction and the room node where the target object is located and store it in the target area; in response to the current navigation task instruction, search for a navigation task instruction whose semantic similarity with the current navigation task instruction exceeds a threshold from the target area; if found, locate the room node according to the mapping relationship and record it as a memory room node; detect whether the room node set contains the memory room node; if it contains, add the memory weight factor of the memory room node to the matching probability as the new matching probability, and then re-sort all the room nodes in the set according to the matching probability; if it does not contain, add the memory weight factor of the memory room node as its matching probability to the set, and re-sort all the room nodes in the set according to the matching probability.

[0075] In some optional embodiments, the system 400 also includes a memory hit information extraction module 450, which is used to obtain the navigation task instructions corresponding to the hit of the room node based on the latest hit records of the room node; and use the large language model to extract the memory hit information from the obtained corresponding navigation task instructions. Correspondingly, the scene graph simplification module 430 is also used to attach corresponding memory hit information to each room node in the simplified scene graph. The target object search module 440 is used to respond to the navigation task instruction, input the simplified scene graph with attached node information, memory hit information, and navigation task instruction into the large language model, so that the large language model locates the room node that matches the navigation task.

[0076] The target object search system based on node information of this embodiment is used to implement the aforementioned target object search method based on node information. Therefore, the specific implementation method of this system can be seen in the embodiment part of the target object search method based on node information in the previous text. Therefore, its specific implementation method can refer to the description of the corresponding embodiments of each part and will not be elaborated here.

[0077] In addition, since the target object search system based on node information of this embodiment is used to implement the aforementioned target object search method based on node information, its function corresponds to that of the aforementioned method and will not be described in detail here.

[0078] Figure 5 A schematic diagram of the structure of a terminal 500 provided in an embodiment of the present invention includes: a processor 510, a memory 520, and a communication unit 530. The processor 510 is configured to implement the following steps when executing a target object search program based on node information stored in the memory 520: Constructing a hierarchical scene graph, wherein the node types of the hierarchical scene graph are buildings, floors, rooms, and objects from top to bottom; Use the large language model to extract the object information in the room node, and use the extracted information as the node information of the room node; Folding the layered scene graph to obtain a simplified scene graph, wherein the simplified scene graph only includes buildings, floors, and rooms; and then attaching corresponding node information to each room node in the simplified scene graph; In response to navigation task instructions, the simplified scene graph with attached node information and navigation task instructions are input into the large language model, so that the large language model locates the room node that matches the navigation task; Expand the located room node and use the large language model to find the target object in the current room node.

[0079] The terminal 500 includes a processor 510, a memory 520, and a communication unit 530. These components communicate via one or more buses. Those skilled in the art will appreciate that the server structure shown in the figure does not limit the present invention; it may be a bus structure or a star structure, and may include more or fewer components than shown, or combine certain components, or arrange the components differently.

[0080] The memory 520 can be used to store the execution instructions of the processor 510, and the memory 520 can be implemented by any type of volatile or non-volatile storage terminal or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk or optical disk. When the execution instructions in the memory 520 are executed by the processor 510, the terminal 500 can execute part or all of the steps in the following method embodiments.

[0081] The processor 510 is the control center of the storage terminal, connects all parts of the electronic terminal through various interfaces and lines, executes the software programs and / or modules stored in the memory 520, and calls the data stored in the memory, so as to execute various functions of the electronic terminal and / or process data. The processor can be composed of an integrated circuit (IC), for example, can be composed of a single packaged IC, or can be composed of multiple packaged ICs with the same function or different functions. For example, the processor 510 can only include a central processing unit (CPU). In the embodiments of the application, the CPU can be a single operation core or can include multiple operation cores.

[0082] The communication unit 530 is used to establish a communication channel, so that the storage terminal can communicate with other terminals. Receive user data sent by other terminals or send user data to other terminals.

[0083] The application also provides a computer storage medium, and the storage medium can be a magnetic disk, an optical disk, a read-only memory (ROM) or a random access memory (RAM), etc.

[0084] The application also provides a computer storage medium, and the storage medium can be a magnetic disk, an optical disk, a read-only memory (ROM) or a random access memory (RAM), etc.

[0085] The computer storage medium stores a target object search program based on node information, and the target object search program based on node information is executed by the processor to implement the following steps: Constructing a hierarchical scene graph, wherein the node types of the hierarchical scene graph are buildings, floors, rooms, and objects from top to bottom; Use the large language model to extract the object information in the room node, and use the extracted information as the node information of the room node; Folding the layered scene graph to obtain a simplified scene graph, wherein the simplified scene graph only includes buildings, floors, and rooms; and then attaching corresponding node information to each room node in the simplified scene graph; In response to navigation task instructions, the simplified scene graph with attached node information and navigation task instructions are input into the large language model, so that the large language model locates the room node that matches the navigation task; Expand the located room node and use the large language model to find the target object in the current room node.

[0086] Those skilled in the art will clearly understand that the techniques in the embodiments of the present invention can be implemented using software plus a necessary general-purpose hardware platform. Based on this understanding, the technical solutions in the embodiments of the present invention, or the portion that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium such as a USB flash drive, a mobile hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk, among other media capable of storing program code, and includes instructions for causing a computer terminal (which can be a personal computer, a server, or a second terminal, a network terminal, etc.) to execute all or part of the steps of the methods described in various embodiments of the present invention.

[0087] In the several embodiments provided by the present invention, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the division of the units is merely a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0088] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0089] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0090] The above description of the disclosed embodiments is intended to enable one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention is not limited to the embodiments shown herein, but is intended to be construed in the widest manner consistent with the principles and novel features disclosed herein.

Claims

1. A target object search method based on node information, characterized in that: The following steps are involved: Constructing a hierarchical scene graph, wherein the node types of the hierarchical scene graph are buildings, floors, rooms, and objects from top to bottom; Use the large language model to extract the object information in the room node, and use the extracted information as the node information of the room node; Folding the layered scene graph to obtain a simplified scene graph, wherein the simplified scene graph only includes buildings, floors, and rooms; and then attaching corresponding node information to each room node in the simplified scene graph; In response to navigation task instructions, the simplified scene graph with attached node information and navigation task instructions are input into the large language model, so that the large language model locates the room node that matches the navigation task; Expand the located room node and use the large language model to find the target object in the current room node.

2. The target object search method based on node information according to claim 1, characterized in that: Use the large language model to extract object information in the room node, and use the extracted information as the node information of the room node, specifically including: Construct node information extraction prompt words; Traverse each room node in the hierarchical scene graph, obtain the properties of the room node and the set of all lower-level nodes it contains; The attributes of the room node, the set of all lower-level nodes it contains, and the prompt words are input into the large language model, so that the large language model can be refined to the node information of the room node.

3. The target object search method based on node information according to claim 1, characterized in that: The large language model is positioned at the room node that matches the navigation task, specifically including: The large language model outputs a set of room nodes with the highest probability of matching the navigation task, and the room nodes in this set are sorted from high to low according to the probability of matching.

4. The target object search method based on node information according to claim 3, characterized in that: Expand the located room node and use the large language model to find the target object in the current room node, including: Select the room node with the highest matching probability from the room node set, expand the room node, and use the large language model to search for the target object in the room node; If the target object is found, the target object search procedure is terminated; If the target object is not found, the room node with the second highest matching probability is selected from the room node set, and the target object is searched from this room node using the large language model. This process is repeated until the target object is found or all room nodes in the set are searched. If the target object is still not found after traversing all room nodes in the collection, an error message will be displayed.

5. The target object search method based on node information according to claim 3, characterized in that: The method further comprises the following steps: Construct a mapping relationship between the navigation task instructions and the room nodes where the target object is located and store it in the target area; In response to the current navigation task instruction, searching for a navigation task instruction whose semantic similarity with the current navigation task instruction exceeds a threshold from the target area; If found, it is located at the room node according to the mapping relationship and recorded as the memory room node; Check whether the room node set contains the memory room node; If included, add the memory weight factor to the matching probability of the memory room node as the new matching probability, and then re-sort all room nodes in the set according to the matching probability; If not included, the memory weight factor of the memory room node is added to the set as its matching probability, and all room nodes in the set are re-sorted according to the matching probability.

6. The target object search method based on node information according to claim 5, characterized in that: The memory weight factor is calculated by the following formula, Where, is the current memory weight factor, is the initial weight base, is the attenuation coefficient, The duration from the last time the memory was verified to the current moment.

7. The target object search method based on node information according to claim 1, characterized in that: The method further comprises the following steps: Obtain the navigation task instructions corresponding to hitting the room node based on the latest hit records of the room node; A large language model is used to extract memory hit information from several corresponding navigation task instructions. Attach corresponding memory hit information to each room node in the simplified scene graph; In response to navigation task instructions, the simplified scene graph with attached node information and memory hit information and the navigation task instructions are input into the large language model, so that the large language model locates the room node that matches the navigation task.

8. A target object search system based on node information, characterized in that: include: A hierarchical scene graph construction module is used to construct a hierarchical scene graph, wherein the node types of the hierarchical scene graph are buildings, floors, rooms, and objects from top to bottom; A node information extraction module is used to extract object information in room nodes using a large language model and use the extracted information as node information of the room nodes; A scene graph simplification module is used to fold the layered scene graph to obtain a simplified scene graph, which only contains buildings, floors, and rooms; and then add corresponding node information to each room node in the simplified scene graph; The target object search module is used to respond to navigation task instructions, input the simplified scene graph with attached node information and navigation task instructions into the large language model, so that the large language model can locate the room node that matches the navigation task; expand the located room node and use the large language model to find the target object in the current room node.

9. A terminal, characterized in that: include: A memory, used for storing a target object search program based on node information; A processor is configured to implement the steps of the target object search method based on node information as claimed in any one of claims 1 to 7 when executing the target object search program based on node information.

10. A computer-readable storage medium, characterized in that The readable storage medium stores a target object search program based on node information, and when the target object search program based on node information is executed by a processor, the steps of the target object search method based on node information as described in any one of claims 1 to 7 are implemented.

Images