Object searching method and device, equipment and storage medium
By predicting the location distribution of target objects through semantic information, generating the optimal scanning path to find and outputting location description information, this solves the problems of low search efficiency and poor user experience of traditional service robots, and achieves efficient and accurate object search and user-friendly interaction.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-12-17
- Publication Date
- 2026-03-27
AI Technical Summary
Traditional service robots are inefficient and have low accuracy in finding target objects, resulting in a poor user experience, especially in complex scenarios.
Predict the location distribution of target objects using semantic information, generate the optimal scanning path for searching, and output easily understandable location description information.
It improves object search efficiency, enhances user interaction experience, and reduces unnecessary scanning paths and time.
Smart Images
Figure CN121743564A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of artificial intelligence technology, and in particular to an object search method, apparatus, device and storage medium. Background Technology
[0002] When traditional service robots help users find target objects, they usually scan along a predefined fixed path or launch a large-scale exhaustive scan. Then, they perform simple keyword matching between the objects identified by the scan and the user's instructions. This method not only consumes a lot of search time, but also has a low recognition accuracy in complex scenarios (such as changes in the appearance of objects, the existence of similar objects, etc.), resulting in a poor user experience when interacting with service robots. Summary of the Invention
[0003] To address the aforementioned technical issues, this application discloses an object search method, apparatus, device, and storage medium. It predicts the possible location distribution of target objects based on semantic information, generates an optimal scanning path for searching, and outputs a corresponding spatial relationship description after discovering the target object, thereby improving object search efficiency and user interaction experience.
[0004] On the one hand, this application provides an object lookup method, the method comprising: In response to a target lookup command for a target object, the target lookup command is parsed to determine the target semantic information corresponding to the target object; If the preset spatial model does not include the preset object node corresponding to the target object, the position probability distribution information of the target object in the current space is determined based on the preset spatial model and the target semantic information; the preset spatial model is a three-dimensional model corresponding to the current space. The target scanning path corresponding to the target object is determined based on the location probability distribution information; Based on the target scanning path, locate the target object and determine the target location information of the target object; Based on the target location information, output the location description information corresponding to the target object.
[0005] In some embodiments, when the preset spatial model does not include the preset object node corresponding to the target object, determining the position probability distribution information of the target object in the current space based on the preset spatial model and the target semantic information includes: If no preset object node corresponding to the target object exists in the preset spatial model, the target attribute of the target object is determined based on the target semantic information; the target attribute includes the category information of the target object and / or the preset spatial relationship between the target object and at least one associated object in the current space. Based on the preset spatial model, the category information, and / or the preset spatial relationship, the location probability distribution information is determined.
[0006] In some embodiments, determining the location probability distribution information based on the preset spatial model, the category information, and / or the preset spatial relationship includes: Based on the category information, a first candidate placement area for the target object in the preset spatial model is determined; Based on the preset spatial relationship, determine the preset object node corresponding to the at least one associated object in the preset spatial model; Based on the preset object node corresponding to the at least one associated object in the preset space model and the preset space relationship, a second placement candidate area for the target object in the preset space model is determined. The location probability distribution information is determined based on the first placement candidate region and the second placement candidate region; the probability that the target object appears in the first placement candidate region and the second placement candidate region is greater than the probability that it appears in other regions, and the other regions are regions in the preset spatial model other than the first placement candidate region and the second placement candidate region.
[0007] In some embodiments, the step of locating the target object based on the target scanning path and determining the target location information of the target object includes: A target scanning instruction is generated based on the target scanning path; the target scanning instruction is used to instruct the scanning device to sequentially acquire scan images of each candidate placement area based on the target scanning path. Visual recognition is performed on the scanned image to determine the visual feature information of the candidate objects in each placement candidate area; Semantically align the target semantic information with the visual feature information to determine the candidate confidence of the candidate object; If the candidate confidence level is greater than a preset confidence threshold, the location information of the candidate object is determined as the target location information; Generate a scan stop command; the scan stop command is used to instruct the scanning device to stop the scanning operation.
[0008] In some embodiments, outputting location description information corresponding to the target object based on the target location information includes: Based on the target location information, determine the adjacent object nodes that are adjacent to the target object from the preset object nodes of the preset spatial model; Based on the location information of the adjacent object nodes and the target location information, the relative positional relationship between the target object and the objects corresponding to the adjacent object nodes is determined; The location description information is generated and output based on the relative positional relationship.
[0009] In some embodiments, after parsing the target lookup instruction in response to the target object and determining the target semantic information corresponding to the target object, the method further includes: When the preset spatial model includes a preset object node corresponding to the target object, the position information of the preset object node corresponding to the target object is determined as the target position information.
[0010] In some embodiments, the step of parsing the target lookup instruction in response to a target object and determining the target semantic information corresponding to the target object includes: If the target search instruction is a voice instruction, the voice instruction will be converted into a text instruction. The text instructions are semantically parsed to obtain the target semantic information.
[0011] On the other hand, this application also provides an object search apparatus, the apparatus comprising: The target semantic information determination module is used to parse the target search instruction for the target object in response to the target search instruction for the target object and determine the target semantic information corresponding to the target object; The location probability distribution information determination module is used to determine the location probability distribution information of the target object in the current space based on the preset spatial model and the target semantic information, when the preset spatial model does not include the preset object node corresponding to the target object; the preset spatial model is a three-dimensional model corresponding to the current space; The target scanning path determination module is used to determine the target scanning path corresponding to the target object based on the location probability distribution information. The target location information determination module is used to locate the target object based on the target scanning path and determine the target location information of the target object. The location description information output module is used to output the location description information corresponding to the target object based on the target location information.
[0012] On the other hand, this application also provides an electronic device, which includes a processor and a memory, wherein the memory stores at least one instruction or at least one program, and the at least one instruction or the at least one program is loaded and executed by the processor to implement the object lookup method as described above.
[0013] On the other hand, this application also provides a computer-readable storage medium storing at least one instruction or at least one program, which is loaded and executed by a processor to implement the object lookup method as described above.
[0014] Implementing the embodiments of this application has the following beneficial effects: The object search method disclosed in this application performs semantic parsing on the target search command issued by the user to determine the target semantic information corresponding to the target object that the user needs to find. If there is no preset object node corresponding to the entity target object in the preset spatial model corresponding to the current space, a visual search is required. Before the visual search, the possible location of the target object is predicted based on the target semantic information corresponding to the target object, and then the optimal scanning path is generated. The visual search is performed according to the optimal scanning path. After the target object is found, location description information that is easy for the user to understand is generated and output, thereby reducing unnecessary scanning paths, reducing the time required to find the target object, and improving the efficiency of object search. Through semantic analysis, the user's intent can be accurately identified, and thus the target object that the user needs to find can be accurately identified. Finally, the user is given easily understandable location description information, which improves the user interaction experience. Attached Figure Description
[0015] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0016] Figure 1 A flowchart illustrating an object lookup method provided in an embodiment of this application; Figure 2 A schematic diagram illustrating a process for robot-user interaction provided in an embodiment of this application; Figure 3 A flowchart illustrating a method for determining target location information provided in an embodiment of this application; Figure 4 This is a schematic diagram of the structure of an object search device provided in an embodiment of this application; Figure 5This is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of this application. Detailed Implementation
[0017] To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are merely some embodiments of this application, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort are within the scope of protection of this application.
[0018] In the description of this application, it should be understood that the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. Furthermore, the terms "first," "second," etc., are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such information can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in a sequence other than that illustrated or described herein.
[0019] See Figure 1 , Figure 1 This is a flowchart illustrating an object lookup method provided in an embodiment of this application. The method is applied to a robot, specifically to the robot's controller, and includes: S101, in response to a target search instruction for a target object, the target search instruction is parsed to determine the target semantic information corresponding to the target object; In some embodiments, the target object refers to the entity object that the user needs to find. The target search instruction is an instruction issued by the user, which contains information related to the target object. The target search instruction can be a voice instruction, a text instruction, or other forms of instruction. By performing semantic parsing on the target search instruction, the target semantic information corresponding to the target object can be obtained. The target semantic information includes the concept, attributes, functions represented by the language symbol of the target object, as well as the relationship between the language symbol and other concepts. For example, if the target object is a mouse, the semantic information of the language symbol "mouse" includes: (1) Core concept: a computer input device; (2) Category: electronic device, computer accessories, office supplies; (3) Function: control cursor, click, scroll; (3) Common attributes: usually have left and right buttons and scroll wheel, usually connected via USB or wirelessly; (2) Common relationships: usually appears together with "laptop", "desktop", "keyboard" and "mouse pad".
[0020] In some embodiments, semantic parsing of the target search instruction can be performed based on a large language model (LLM) to obtain a target semantic vector corresponding to the target object, which contains the target semantic information corresponding to the target object.
[0021] S103, if the preset spatial model does not include the preset object node corresponding to the target object, determine the position probability distribution information of the target object in the current space based on the preset spatial model and the target semantic information; the preset spatial model is a three-dimensional model corresponding to the current space; In some embodiments, a three-dimensional model corresponding to the current space is obtained by pre-modeling the current space, namely a preset space model. The preset space model includes multiple preset object nodes, and each preset object node corresponds to an entity object in the current space.
[0022] In some embodiments, the target object is matched with the preset object nodes included in the preset spatial model based on the target semantic information corresponding to the target object. If there is no preset object node corresponding to the target object in the preset spatial model, the position of the target object needs to be determined by visual search. Before performing visual search, the possible position of the target object needs to be predicted based on the target semantic information corresponding to the target object, that is, the position probability distribution information of the target object in the current space is determined based on the preset spatial model and the target semantic information corresponding to the target object.
[0023] S105, determine the target scanning path corresponding to the target object based on the location probability distribution information; In some embodiments, the position probability distribution information represents the probability that a target object appears in different regions of the current space. The higher the probability of a region, the higher the likelihood that the target object will appear in that region, and the higher the priority of visual search for that region. The target scanning path is planned based on the magnitude of each probability value in the position probability distribution information. Specifically, the probability values in the position probability distribution information are arranged in descending order to obtain a position probability sequence. The regions corresponding to each probability value in the position probability sequence are determined, resulting in a region sequence corresponding to the position probability sequence. Then, a target scanning path corresponding to the target object is generated based on the order of the regions in the region sequence. For example, if the probability of the target object appearing in region A is 60%, in region B is 25%, and in region C is 15%, then the target scanning path corresponding to the target object is A→B→C.
[0024] S107, Based on the target scanning path, locate the target object and determine the target location information of the target object; In some embodiments, the controller controls the scanning device with an image acquisition device to acquire images according to the target scanning path. The scanning device sends the acquired images to the controller. The controller identifies the target object by recognizing the image, and then determines the target location information of the target object. After the controller identifies the target object, it controls the scanning device to stop the scanning operation.
[0025] S109, based on the target location information, output the location description information corresponding to the target object.
[0026] In some embodiments, the target location information of the target object is the three-dimensional coordinates of the target object in the current space. After determining the target location information of the target object, the target location information is converted into location description information that is easy for users to understand. The location description information represents the relative positional relationship between the target object and other objects in the current space.
[0027] In some embodiments, location description information can be fed back to the user in the form of voice or text. If the robot has interactive action capabilities, it can also perform subsequent actions based on user commands, such as pointing to the target object, marking the location of the target object with the screen or laser, or helping to move the target object to the user.
[0028] In some embodiments, the robot includes a Speech Interaction Module, a Large Model Decision Module (LLM Agent & Planner), a Modular Capability Protocol Manager (MCP), a Multimodal Perception Module, an Action & Motion Control Module, and a State Manager & Execution Engine.
[0029] In some embodiments, the voice interaction module implements end-to-end voice input and output processing, supporting natural and fluent voice interaction; a locally running lightweight voice wake-up engine (such as Porcupine) supports custom wake-up words such as "Hello Xiaodong"; it monitors user interruptions in real time to achieve natural dialogue interruption and switching; it achieves speech-to-text conversion through ASR (Automatic Speech Recognition); and it achieves text-to-speech conversion through TTS (Text-to-Speech) synthesis, supporting voice cloning.
[0030] In some embodiments, the large model decision module acts as the "brain" of the agent, responsible for understanding user intent, combining context and available capabilities, and generating an executable task plan. This includes the following aspects: natural language understanding and multi-turn dialogue management, integrating a memory module (short-term dialogue memory + long-term user preferences); task decomposition and planning based on the MCP tool list, supporting Function Calling / Tool Use mechanisms; generating execution flows including sequence, conditional branches, loops, and exception handling; and dynamically adjusting strategies (such as retries, degradation, and alternative solutions).
[0031] In some embodiments, the MCP capability management module provides a unified mechanism for capability abstraction, registration, discovery, and invocation, enabling plug-and-play functionality integration. It includes the following aspects: MCP server: running on the robot or edge gateway, managing the metadata of all available tools; MCP client: for LLM calls, used to query the capability list and initiate tool requests; Tool registry center: each functional module registers its own interface according to the MCP specification, supporting rapid access from third-party devices / services (such as facial expression LEDs), enabling capability sharing across robot platforms.
[0032] In some embodiments, the multimodal perception module integrates information from multiple sources, including vision, speech, gestures, and environmental sensors, to build a comprehensive understanding of the scene. This includes the following aspects: a Vision-Language Model (VLM) for image understanding, object recognition, and state determination; gesture recognition based on a camera to identify user handshakes, waves, and other actions; and environmental sensor access for data such as temperature, humidity, light intensity, and human body sensing for contextual reasoning.
[0033] In some embodiments, the behavior control module is responsible for executing the robot's physical actions, including movement, grasping, facial expression interaction, and device linkage. It includes the following aspects: Motion control system: SLAM (Simultaneous Localization and Mapping) navigation, path planning, obstacle avoidance, and robotic arm control; Facial and body feedback: Non-verbal interaction such as head rotation, screen expressions, and gesture simulation; IoT device control interface: Controlling smart home devices via Wi-Fi / Bluetooth / Zigbee protocols.
[0034] In some embodiments, the State and Execution Engine module acts as the system's "central scheduler," responsible for maintaining task states, parsing LLM-generated plans, executing tool calls in sequence, and handling feedback and exceptions. This includes: storing the current task context (user intent, executed steps, environment state); parsing the JSON-formatted execution plan output by the LLM; sequentially calling MCP tools, supporting synchronous / asynchronous execution; capturing execution results or errors and feeding them back to the LLM for further decision-making; and supporting process control such as pause, rollback, and retry.
[0035] In some embodiments, see Figure 2 , Figure 2 This is a schematic diagram illustrating a robot-user interaction process provided in an embodiment of this application. The user issues a voice command; the edge-side voice processing module collects the voice and performs wake-up detection; if a wake-up word is triggered, local ASR (Automatic Speech Recognition) is performed; the intent recognition module performs preliminary text classification to determine the command type; if it is a knowledge-based question (e.g., "Tell me about Company C's products"), a knowledge base retrieval process is triggered; if it is another question, the MCP client initiates a request to the MCP server to dynamically obtain a list of currently available functional tools, such as enabling visual assistance functions or navigation guidance functions; the LLM module integrates various information for task understanding and planning, generating a structured execution plan; the behavior decision module receives the generated tool call sequence and hands it over to the execution engine for sequential scheduling; the execution engine calls the corresponding functional modules through the MCP client; the TTS (Text-to-Speech) module converts the system response into voice output and then into voice playback; the behavior decision module generates robot action commands based on semantic and visual information; the system drive responds to the commands to complete the corresponding actions (e.g., nodding, turning, shaking hands). The entire process forms a task-level autonomous closed loop: "continuous perception—semantic understanding—task planning—MCP tool call—behavior execution—feedback".
[0036] In some embodiments, when the preset spatial model does not include the preset object node corresponding to the target object, determining the position probability distribution information of the target object in the current space based on the preset spatial model and the target semantic information includes: If no preset object node corresponding to the target object exists in the preset spatial model, the target attribute of the target object is determined based on the target semantic information; the target attribute includes the category information of the target object and / or the preset spatial relationship between the target object and at least one associated object in the current space. Based on the preset spatial model, the category information, and / or the preset spatial relationship, the location probability distribution information is determined.
[0037] In some embodiments, the target object is matched with preset object nodes included in the preset spatial model based on the target semantic information corresponding to the target object. If there is no preset object node corresponding to the target object in the preset spatial model, the position of the target object needs to be determined by visual search. Before performing visual search, the possible position of the target object needs to be predicted based on the target semantic information corresponding to the target object. Specifically, the target attributes of the target object are determined based on the target semantic information. The target attributes include the category information of the target object and / or the preset spatial relationship between the target object and at least one associated object in the current space. The preset spatial model includes the preset object nodes corresponding to the associated objects. The preset spatial relationship refers to the typical positional relationship between different objects. For example, if the target object is a mouse, the category information of the target object is electronic office equipment, and the associated object can be a laptop. The preset spatial relationship between the target object and at least one associated object in the current space means that the mouse usually appears in the right area of the laptop.
[0038] If, in this embodiment of the application, there is no preset object node corresponding to the entity target object in the preset spatial model corresponding to the current space, a visual search is required. Before the visual search, the possible location of the target object is predicted based on the category information of the target object and / or the preset spatial relationship between the target object and at least one associated object in the current space. Then, an optimal scanning path is generated, and a visual search is performed according to the optimal scanning path. This reduces unnecessary scanning paths, reduces the time required to find the target object, and improves the object search efficiency.
[0039] In some embodiments, determining the location probability distribution information based on the preset spatial model, the category information, and / or the preset spatial relationship includes: Based on the category information, a first candidate placement area for the target object in the preset spatial model is determined; Based on the preset spatial relationship, determine the preset object node corresponding to the at least one associated object in the preset spatial model; Based on the preset object node corresponding to the at least one associated object in the preset space model and the preset space relationship, a second placement candidate area for the target object in the preset space model is determined. The location probability distribution information is determined based on the first placement candidate region and the second placement candidate region; the probability that the target object appears in the first placement candidate region and the second placement candidate region is greater than the probability that it appears in other regions, and the other regions are regions in the preset spatial model other than the first placement candidate region and the second placement candidate region.
[0040] In some embodiments, a first candidate placement area for the target object in a preset spatial model is determined based on the target object's category information. For example, if the target object is a mouse and its category information is electronic office equipment, then the first candidate placement area for the mouse can be determined to be the desktop area. A preset object node corresponding to at least one associated object of the target object in the preset spatial model is determined based on a preset spatial relationship. A second candidate placement area for the target object in the preset spatial model is determined based on the preset object node corresponding to at least one associated object in the preset spatial model and the preset spatial relationship between the target object and at least one associated object in the current space. For example, if the target object is a mouse, the associated object is a laptop, the preset spatial model includes a preset object node corresponding to the laptop, and the preset spatial relationship is that the mouse usually appears in the right-side area of the laptop, then the second candidate placement area for the mouse can be determined to be the right-side area of the laptop based on the preset spatial relationship between the mouse and the laptop. The probability distribution information of the target object's location is determined based on the first and second candidate placement areas, where the probability of the target object appearing in the first and second candidate placement areas is greater than the probability of it appearing in other areas of the preset spatial model.
[0041] If, in this embodiment of the application, there is no preset object node corresponding to the entity target object in the preset spatial model corresponding to the current space, a visual search is required. Before the visual search, the possible location of the target object is predicted based on the category information of the target object and / or the preset spatial relationship between the target object and at least one associated object in the current space. Then, an optimal scanning path is generated, and a visual search is performed according to the optimal scanning path. This reduces unnecessary scanning paths, reduces the time required to find the target object, and improves the object search efficiency.
[0042] In some embodiments, see Figure 3 , Figure 3 This is a flowchart illustrating a method for determining target location information provided in an embodiment of this application. The step of locating the target object based on the target scanning path and determining the target location information of the target object includes: S301, Generate a target scanning instruction based on the target scanning path; the target scanning instruction is used to instruct the scanning device to sequentially acquire scan images of each candidate placement area based on the target scanning path; In some embodiments, the controller generates a target scanning instruction based on the target scanning path corresponding to the target object and sends the target scanning instruction to the scanning device, so that the scanning device sequentially acquires the scanned images of each placement candidate area based on the target scanning path. The placement candidate area refers to the area included in the target scanning path.
[0043] S303, Perform visual recognition on the scanned image to determine the visual feature information of the candidate objects in each placement candidate area; In some embodiments, the scanning device sends the acquired scanned image to the controller, which performs visual recognition on the scanned image, determines at least one candidate object in the candidate area, and extracts visual feature information of at least one candidate object.
[0044] S305, Semantically align the target semantic information with the visual feature information to determine the candidate confidence of the candidate object; In some embodiments, the target semantic information corresponding to the target object is semantically aligned with the visual feature information of at least one candidate object to determine the candidate confidence level corresponding to each candidate object. The candidate confidence level represents the similarity between the candidate object and the target object.
[0045] S307, if the candidate confidence level is greater than a preset confidence threshold, the location information of the candidate object is determined as the target location information; In some embodiments, if the candidate confidence level of a candidate object is greater than a preset confidence threshold, the candidate object can be considered as the target object. The position information of the candidate object in the preset spatial model is back-projected onto the 3D world coordinate system of the current space to obtain the three-dimensional coordinates of the candidate object in the current space, that is, the target position information of the target object.
[0046] S309, Generate a scan stop command; the scan stop command is used to instruct the scanning device to stop the scanning operation.
[0047] In some embodiments, after the controller identifies the target object, it generates a scan stop command and sends it to the scanning device so that the scanning device stops scanning and does not need to scan other areas in the current space.
[0048] In this embodiment, before visual search, the possible location of the target object is predicted based on the target semantic information corresponding to the target object, and then the optimal scanning path is generated. Visual search is performed according to the optimal scanning path, and the target object is determined from the candidate objects found based on the confidence level. After the target object is identified, the scanning operation is stopped, thereby reducing unnecessary scanning paths, reducing the time required to find the target object, and improving the object search efficiency.
[0049] In some embodiments, outputting location description information corresponding to the target object based on the target location information includes: Based on the target location information, determine the adjacent object nodes that are adjacent to the target object from the preset object nodes of the preset spatial model; Based on the location information of the adjacent object nodes and the target location information, the relative positional relationship between the target object and the objects corresponding to the adjacent object nodes is determined; The location description information is generated and output based on the relative positional relationship.
[0050] In some embodiments, the target location information of the target object is its three-dimensional coordinates within the current space. After determining the target location information, it needs to be converted into location description information that is easy for users to understand. The location description information represents the relative positional relationship between the target object and other objects in the current space. Specifically, neighboring object nodes corresponding to adjacent objects near the target object are determined from a preset spatial model. Based on the target location information of the target object and the location information of the neighboring object nodes, the relative positional relationship between the target object and its neighboring objects is determined, and the relative positional relationship is fed back to the user in the form of voice or text. For example, if the target object is a mouse, the neighboring objects of the target object could be objects such as a laptop or a cup. The location description information could be, "I found it. Your mouse is on the right side of the desktop, slightly to the right front of the laptop, near the cup." This application embodiment generates and outputs location description information that is easy for users to understand after discovering the target object, thereby improving the user interaction experience.
[0051] In some embodiments, after parsing the target lookup instruction in response to the target object and determining the target semantic information corresponding to the target object, the method further includes: When the preset spatial model includes a preset object node corresponding to the target object, the position information of the preset object node corresponding to the target object is determined as the target position information.
[0052] In some embodiments, if a preset object node corresponding to the target object exists in the preset spatial model, the position information of the preset object node corresponding to the target object is determined as the target position information. The position information of each preset object node in the preset spatial model can be updated periodically or after the position of the entity object corresponding to each preset object node changes.
[0053] This application embodiment performs semantic parsing on the target search command issued by the user to determine the target semantic information corresponding to the target object that the user needs to search for. Through semantic analysis, the user's intent can be accurately identified, and thus the target object that the user needs to search for can be accurately identified.
[0054] In some embodiments, the step of parsing the target lookup instruction in response to a target object and determining the target semantic information corresponding to the target object includes: If the target search instruction is a voice instruction, the voice instruction will be converted into a text instruction. The text instructions are semantically parsed to obtain the target semantic information.
[0055] In some embodiments, if the target user issues a target search command as a voice command, the voice command is first converted into a text command using ASR, and then the text command is semantically parsed based on LLM to obtain the target semantic information corresponding to the target object.
[0056] This application embodiment performs semantic parsing on the target search command issued by the user to determine the target semantic information corresponding to the target object that the user needs to search for. Through semantic analysis, the user's intent can be accurately identified, and then the target object that the user needs to search for can be accurately identified, ultimately realizing cross-modal search from voice perception, visual search to language feedback.
[0057] This application provides an object search method, the method comprising: responding to a target search instruction for a target object, parsing the target search instruction to determine target semantic information corresponding to the target object; determining the position probability distribution information of the target object in the current space based on the preset spatial model and the target semantic information, when a preset spatial model does not include a preset object node corresponding to the target object; the preset spatial model being a three-dimensional model corresponding to the current space; determining a target scanning path corresponding to the target object based on the position probability distribution information; searching for the target object based on the target scanning path to determine the target position information of the target object; and outputting position description information corresponding to the target object based on the target position information. This application performs semantic parsing on the user's target search command to determine the target semantic information corresponding to the target object the user needs to find. If the preset object node corresponding to the entity target object does not exist in the preset spatial model corresponding to the current space, a visual search is required. Before the visual search, the possible location of the target object is predicted based on the target semantic information corresponding to the target object, and then the optimal scanning path is generated. The visual search is performed according to the optimal scanning path. After the target object is found, location description information that is easy for the user to understand is generated and output, thereby reducing unnecessary scanning paths, reducing the time required to find the target object, and improving the efficiency of object search. Through semantic analysis, the user's intent can be accurately identified, and thus the target object that the user needs to find can be accurately identified. Finally, the user is given easily understandable location description information, which improves the user interaction experience.
[0058] This application also provides an object search device, see [link to relevant documentation]. Figure 4 , Figure 4 This is a schematic diagram of an object search device provided in an embodiment of this application. The device includes: The target semantic information determination module 410 is used to parse the target search instruction for the target object in response to the target search instruction for the target object and determine the target semantic information corresponding to the target object; The position probability distribution information determination module 420 is used to determine the position probability distribution information of the target object in the current space based on the preset spatial model and the target semantic information, when the preset spatial model does not include the preset object node corresponding to the target object; the preset spatial model is a three-dimensional model corresponding to the current space; The target scanning path determination module 430 is used to determine the target scanning path corresponding to the target object based on the location probability distribution information. The first target location information determination module 440 is used to locate the target object based on the target scanning path and determine the target location information of the target object. The location description information output module 450 is used to output the location description information corresponding to the target object based on the target location information.
[0059] In some embodiments, the location probability distribution information determination module 420 includes: The target attribute determination unit is used to determine the target attribute of the target object based on the target semantic information when there is no preset object node corresponding to the target object in the preset spatial model; the target attribute includes the category information of the target object and / or the preset spatial relationship between the target object and at least one associated object in the current space; The location probability distribution information determination unit is used to determine the location probability distribution information based on the preset spatial model, the category information, and / or the preset spatial relationship.
[0060] In some embodiments, the location probability distribution information determination unit includes: The first placement candidate region determination subunit is used to determine the first placement candidate region of the target object in the preset spatial model based on the category information; The associated object node determination subunit is used to determine the preset object node corresponding to the at least one associated object in the preset spatial model based on the preset spatial relationship. The second placement candidate region determination subunit is used to determine the second placement candidate region of the target object in the preset space model based on the preset object node corresponding to the at least one associated object in the preset space model and the preset space relationship. The location probability distribution information determination subunit is used to determine the location probability distribution information based on the first placement candidate region and the second placement candidate region; the probability that the target object appears in the first placement candidate region and the second placement candidate region is greater than the probability that it appears in other regions, and the other regions are regions in the preset spatial model other than the first placement candidate region and the second placement candidate region.
[0061] In some embodiments, the first target location information determination module 440 includes: A target scanning instruction generation unit is used to generate a target scanning instruction based on the target scanning path; the target scanning instruction is used to instruct the scanning device to sequentially acquire scan images of each candidate placement area based on the target scanning path. A visual recognition unit is used to perform visual recognition on the scanned image and determine the visual feature information of candidate objects in each placement candidate area; A semantic alignment unit is used to semantically align the target semantic information with the visual feature information to determine the candidate confidence of the candidate object; The target location information determination unit is used to determine the location information of the candidate object as the target location information when the candidate confidence is greater than a preset confidence threshold. A scan stop command generation unit is used to generate a scan stop command; the scan stop command is used to instruct the scanning device to stop the scanning operation.
[0062] In some embodiments, the location description information output module 450 includes: The adjacent object node determination unit is used to determine the adjacent object node that is adjacent to the target object from the preset object nodes of the preset spatial model based on the target location information; The relative position relationship determination unit is used to determine the relative position relationship between the target object and the object corresponding to the adjacent object node based on the position information of the adjacent object node and the target position information; A location description information output unit is used to generate and output the location description information based on the relative positional relationship.
[0063] In some embodiments, the apparatus further includes: The second target location information determination module is used to determine the location information of the preset object node corresponding to the target object as the target location information when the preset spatial model includes the preset object node corresponding to the target object.
[0064] In some embodiments, the target semantic information determination module 410 includes: A voice command conversion unit is used to convert the voice command into a text command when the target search command is a voice command. The semantic parsing unit is used to perform semantic parsing on the text instructions to obtain the target semantic information.
[0065] The apparatus provided in the above embodiments can execute the method provided in any embodiment of this application, and has the corresponding functional modules and beneficial effects for executing the method. Technical details not described in detail in the above embodiments can be found in an object search method provided in any embodiment of this application.
[0066] This embodiment also provides a computer-readable storage medium storing computer-executable instructions, which are loaded by a processor and executed by the object lookup method described above in this embodiment.
[0067] This embodiment also provides an electronic device, which includes a processor and a memory, wherein the memory stores a computer program adapted to be loaded by the processor and executed as described above in this embodiment of an object lookup method.
[0068] The electronic device may be a computer terminal, a mobile terminal, or a server, and may also participate in constituting the apparatus or system provided in the embodiments of this application. For example... Figure 5 As shown, the electronic device 5 may include one or more (shown as 502a, 502b, ..., 502n in the figure) processors 502 (processors 502 may include, but are not limited to, microprocessors MCUs or programmable logic devices FPLDs), a memory 504 for storing information, and a transmission device 506 for communication functions. In addition, it may also include input / output interfaces (I / O interfaces) and network interfaces. Those skilled in the art will understand that... Figure 5 The structure shown is for illustrative purposes only and does not limit the structure of the aforementioned electronic device. For example, electronic device 5 may also include... Figure 5 The more or fewer components shown, or having the same Figure 5 The different configurations shown.
[0069] It should be noted that the aforementioned one or more processors 502 and / or other information processing circuits are generally referred to herein as "information processing circuits". These information processing circuits may be wholly or partially embodied in software, hardware, firmware, or any other combination thereof. Furthermore, the information processing circuits may be a single, independent processing module, or may be wholly or partially integrated into any other element in the electronic device 5.
[0070] The memory 504 can be used to store software programs and modules of application software, such as the program instruction / information storage device corresponding to the method described in the embodiments of this application. The processor 502 executes various functional applications and information processing by running the software programs and modules stored in the memory 504, thereby realizing the above-mentioned object lookup method. The memory 504 may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 504 may further include memory remotely located relative to the processor 502, and these remote memories can be connected to the electronic device 5 via a network. Examples of the above-mentioned networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.
[0071] The transmission device 506 is used to receive or send information via a network. Specific examples of the network described above may include a wireless network provided by the communication provider of the electronic device 5. In one example, the transmission device 506 includes a Network Interface Controller (NIC), which can connect to other network devices via a base station to communicate with the Internet. In another example, the transmission device 506 may be a Radio Frequency (RF) module, used for wireless communication with the Internet.
[0072] This specification provides the operational steps of the methods described in the embodiments or flowcharts, but more or fewer operational steps may be included based on conventional or non-inventive labor. The steps and order listed in the embodiments are merely one possible execution order among many steps and do not represent the only execution order. In actual system or interrupt product execution, the methods shown in the embodiments or drawings can be executed sequentially or in parallel (e.g., in a parallel processor or multi-threaded processing environment).
[0073] The structure shown in this embodiment is only a partial structure related to the solution of this application and does not constitute a limitation on the device to which the solution of this application is applied. Specific devices may include more or fewer components than shown, or combinations of certain components, or arrangements of different components. It should be understood that the methods, apparatuses, etc., disclosed in this embodiment can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative. For example, the division of modules is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection between devices or unit modules through some interfaces.
[0074] Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, a portable hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.
[0075] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed in this specification can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of each example have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.
[0076] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.
Claims
1. An object lookup method, characterized in that, The method includes: In response to a target lookup command for a target object, the target lookup command is parsed to determine the target semantic information corresponding to the target object; If the preset spatial model does not include the preset object node corresponding to the target object, the position probability distribution information of the target object in the current space is determined based on the preset spatial model and the target semantic information; the preset spatial model is a three-dimensional model corresponding to the current space. The target scanning path corresponding to the target object is determined based on the location probability distribution information; Based on the target scanning path, locate the target object and determine the target location information of the target object; Based on the target location information, output the location description information corresponding to the target object.
2. The object search method according to claim 1, characterized in that, In the case where the preset spatial model does not include the preset object node corresponding to the target object, determining the position probability distribution information of the target object in the current space based on the preset spatial model and the target semantic information includes: If no preset object node corresponding to the target object exists in the preset spatial model, the target attribute of the target object is determined based on the target semantic information; the target attribute includes the category information of the target object and / or the preset spatial relationship between the target object and at least one associated object in the current space. Based on the preset spatial model, the category information, and / or the preset spatial relationship, the location probability distribution information is determined.
3. The object search method according to claim 2, characterized in that, The step of determining the location probability distribution information based on the preset spatial model, the category information, and / or the preset spatial relationship includes: Based on the category information, a first candidate placement area for the target object in the preset spatial model is determined; Based on the preset spatial relationship, determine the preset object node corresponding to the at least one associated object in the preset spatial model; Based on the preset object node corresponding to the at least one associated object in the preset space model and the preset space relationship, a second placement candidate area for the target object in the preset space model is determined. The location probability distribution information is determined based on the first placement candidate region and the second placement candidate region; the probability that the target object appears in the first placement candidate region and the second placement candidate region is greater than the probability that it appears in other regions, and the other regions are regions in the preset spatial model other than the first placement candidate region and the second placement candidate region.
4. The object search method according to claim 1, characterized in that, The step of locating the target object based on the target scanning path and determining the target location information of the target object includes: A target scanning instruction is generated based on the target scanning path; the target scanning instruction is used to instruct the scanning device to sequentially acquire scan images of each candidate placement area based on the target scanning path. Visual recognition is performed on the scanned image to determine the visual feature information of the candidate objects in each placement candidate area; Semantically align the target semantic information with the visual feature information to determine the candidate confidence of the candidate object; If the candidate confidence level is greater than a preset confidence threshold, the location information of the candidate object is determined as the target location information; Generate a scan stop command; the scan stop command is used to instruct the scanning device to stop the scanning operation.
5. The object search method according to claim 1, characterized in that, The step of outputting the location description information corresponding to the target object based on the target location information includes: Based on the target location information, determine the adjacent object nodes that are adjacent to the target object from the preset object nodes of the preset spatial model; Based on the location information of the adjacent object nodes and the target location information, the relative positional relationship between the target object and the objects corresponding to the adjacent object nodes is determined; The location description information is generated and output based on the relative positional relationship.
6. The object lookup method according to claim 1, characterized in that, After parsing the target search instruction in response to the target object and determining the target semantic information corresponding to the target object, the method further includes: When the preset spatial model includes a preset object node corresponding to the target object, the position information of the preset object node corresponding to the target object is determined as the target position information.
7. The object search method according to claim 1, characterized in that, The step of responding to a target lookup command for a target object by parsing the target lookup command and determining the target semantic information corresponding to the target object includes: If the target search instruction is a voice instruction, the voice instruction will be converted into a text instruction. The text instructions are semantically parsed to obtain the target semantic information.
8. An object search device, characterized in that, The device includes: The target semantic information determination module is used to parse the target search instruction for the target object in response to the target search instruction for the target object and determine the target semantic information corresponding to the target object; The location probability distribution information determination module is used to determine the location probability distribution information of the target object in the current space based on the preset spatial model and the target semantic information, when the preset spatial model does not include the preset object node corresponding to the target object; the preset spatial model is a three-dimensional model corresponding to the current space; The target scanning path determination module is used to determine the target scanning path corresponding to the target object based on the location probability distribution information. The target location information determination module is used to locate the target object based on the target scanning path and determine the target location information of the target object. The location description information output module is used to output the location description information corresponding to the target object based on the target location information.
9. An electronic device, characterized in that, The electronic device includes a processor and a memory, the memory storing at least one instruction or at least one program, the at least one instruction or the at least one program being loaded and executed by the processor to implement the object lookup method as described in any one of claims 1-7.
10. A computer-readable storage medium, characterized in that, The storage medium stores at least one instruction or at least one program segment, which is loaded and executed by a processor to implement the object lookup method as described in any one of claims 1-7.