An object retrieval method and device, an electronic device, and a storage medium

Abstract

Embodiments of the present application provide an object retrieval method and device, electronic equipment and storage medium, which are applied to the field of computer application technology, and the method comprises: obtaining a to-be-retrieved target corresponding to an object query request, and obtaining a recorded track from a data source of a home monitoring system according to the to-be-retrieved target, wherein the recorded track comprises a current target moving track, a historical target moving track and a holder moving track; generating an object disappearance position, an object habit position and an object carrying presumed position of the to-be-retrieved target based on the recorded track; generating a semantic detection result according to the object disappearance position, the object habit position and the object carrying presumed position; obtaining a monitoring image evidence matching the semantic detection result in the data source, and generating an object query feedback result according to the semantic detection result and the monitoring image evidence. Embodiments of the present application can realize the intelligent degree of object retrieval and improve the data credibility.

Description

Technical Field

[0001] This invention relates to the field of computer application technology, and in particular to an object retrieval method, apparatus, electronic device, and storage medium. Background Technology

[0002] With the rapid popularization and widespread implementation of home service robots and smart home devices, intelligent home sensing and question-and-answer interaction capabilities are gradually becoming the core development direction of the industry. Users have put forward higher requirements for the scenario-based semantic question-and-answer capabilities of smart home systems, no longer limited to simple command interaction, and expect the system to accurately respond to life-related and scenario-based object location query needs such as "Where are my keys?"

[0003] Currently, existing smart home and robot visual question-answering technologies still have many inherent shortcomings, making it difficult to meet the needs of object localization and intelligent question-answering in complex home scenarios. Existing technologies generally employ a single reasoning strategy, enabling only simple, single-layer reasoning, which cannot adapt to the complex and ever-changing interaction scenarios in the home environment, resulting in poor scenario adaptability and reasoning robustness. Furthermore, current mainstream solutions fail to achieve multi-layered integrated applications that incorporate long-term user memory and usage habit prediction, lacking long-term user behavior data support, leading to a low degree of relevance between question-answering reasoning results and actual user habits. Existing object localization question-answering systems heavily rely on manual user input of query commands, failing to achieve autonomous perception, proactive reasoning, and intelligent response, thus exhibiting insufficient intelligence. Summary of the Invention

[0004] This invention provides an object retrieval method, device, electronic device, and storage medium, which can solve the problems of existing object retrieval processes failing to integrate multiple layers of applications based on users' long-term memory and usage habits, lacking long-term user behavior data support, resulting in a low degree of relevance between question-and-answer reasoning results and actual user usage habits. It can realize autonomous perception functions in the object retrieval process and improve the intelligence level of object retrieval.

[0005] According to one aspect of the present invention, an object retrieval method is provided, wherein the method includes: Obtain the target to be retrieved corresponding to the object query request, and obtain the recorded trajectory of the target to be retrieved from the data source of the home monitoring system. The recorded trajectory includes the current target movement trajectory, the historical target movement trajectory, and the user movement trajectory. Based on the recorded trajectory, the object disappearance location, the object's usual location, and the object's estimated location are generated for the target to be retrieved; A semantic detection result is generated based on the object's disappearance location, the object's usual location, and the inferred location carried by the object. Obtain surveillance image evidence from the data source that matches the semantic detection result, and generate object query feedback results based on the semantic detection result and the surveillance image evidence.

[0006] According to another aspect of the present invention, an object retrieval device is provided, wherein the device comprises: The trajectory extraction module is used to obtain the target to be retrieved corresponding to the object query request, and to obtain the recorded trajectory of the target to be retrieved from the data source of the home monitoring system. The recorded trajectory includes the current target movement trajectory, the historical target movement trajectory, and the user movement trajectory. The location reasoning module is used to generate the object disappearance location, the object's usual location, and the object's estimated location based on the recorded trajectory. The semantic results module is used to generate semantic detection results based on the object's disappearance location, the object's habitual location, and the object's inferred location. The verification and feedback module is used to obtain monitoring image evidence that matches the semantic detection result within the data source, and generate object query feedback results based on the semantic detection result and the monitoring image evidence.

[0007] According to another aspect of the present invention, an electronic device is provided, the electronic device comprising: At least one processor; and A memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform the object retrieval method according to any embodiment of the present invention.

[0008] According to another aspect of the present invention, a computer-readable storage medium is provided, the computer-readable storage medium storing computer instructions for causing a processor to execute and implement the object retrieval method according to any embodiment of the present invention.

[0009] The technical solution of this invention obtains the target object corresponding to a user's object query request and, relying on the data source of a home monitoring system, comprehensively retrieves the recorded trajectory data corresponding to the target object. This trajectory data completely covers the target's current movement trajectory, historical movement trajectory, and the user's movement trajectory, achieving complete collection of multi-dimensional behavioral data. Based on this, the disappearance location, habitual location, and inferred carrying location of the target object are derived by combining the multi-dimensional trajectory data. Instantaneous motion state, long-term user habits, and carrying behavior are integrated to complete multi-layered reasoning, and accurate semantic detection results are generated based on the comprehensive analysis of these three types of location information. Furthermore, this invention matches surveillance image evidence corresponding to the semantic detection results in the data source, using real surveillance footage to corroborate the reasoning results. Finally, the semantic detection results and surveillance image evidence are combined to generate a traceable and highly reliable object query feedback result. This invention achieves habitual location reasoning by integrating historical trajectories and user behavior characteristics, completing a deep integration of long-term user memory and behavioral patterns. It breaks away from the limitations of traditional object retrieval, which relies heavily on users manually inputting query information. It realizes autonomous perception and intelligent reasoning in object retrieval scenarios, significantly improving the intelligence, relevance, and reasoning accuracy of home object retrieval questions and answers.

[0010] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of the present invention, nor is it intended to limit the scope of the invention. Other features of the invention will become readily apparent from the following description. Attached Figure Description

[0011] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0012] Figure 1 This is a flowchart of an object retrieval method provided according to Embodiment 1 of the present invention; Figure 2 This is a flowchart of another object retrieval method provided in Embodiment 2 of the present invention; Figure 3 This is a flowchart of another object retrieval method provided in Embodiment 3 of the present invention; Figure 4 This is an example diagram of an object detection system provided according to Embodiment 4 of the present invention; Figure 5 This is an example diagram of an object retrieval process provided in Embodiment 4 of the present invention; Figure 6This is a schematic diagram of the structure of an object retrieval device according to Embodiment 5 of the present invention; Figure 7 This is a schematic diagram of the structure of an electronic device that implements the object retrieval method of this invention. Detailed Implementation

[0013] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0014] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0015] Example 1 Figure 1 This is a flowchart of an object retrieval method according to Embodiment 1 of the present invention. This embodiment is applicable to intelligent object retrieval based on monitoring records in a home monitoring system. The method can be executed by an object retrieval device, which can be implemented in hardware and / or software. The object retrieval device can be configured in the storage device or host computer of the home monitoring system. Figure 1 As shown, the method includes: Step 110: Obtain the target to be retrieved corresponding to the object query request. Obtain the recorded trajectory of the target to be retrieved from the data source of the home monitoring system. The recorded trajectory includes the current target movement trajectory, the historical target movement trajectory, and the user movement trajectory.

[0016] The object query request can be a request to query the location of an object within a home surveillance scenario. This request can be made via voice, text, or other means. The target to be retrieved can be the object or person requested for retrieval, and this object or person can be located within the home environment. The data source can be the location of the acquired home surveillance video. Data sources can include home vision systems, home robot positioning and navigation systems, and home visual memory storage systems, and can be located locally or in third-party storage. Recorded trajectories can be the sum of various movement trajectories acquired from the data source. Recorded trajectories may include, but are not limited to, the current target movement trajectory, historical target movement trajectories, and user movement trajectories. The current target movement trajectory can be the movement trajectory of the object corresponding to the target to be retrieved at the current moment, while historical target movement trajectories can be the movement trajectories of the object corresponding to the target to be retrieved over a past period, and user movement trajectories can be the movement trajectories of users who have held the target to be retrieved over a past period.

[0017] In this embodiment of the invention, upon receiving a user's object query request in the form of voice or text, the specific target to be retrieved within the object query request can be parsed out. For example, the object query request may carry the identifier of the target to be retrieved, or the target to be retrieved corresponding to the object query request may be identified through semantic parsing technology. Subsequently, based on the parsed target to be retrieved, the full-domain data from data sources such as the home vision system, robot positioning and navigation system, and home visual memory storage repository can be uniformly scheduled and retrieved to comprehensively cover real-time images and historical archived data. On this basis, the current target movement trajectory, historical target movement trajectory, and user movement trajectory of the target to be retrieved are simultaneously extracted from multiple data sources as recorded trajectories, constructing a comprehensive target spatiotemporal behavior data set.

[0018] Step 120: Generate the object disappearance location, the object's usual location, and the object's estimated location based on the recorded trajectory.

[0019] Among them, the object disappearance location can refer to the spatial location where the target to be retrieved disappears; the object habitual location can refer to the location where the target to be detected is habitually located, such as the location where children usually play, or the location where keys are habitually placed; the object carrying inferred location can be the location inferred based on the person or object that may be carrying the target to be retrieved.

[0020] In this embodiment of the invention, the recorded trajectory can be analyzed for each target to be retrieved to determine the corresponding object disappearance location, the object's usual location, and the object's estimated location.

[0021] For example, the current target movement trajectory, historical target movement trajectory, and user movement trajectory can be intelligently inferred to generate three types of key location information for the target to be retrieved: the object's disappearance location, the object's habitual location, and the object's inferred location. The definitions and specific implementation methods of these three types of locations are as follows: (1) Object Disappearance Location: The object disappearance location refers to the spatial location of the target in the home scene last captured by the vision system and positioning device. It is the baseline positioning basis after the target is temporarily occluded, moves out of the monitoring range, or becomes stationary and hidden. The implementation method is as follows: The system traverses the current and recent historical movement trajectory of the target to be retrieved, filters the valid positioning coordinates at the end of the trajectory time sequence, removes abnormal and invalid points such as screen occlusion, detection failure, and device offline, and matches the target spatial coordinates and scene position of the last valid detection frame to accurately lock the final location of the target disappearance. It is suitable for rapid retrieval in scenes where the target is suddenly lost or temporarily hidden.

[0022] (2) Habitual Location of Objects: Habitual location of objects refers to the fixed spatial location where the target to be retrieved habitually stays and is placed during long-term family activities. It covers the regular placement points of items and the areas where people are frequently active, such as the fixed placement of keys and remote controls, and the daily play areas of children. The implementation method is as follows: The system mines the long-term historical movement trajectory and stay data of the target to be retrieved, performs cluster statistics and frequency analysis on all valid trajectory coordinates, and selects the family space points with the highest target stay frequency, the longest cumulative stay time, and the highest probability of recurrence. Temporary and accidental stay locations are filtered through time-series weight optimization, and finally iteratively updates to form the target's normalized habitual location, which is suitable for the search scenarios of daily item placement and personnel activity patterns.

[0023] (3) Object Carrying Predicted Location: The object carrying predicted location is the potential location of the target derived from the trajectory of the relationship between people and objects. It can indirectly locate objects that are stationary and have no trajectory and have no autonomous movement capability. The implementation method is as follows: analyze the historical movement trajectory of the target to be retrieved, screen out family members who have held, touched or carried the target, match the real-time location, current movement trajectory and recent activity range of the corresponding person, and combine the temporal relationship of the person-object binding to determine the person who is most likely to carry the target and the corresponding activity location. This is used as the object carrying predicted location, which effectively solves the problem that the target cannot be directly detected and located due to the loss of the item or the item being carried and moved by the person.

[0024] Step 130: Generate semantic detection results based on the object's disappearance location, the object's habitual location, and the inferred location carried by the object.

[0025] The semantic detection result can be information described in human language. The semantic detection result can be generated based on the object's disappearance location, the object's habitual location, and the object's inferred location. The semantic detection result can be a description of the three-dimensional spatial coordinates of the object's disappearance location, the object's habitual location, and the object's inferred location that conforms to human language habits. The semantic detection result includes at least the location, status, and confidence level of the target to be retrieved in the home scene.

[0026] In this embodiment of the invention, a unified semantic detection result is generated by fusing the object's disappearance location, the object's habitual location, and the object's inferred location. For example, the most likely location of the target to be retrieved can be selected based on the confidence levels of the object's disappearance location, the object's habitual location, and the object's inferred location, and the information of this location can be processed into semantic description information as the semantic detection result. Alternatively, semantic description information can be generated for the object's disappearance location, the object's habitual location, and the object's inferred location, and the sum of the semantic description information can be used as the semantic detection result.

[0027] Step 140: Obtain monitoring image evidence that matches the semantic detection results within the data source, and generate object query feedback results based on the semantic detection results and monitoring image evidence.

[0028] Among them, surveillance image evidence can be images that prove the semantic detection results, and the image content of surveillance image evidence can have a matching relationship with the semantic detection results.

[0029] Specifically, the system can obtain location-related descriptive information of the target object from the semantic detection results, search for image frames matching this descriptive information within the data source as surveillance image evidence, and merge this surveillance image evidence with the semantic detection results into an object query feedback result. This object query feedback result can be displayed to the user in the form of images, audio, or text to assist the user in finding the target object. It is understood that when the object query feedback result is audio or text, it may include the identification information or storage location of the surveillance image evidence.

[0030] This invention acquires the target object corresponding to a user's object query request and, relying on a home monitoring system data source, comprehensively retrieves the recorded trajectory data corresponding to the target object. This trajectory data fully covers the target's current movement trajectory, historical movement trajectory, and the user's movement trajectory, achieving complete collection of multi-dimensional behavioral data. Based on this, the invention infers the object's disappearance location, habitual location, and estimated carrying location by combining the multi-dimensional trajectory data. It integrates instantaneous motion state, long-term user habits, and carrying behavior to complete multi-layered inference and generates accurate semantic detection results based on the three types of location information. Furthermore, this invention matches surveillance image evidence corresponding to the semantic detection results in the data source, using real surveillance footage to corroborate the inference results. Finally, it combines the semantic detection results and surveillance image evidence to generate a traceable and highly reliable object query feedback result. This invention achieves habitual location reasoning by integrating historical trajectories and user behavior characteristics, completing a deep integration of long-term user memory and behavioral patterns. It breaks away from the limitations of traditional object retrieval, which relies heavily on users manually inputting query information. It realizes autonomous perception and intelligent reasoning in object retrieval scenarios, significantly improving the intelligence, relevance, and reasoning accuracy of home object retrieval questions and answers.

[0031] Example 2 Figure 2 This is a flowchart of another object retrieval method provided in Embodiment 2 of the present invention. This embodiment of the present invention describes the extraction process of recorded trajectories. See [link to flowchart]. Figure 2 The method provided in this embodiment of the invention specifically includes the following steps: Step 210: Read the real-time video stream of the home vision system, identify the current spatial position of the target to be retrieved in the real-time video stream based on the preset target detection module, and arrange the current spatial positions according to the acquisition time of the target to be retrieved in the real-time video stream as the current target movement trajectory.

[0032] The preset target detection module can be a neural network model used for image recognition of the target to be retrieved. The preset target detection module can be implemented based on models such as YOLO model and Faster R-CNN model.

[0033] In this embodiment of the invention, a home vision system can be accessed and the real-time video stream collected by the home vision system can be read. A pre-trained preset target detection model can be called to identify the target to be retrieved and determine the location of the target to be retrieved as the current spatial location. For each current spatial location, the acquisition time corresponding to the target to be retrieved in the real-time video stream is taken as the appearance time of the target to be retrieved at the current spatial location. The current spatial locations can be arranged according to each acquisition time to obtain the current target movement trajectory.

[0034] Step 220: Locate the map location of the target to be retrieved within the positioning and navigation system, and arrange the map location points in chronological order to form the current target's movement trajectory.

[0035] The positioning and navigation system can be a positioning and navigation system for a home-use mobile device, which may include a robot vacuum cleaner, a home robot dog, a patrol camera, etc. The map location point can be the map location of the target to be retrieved within the positioning and navigation system.

[0036] In other embodiments of the invention, the positioning and navigation system of one or more home devices can be accessed. The map location points of the target to be retrieved identified by each home device can be extracted from the positioning and navigation system. The map location points can be arranged in the order of time when the target to be retrieved was identified, thereby obtaining the current target movement trajectory.

[0037] Step 230: Determine the semantic description information of the target to be retrieved, and extract at least one historical target movement trajectory from the family visual memory repository according to the semantic description information.

[0038] Semantic description information can be information that describes the target to be retrieved in a way that conforms to human language habits. Semantic description information can include descriptions of attributes such as the shape, color, outline, or brand of the target to be retrieved.

[0039] In this embodiment of the invention, a large language model can be used to generate semantic description information of the target to be retrieved. This semantic description information may include textual semantic features such as the category attributes, appearance features, scene affiliation, and associated persons of the target to be retrieved, which can accurately characterize the target to be retrieved. The large language model can be used to retrieve each monitoring video in the family visual memory bank according to the above semantic description information, and determine and filter out data entries that highly match the semantic features of the current target to be retrieved. Finally, at least one complete historical target movement trajectory corresponding to the target to be retrieved is extracted. It is understood that the family visual memory bank can store the movement trajectories or associated videos of various objects.

[0040] Step 240: Obtain the target-related image information of the target to be retrieved in the family visual memory storage repository, determine the user of the target to be retrieved based on the target-related image information, and obtain the user's movement trajectory in the family visual memory storage repository.

[0041] The target-related image information can be an image that is related to the image of the target to be retrieved. This image may include the image of the user of the target to be retrieved or the image of other objects that are related to the target to be retrieved.

[0042] Specifically, based on the category characteristics, spatiotemporal relationships, and human interaction records of the target to be retrieved, the system can accurately match and identify associated objects or people that have contact, companionship, or binding relationships with the target in the family visual memory repository. Monitoring images from the corresponding time moments can then be extracted as target-related image information. A preset target detection module performs visual recognition and interactive behavior analysis on the target-related image information, automatically identifying the user who has previously or currently held, touched, or carried the target. Furthermore, based on the identified user information, the system can accurately filter and extract the user's full-domain activity data within the corresponding time period from the family visual memory repository, thereby obtaining a complete and continuous user movement trajectory.

[0043] Step 250: Generate the object disappearance location, the object's usual location, and the object's estimated location based on the recorded trajectory.

[0044] Step 260: Generate semantic detection results based on the object's disappearance location, the object's habitual location, and the inferred location carried by the object.

[0045] Step 270: Obtain monitoring image evidence that matches the semantic detection results within the data source, and generate object query feedback results based on the semantic detection results and monitoring image evidence.

[0046] This invention, through real-time reading of the video stream from a home vision system, continuously identifies the real-time spatial location of the target to be retrieved in the image using a preset target detection module. It then arranges all location points chronologically according to the video frame acquisition time sequence, automatically generating the current target movement trajectory. Additionally, it arranges the map location points of the target within the positioning and navigation system chronologically to form the current target movement trajectory, determines the semantic description information of the target, extracts at least one historical target movement trajectory from the home visual memory repository based on the semantic description information, obtains target-related image information of the target within the home visual memory repository, determines the user of the target based on the target-related image information, and obtains the user's movement trajectory within the home visual memory repository. This invention can integrate multi-dimensional trajectory data to deduce the object's disappearance location, habitual location, and estimated carrying location. It combines the target's instantaneous motion state, the user's long-term placement and activity habits, and the associated carrying behavior to achieve multi-level intelligent reasoning, and outputs accurate scene semantic detection results based on the comprehensive integration of these three types of location information. Simultaneously, monitoring image materials corresponding to the semantic detection results are matched from multiple data sources. Real video footage is used to verify and trace the inference conclusions. Finally, combining the semantic inference conclusions with visual image evidence, a traceable and highly reliable object query feedback result is output. This solution integrates the target's historical movement trajectory with user's object-holding behavior characteristics to achieve intelligent inference of the object's habitual placement location. It deeply integrates long-term home scene memory and user behavior patterns, overcoming the limitations of traditional object retrieval methods that heavily rely on user-initiated query commands. It can autonomously perceive, correlate inference, and intelligently respond in object retrieval scenarios, effectively improving the intelligence level, scene adaptability, and location inference accuracy of home object query services.

[0047] Based on the above embodiments of the invention, determining the user holding the target to be retrieved according to the target-related image information includes: Identify user images that have spatial overlap with the target to be retrieved within the target-related image information; determine the users corresponding to the target user images whose intersection-union ratio (IUU) of the target image to be retrieved within each user image is greater than a preset threshold as the users holding the target image.

[0048] Specifically, it can identify user human image regions within the family visual memory repository that have spatial overlap with the target to be retrieved, thereby identifying potential user objects who have close interaction, contact, or carrying behavior with the target. It calculates the intersection-union ratio (IUR) of each user image region with the target image region, quantifies the spatial fit and interaction tightness of people or objects by using a preset IUR threshold, filters out target user images with IUR greater than the preset threshold, and finally determines the user in the corresponding image as the holder of the target to be retrieved.

[0049] For example, multiple consecutive frames of images are read within the last visible time window of the target object; for each frame, the target object detection box, the person detection box, and the person's hand region box are obtained respectively; the spatial overlap relationship between the target object detection box and the person's hand region box is calculated; when the spatial overlap relationship meets the preset conditions in multiple consecutive frames, the corresponding person is marked as the user holding the device; if there is only a brief intersection or the person's body box overlaps but the hand region does not meet the conditions, it is marked as "passing interference" and is not considered as carrying behavior, that is, the corresponding person is not certain to be the user holding the device.

[0050] Example 3 Figure 3 This is a flowchart of another object retrieval method provided in Embodiment 3 of the present invention. The embodiments of the present invention illustrate the process of generating semantic detection results. See [link to documentation]. Figure 3 The method provided in this embodiment of the invention includes the following steps: Step 310: Obtain the target to be retrieved corresponding to the object query request. Obtain the recorded trajectory of the target to be retrieved from the data source of the home monitoring system. The recorded trajectory includes the current target movement trajectory, the historical target movement trajectory, and the user movement trajectory.

[0051] Step 320: Extract the current target movement trajectory within the recorded trajectory, obtain the current video information of the target to be retrieved in the home space from the data source according to the current target movement trajectory, determine the target home location in the current video information whose disappearance time is greater than the threshold as the object disappearance location, and generate the first confidence score according to the time difference between the image acquisition time of the object disappearance location and the current time.

[0052] In this embodiment of the invention, for the current target movement trajectory included in the recorded trajectory, the current video information corresponding to different home spaces through which the current target movement trajectory passes can be obtained from the data source. The current video information can indicate that the target to be retrieved is located in a home space. The home space can include objects, bedrooms, kitchens, bathrooms, entertainment rooms, etc. It is understood that the home location can be further refined, such as object area 1, object area 2, object area 3, etc. The disappearance time of the target to be retrieved can be counted for each home space according to the corresponding current video information. If the disappearance time of the target to be retrieved is greater than a threshold, the home space is determined as the target home location and the target home location is determined as the object disappearance location. The image acquisition time when the target to be retrieved disappears at the target home location can be obtained as the last time. The difference between the last time and the current time can be mapped to a first confidence level between 0 and 1 according to a preset mapping relationship.

[0053] Step 330: Extract the user's movement trajectory within the recorded trajectory, extract user video information of the user in the home space of the target to be retrieved within the data source according to the user's movement trajectory, and generate the object carrying the inferred location and second confidence level based on the spatial position relationship between the target to be retrieved and the user in the current video information and the trajectory time series prediction model.

[0054] In this embodiment of the invention, the user's trajectory within the recorded trajectory can be obtained. Image information of different home spaces along the path of the user's trajectory can be extracted from various data sources according to the user's trajectory as user video information. The trajectory temporal prediction model can be called to process the user video information associated with the user's trajectory and the current video information associated with the current target's movement trajectory. Based on the trajectory temporal prediction model, the object carrying inferred position and second confidence level are generated according to the spatial positional relationship between the target to be retrieved and the user. The trajectory temporal prediction model may include a lightweight temporal Transformer model, a temporal convolutional prediction model, an LSTM temporal model, etc.

[0055] Step 340: Extract the historical target movement trajectory within the recorded trajectory and the family space topology information within the data source, establish the mapping relationship between the historical trajectory points within the historical target movement trajectory and the spatial location within the family space topology information, and generate the object habitual location and third confidence level of the target to be retrieved based on the spatiotemporal feature prediction model and the mapping relationship.

[0056] In this embodiment of the invention, historical target movement trajectories can be extracted from recorded trajectory data, and family spatial topology information within the data source can be obtained. This family spatial topology information can cover structured spatial parameters such as the division of functional areas of the family, spatial point associations, and regional connectivity. By establishing a one-to-one mapping relationship between historical trajectory points and actual spatial locations within the family, discrete temporal coordinate trajectories can be converted into structured spatial activity data that fits the home scenario. A spatiotemporal feature prediction model is then used to analyze the structured spatial activity data. Through the aforementioned spatiotemporal feature prediction model, the long-term dwelling patterns, regional preferences, and placement habits of the target to be retrieved can be analyzed. Clustering and trend prediction can be performed on spatial areas where the target frequently appears, stays for a long time, and is repeatedly located. Based on the spatiotemporal feature prediction model and the aforementioned clustering and trend prediction, the habitual location and third confidence level of the target to be retrieved can be determined.

[0057] Specifically, the aforementioned spatiotemporal prediction model may include a spatial feature encoding module, a temporal feature encoding module, a spatiotemporal fusion module, and an output module. The spatial feature encoding module takes the family's spatial topology information as input and performs structured encoding on static spatial features such as functional areas, spatial coordinates, regional connectivity, and scene attributes, transforming the discrete physical space of the family into a high-dimensional spatial feature vector. Simultaneously, it receives the spatial coordinate information corresponding to historical trajectory points, completing the spatial association mapping between trajectory points and family areas, providing spatial semantic support for subsequent habitual area clustering. This module can employ lightweight convolutional networks or micro-graph convolutional structures to reduce edge computing overhead while maintaining spatial topology modeling capabilities. The temporal feature encoding module can input long-sequence historical movement trajectory data of the target to be retrieved, responsible for extracting the target's temporal movement patterns and dwell characteristics. It can employ lightweight temporal networks (lightweight GRU / LSTM, micro-TCN, or lightweight temporal Transformer). The spatiotemporal fusion module integrates static home space topological features with dynamic trajectory temporal features to establish a bidirectional correlation between "spatial location and temporal behavior." It can use an attention mechanism to weight high-frequency dwell areas and highly correlated spatial points. The output module decodes and regresses the high-dimensional features after spatiotemporal fusion, outputting the optimal habitual location coordinates of the target object and the corresponding semantic information of the home area; simultaneously, it calculates a third confidence level through a feature confidence branch.

[0058] Step 350: Compare the first confidence level and the first threshold of the object's disappearance location. If the first confidence level is greater than or equal to the first threshold, then the object's disappearance location is determined as the final location.

[0059] In this embodiment of the invention, the first confidence level can be accurately compared with the preset first threshold to complete the validity screening and determination; when the first confidence level is greater than or equal to the preset first threshold, it can be indicated that the reasoning result of the object disappearance position obtained in this inference is credible, and the object disappearance position can be directly determined as the final search position of the target to be searched.

[0060] Step 360: If the first confidence level is less than the first threshold, compare the second confidence level of the object carrying the predicted position with the second threshold. If the second confidence level is greater than or equal to the second threshold, then determine the object carrying the predicted position as the final position.

[0061] Specifically, if the first confidence level is less than the first threshold, the second confidence level of the object carrying the predicted position can be compared with the second threshold. If the second confidence level of the object carrying the predicted position is greater than or equal to the second threshold, then the predicted position of the object is determined as the final position.

[0062] Step 370: If the second confidence level is less than the second threshold, then the object's habitual position is determined as the final position.

[0063] In this embodiment of the invention, if the second confidence level of the object carrying the predicted position is less than the second threshold, then the object's habitual position is determined as the final position.

[0064] Step 380: Set the weighted sum of the first confidence level, the second confidence level, and the third confidence level as the fusion weight of the final position; call the preset semantic generation module to generate the semantic detection result of the target to be retrieved according to the final position and the fusion weight.

[0065] In this embodiment of the invention, preset weights corresponding to the first confidence level, the second confidence level, and the third confidence level can be obtained. A weighted sum of the first confidence level, the second confidence level, and the third confidence level is calculated according to the preset weights, and this weighted sum can be used as the fusion weight for the final position. Based on a preset semantic generation module, the final position and fusion weights are converted into a semantic detection result that conforms to human semantic expression. This preset semantic generation module can be implemented based on rule-based semantic templates, semantic understanding models, knowledge graph models, etc.

[0066] Step 390: Obtain monitoring image evidence that matches the semantic detection results within the data source, and generate object query feedback results based on the semantic detection results and monitoring image evidence.

[0067] Furthermore, based on the above embodiments of the invention, it also includes: determining that the fusion weight is less than a third threshold, and controlling the mobile monitoring device to perform video verification of the object's habitual location.

[0068] Among them, mobile monitoring devices can be movable household devices in home monitoring scenarios, including mobile cameras, robot vacuum cleaners, and household robot dogs.

[0069] In this embodiment of the invention, the fusion weight can be compared with a third threshold. If the fusion weight is less than the third threshold, control information for the mobile monitoring device can be generated. The mobile monitoring device can be driven to verify the determined habitual position of the object according to the control information to determine whether the target to be searched is located at the habitual position of the object.

[0070] Based on the above embodiments of the invention, obtaining surveillance image evidence that matches semantic detection results within the data source includes: extracting the final location of the target to be retrieved within the semantic detection results, and determining the time point at which the target to be retrieved is located at the final location based on the historical surveillance video of the target to be retrieved within the data source; and extracting at least one surveillance video frame from the historical surveillance video as surveillance image evidence according to the time point.

[0071] In this embodiment of the invention, the semantic detection results can be parsed to accurately decompose and lock the final location corresponding to the target to be retrieved, and to clarify the spatial range of evidence retrieval. Then, the historical monitoring video data archived in the home vision system and the home visual memory storage repository can be traversed to match all valid time points corresponding to the stable appearance of the target to be retrieved at the final location, and to accurately locate the time period and timestamp of the target's appearance at the corresponding spatial location. After determining the precise time point, at least one valid monitoring video frame can be extracted from the corresponding historical monitoring video stream based on the timestamp index as monitoring image evidence to support the semantic detection results.

[0072] This invention enables precise binding of inference conclusions with real monitoring footage. By visualizing the original video footage, it verifies the authenticity of location inference, behavior deduction, and semantic output, effectively solving the problems of weak interpretability and lack of traceability in pure data inference. Ultimately, it forms a dual verification system of "trajectory data inference + image evidence verification," ensuring that the object query feedback results are traceable and highly reliable.

[0073] Example 4 This invention addresses the growing demand from users for home service robots and smart home devices, where systems can answer questions like "Where are my keys?". It provides an object retrieval system, see [link to relevant documentation]. Figure 4 The object retrieval system can include the following four core modules: natural language interface module, three-layer inference engine module, confidence fusion module, and evidence association module.

[0074] The natural language interface module can receive natural language queries; parse the query request, identify the target object; and return a structured answer, which includes location, confidence level, and video evidence.

[0075] The three-layer inference engine comprises four inference layers: the last observed location inference layer, the carried behavior inference layer, the habitual location prediction layer, and the robot verification layer. The robot verification layer can be freely configured as needed.

[0076] Layer 1 Last Observation Position Inference Layer: Queries the last observed position of an object from the visual memory system and calculates the first confidence level based on the last observation time; Typical decay curve: 0.9 within 1 hour, 0.7 for 1-24 hours, 0.5 for 24-72 hours, and 0.3 for more than 72 hours; Layer 2 carries the behavior inference layer: —When the confidence of Layer 1 is lower than the threshold, the user is detected and judged based on the spatial overlap relationship, and the object position is predicted based on the user's trajectory tracking. Layer 3 Habitual Location Prediction Layer: When both Layer 1 and Layer 2 are below the threshold, the object placement location is predicted based on the correlation analysis between the object's historical trajectory data and the home space topology. Layer 4 Robot Verification Layer: When the confidence level of the three-layer fusion is lower than the verification threshold, the embodied agent is driven to go to the target location for on-site verification, and the verification result is used as the input of the fourth inference layer.

[0077] The confidence fusion module can perform confidence fusion on the outputs of the three-layer inference engine, such as a weighted fusion of w1·C1 + w2·C2 + w3·C3, where w1, w2, and w3 are the weights of the confidence scores output by the three-layer inference engine, with default weights of w1=0.5, w2=0.3, and w3=0.2. The fusion weights generated by this weighted fusion can be used by the robot to verify the prediction results of each inference layer, thereby enabling dynamic adjustment of the historical accuracy of each inference layer.

[0078] The evidence association module can perform video evidence integration, identifying video clips associated with the reasoning result as evidence, such as matching video clips containing the object to be searched at the corresponding location in the reasoning result. It can return video clips, timestamps, and thumbnails for users to verify the reasoning result.

[0079] See Figure 5 The object retrieval method provided in this embodiment of the invention may include the following steps: Step 1: Natural Language Query Reception and Target Object Recognition. The system receives the user's natural language query, parses the query type, target object name, and selectable time range; it matches the target object name with the object identifier database to obtain the target object's unique ID. This step forms a unified input object for subsequent inference, avoiding the inability to correlate long-term trajectories with only the target category.

[0080] Step 2: Layer 1 Last Observation Location Inference. The system queries the long-term object trajectory database based on the target object ID, retrieving the location and timestamp of the target object's most recent observation by the vision system; it then calculates the first confidence level based on the last observation time. If the first confidence level reaches a preset confidence threshold, the location becomes a candidate answer; if the first confidence level is below the threshold, Layer 2 behavior inference is triggered. This step embodies the first layer of the M5 three-layer progressive inference architecture.

[0081] Step 3: Layer 2 Spatial Overlap Detection and User Trajectory Tracking. When the Layer 1 confidence level is insufficient, the system backtracks to multiple consecutive video frames before the object disappears, detecting the spatial overlap between the object boundary and the person's hand detection box. When the IoU is greater than a preset threshold and holds true for multiple consecutive frames, the person is identified as the user, thus distinguishing between "person passing by" and "person holding an object." The system then tracks the user's movement trajectory, recording the spatial areas they traverse, the areas they remain in, and the position where they disappear from the field of view. If the user is an embodied agent, their SLAM trajectory is read as the trajectory source. This step outputs a second candidate position and a second confidence level, reflecting M1 spatial overlap detection and M2 user trajectory tracking.

[0082] Step 4: Layer 3 Habitual Placement Prediction. When the results from Layer 1 and Layer 2 are still insufficient to form a high-confidence answer, the system reads the historical placement trajectory of the target object, maps the historical location to a semantic region in the home space topology, and combines time period, room, and family member activity patterns to count candidate placement areas, outputting the candidate location with the highest placement frequency and a third confidence level. This step continues to demonstrate the M5 three-layer progressive inference architecture, supplementing the long time interval scenarios that the final observation and carried inference cannot cover.

[0083] Step 5: Optional Layer 4 Robot Verification and Feedback Update. When the fusion confidence of Layers 1-3 is lower than the preset verification threshold, the system generates a verification task and sends the navigation path and target location through the embodied agent access protocol. After the embodied agent moves to the target area, it performs visual detection and returns a found, not found, or position deviation result. This verification result serves as the input to the fourth inference layer: if found, the confidence of the corresponding position is increased; if not found, the corresponding candidate position is cleared and re-inference is triggered; if there is a position deviation, the confidence is adjusted according to the degree of deviation. Simultaneously, this verification result is recorded for subsequent adjustment of the fusion weights.

[0084] Step 6: Multi-Inference Layer Confidence Fusion and Probability Result Output. The system fuses the candidate positions and confidence scores of Layer 1, Layer 2, Layer 3, and the optional Layer 4. A weighted fusion confidence score is calculated for each position; when multiple inference layers return the same position, the fusion confidence scores for that position are superimposed; the fusion weights are dynamically adjusted based on the historical accuracy of each inference layer according to the robot's verification results. Finally, the system selects the position with the highest fusion confidence score as the primary position and retains other high-scoring positions as a candidate position list, outputting the primary position + candidate positions + probability or confidence score distribution. This step demonstrates the M3 multi-inference layer confidence fusion algorithm and weighted formula calculation method, and the M5 probability result output.

[0085] Step 7: Video Evidence Integration and Structured Answer Generation. The system queries the video clip library based on the primary location, alternative locations, and their corresponding timestamps, matching video clips containing footage of the target object appearing at that location or being carried by a user. Keyframes are extracted as evidence thumbnails, and the video clip paths and timestamps are associated to generate a structured answer that includes the primary location, fusion confidence level, a list of alternative locations, video evidence links, keyframe thumbnails, and the source of the inference layer. This step embodies the M4 video evidence integration process, enabling users to verify AI judgments rather than simply receiving black-box classification results.

[0086] Step 8: User Confirmation or Verification Result Write-back. After the user clicks on evidence, confirms finding an object, rejects the answer, or the embodied agent completes the verification, the system records whether the result is consistent with the predictions of each inference layer. If a certain inference layer is verified to be accurate multiple times consecutively, its fusion weight is increased; otherwise, its weight is decreased. This step forms a closed loop from the query result to the subsequent update of the inference weight, but the specific details of the weight adjustment are not disclosed.

[0087] In the above embodiments of the invention, the spatial overlap detection of M1 may include the following steps: reading multiple consecutive frames of images within the last visible time window of the target object; obtaining the target object detection box, the person detection box, and the person's hand region box for each frame; calculating the spatial overlap relationship between the target object detection box and the person's hand region box; when the spatial overlap relationship meets the preset conditions in multiple consecutive frames, marking the corresponding person as a candidate holder; if only a brief intersection occurs or the person's body box overlaps but the hand region does not meet the conditions, it is marked as "passing interference" and is not considered as carrying behavior.

[0088] Based on the above embodiments of the invention, M2 user trajectory tracking includes the following steps: using the candidate user identifier as an index, reading the person's cross-camera movement trajectory after the object disappears; mapping the movement trajectory to the semantic locations of rooms, areas, and furniture in the home space topology; recording the areas where the user stays for a long time, the last location left the frame, and the placement scene related to the target object; converting the above areas into candidate object locations, and forming a second confidence level based on trajectory continuity, dwell time, and camera coverage integrity; if the user is an embodied intelligent agent, reading its SLAM trajectory and operation log as the trajectory source.

[0089] Based on the above embodiments of the invention, the M3 multi-inference layer confidence fusion algorithm and weighted formula calculation method and the M5 probability result output include the following steps: mapping the last observed position of Layer1, the inferred position carried by Layer2, the habitual position of Layer3, and the optional verification position of Layer4 to the same family spatial position granularity; merging multiple candidate results for the same semantic position, and retaining different semantic positions as parallel candidates; reading the confidence of each inference layer and the historical verification accuracy of that inference layer respectively; forming the fusion confidence of each candidate position according to the preset fusion weight; taking the position with the highest fusion confidence as the primary position, and the remaining positions that meet the minimum display conditions as alternative positions; writing the user confirmation, denial, or robot verification results back to the historical accuracy statistics for subsequent adjustment of the weights of each inference layer.

[0090] Based on the above-mentioned application embodiments, the M4 video evidence integration process includes the following steps: determining the inference source and time window for the primary location and candidate locations respectively; querying the video clip library according to the time window, and prioritizing the selection of clips that can display the target object, the user's actions, or the candidate location environment; extracting at least one keyframe thumbnail and video timestamp for each candidate location; combining the candidate location, confidence level, inference source, video clip, and keyframe into a structured answer; when evidence is missing, the structured answer retains the candidate location and confidence level, while marking "no direct video evidence" to avoid disguising a result without evidence as a verified result.

[0091] In one example, a family with an infant is searching for their keys. The scenario is described as follows: A couple has a one-year-old baby who is cared for by grandparents during weekdays; the family has four cameras (entrance hall, living room, master bedroom, and second bedroom); the system has been running for three months and has over 500 video clips of the baby. One morning at 7:30, the father is about to leave and discovers his keys are missing, so he asks the system, "Where are my keys?"

[0092] Input: Natural language query "Where is my key?"; Time range: 6:00-7:30 this morning; Object ID: key (already recognized by the system).

[0093] Processing flow: Step 1: Parse the query and identify the target object = key, time range = 6:00-7:30 this morning; Step 2: Layer 1 queries the last observed location of the key → detected at 6:15 this morning in the entryway cabinet, 1.25 hours ago → confidence level 0.85 (> threshold 0.7, Layer 1 returns directly); Layers 2-3 are not triggered; Step 6: Fusion confidence level = 0.85 → directly return this location as the answer; Step 7: Query the video clip library → match the 5-second video clip of the entryway cabinet at 6:15 → extract keyframes → generate evidence → return the structured answer.

[0094] Output: Main location: entryway cabinet (confidence level 0.85) Video evidence: / evidence / key_20260604_061500.mp4 (keyframe thumbnails) Inference layer source: Layer 1 - Last observed position Response time: <2 seconds from query to return User experience: The father knows the location of the key in seconds and can directly retrieve it to leave the house. In one example, consider the scenario of elderly people living alone searching for medication. Two elderly people live alone while their children work in other cities. They are monitored remotely via webcams. The family has three webcams deployed (entrance hall, living room, and kitchen). One elderly person discovers their blood pressure medication is missing when taking it in the morning and asks the system, "Where is my blood pressure medication?"

[0095] Input: Natural language query "Where is my blood pressure medication?"; Time range: 7:00-8:00 AM today; Object ID: Blood pressure medication (already recognized by the system).

[0096] Processing flow: Step 1: Parse the query and identify the target object = antihypertensive medication, time range = 7:00-8:00 AM today; Step 2: Layer 1: Query the last observed location of the antihypertensive medication → detected on the bedside table in the bedroom at 10:00 PM yesterday, more than 24 hours ago → confidence level 0.5 (< threshold 0.7, trigger Layer 2); Layer 2: Carry-over behavior inference (M1+M2) — M1: Spatial overlap detection: detected the antihypertensive medication and the elderly person's hand IoU > 0.3 (5 consecutive frames) in the 7:15 video → determined that the elderly person is the user; M2: User trajectory tracking: track the elderly person's movement from 7:15 to 7:25 (in the kitchen). Traces; The elderly person took breakfast in the kitchen and left the kitchen at 7:30; Prediction: The antihypertensive medication is most likely near the breakfast table in the kitchen (staying area) or the kitchen cabinet (disappearance location) → Confidence 0.7; Layer 3 not triggered; Step 6 fusion — Layer 1 (0.5) + Layer 2 (0.7) return to different locations → weighted fusion → bedroom bedside table w1·C1=0.5×0.5=0.25; kitchen breakfast table w2·C2=0.3×0.7=0.21 → bedroom bedside table is the main location; Step 7 match the bedroom bedside table 22:00 keyframe + kitchen 7:15-7:30 user trajectory fragment → return to structured answer.

[0097] Output: Main location: Bedroom bedside table (confidence level 0.25) Alternative location: Kitchen breakfast table (confidence level 0.21) Video evidence: 2 video clips (bedroom 22:00 + kitchen 7:15-7:30) Inference layer source: Layer 1 - Last observed position + Layer 2 - Carrying behavior inference Response time: <3 seconds from query to return User feedback: The elderly person checked the bedside table in the bedroom and found that the blood pressure medication was indeed missing (it was possible that the medication was taken at 7:15 and then forgotten to be put back). They then went to the kitchen and found it.

[0098] In one example, let's describe a scenario where a pet-owning family is looking for their remote control. A couple owns a dog and checks on it via a webcam during work hours. They have two webcams installed in their home (one in the living room and one in the entryway). The system has been running for two months. One evening at 8:00 PM, the couple is about to watch TV when they realize the remote control is missing and asks the system, "Where is my remote control?"

[0099] Input: Natural language query "Where is my remote control?"; Time range: today 0:00-8:00; Object ID: remote control (system has recognized).

[0100] Processing flow: Step 1: Parse the query and identify the target object = remote control; Step 2: Layer 1: Query the last observed position of the remote control → Detected on the coffee table in the living room at 22:00 yesterday, 2 hours ago → Confidence 0.7 (=threshold, needs to be supplemented by Layer 2); Layer 2: M1: Spatial overlap detection: No person holding the remote control was detected in the 22:05 video (M1 failed); M2: User trajectory tracking: No user → Confidence 0; Layer 3: Habitual position prediction: Query the historical placement habits of the remote control → Coffee table 70% / TV cabinet 30% → Confidence 0.6; Layer 4: Robot verification - Triggering condition: Layer 1 (0.7) + Layer 2 (0) + Layer 3 (0.6) fusion confidence = 0.5 × 0.7 + 0. 3×0+0.2×0.6=0.35+0+0.12=0.47 (<verification threshold 0.6, trigger Layer4); The system sends a robot to the coffee table to verify; The robot reports: No remote control was found on the coffee table, but the remote control was detected next to the TV cabinet; The fourth inference layer feedback: Update the confidence of Layer1 according to the verification result → The confidence of the coffee table is cleared to zero, and the confidence of the TV cabinet is 0.85; Step 6 fusion—Layer1(0, coffee table)+Layer2(0)+Layer3(TV cabinet 0.6)+Layer4(TV cabinet 0.85)→ Superimpose at the same position: TV cabinet 0.2×0.6+0.85=0.97→The TV cabinet is the main position; Step 7 match the 22:30 keyframe of the TV cabinet + the robot verification video→ Return the structured answer.

[0101] Output: Main location: TV cabinet (confidence level 0.97) Video evidence: 2 video clips (TV cabinet at 22:30 + verified by robot) Inference layer source: Layer 3 - Habitual position prediction + Layer 4 - Robot verification Response time: From query to robot verification completion <10 seconds User experience: The couple found the remote control in the TV cabinet and could watch TV immediately; Learning outcome: The results of this investigation will serve as feedback to continuously optimize the fusion weights of Layer 1-3.

[0102] Example 5 Figure 6 This is a schematic diagram of the structure of an object retrieval device according to Embodiment 5 of the present invention, as shown below. Figure 6 As shown, the device includes: The trajectory extraction module 410 is used to obtain the target to be retrieved corresponding to the object query request, and to obtain the recorded trajectory of the target to be retrieved from the data source of the home monitoring system. The recorded trajectory includes the current target movement trajectory, the historical target movement trajectory, and the user movement trajectory.

[0103] The location reasoning module 420 is used to generate the object disappearance location, the object's habitual location, and the object's inferred location based on the recorded trajectory.

[0104] The semantic result module 430 is used to generate semantic detection results based on the object's disappearance location, the object's habitual location, and the object's inferred location.

[0105] The verification feedback module 440 is used to obtain monitoring image evidence that matches the semantic detection result within the data source, and generate object query feedback results based on the semantic detection result and the monitoring image evidence.

[0106] Optionally, the trajectory extraction module 410 includes: The first current unit is used to read the real-time video stream of the home vision system, identify the current spatial position of the target to be retrieved in the real-time video stream based on the preset target detection module, and arrange the current spatial positions according to the acquisition time of the target to be retrieved in the real-time video stream as the current target movement trajectory.

[0107] The second current unit is used to find the map location of the target to be retrieved within the positioning and navigation system, and arrange the map location points in chronological order as the current target's movement trajectory.

[0108] The historical trajectory unit is used to determine the semantic description information of the target to be retrieved, and to extract at least one historical target movement trajectory of the target to be retrieved from the family visual memory repository according to the semantic description information.

[0109] The user trajectory unit is used to acquire target-related image information of the target to be retrieved in the family visual memory repository, determine the user of the target to be retrieved based on the target-related image information, and acquire the user's movement trajectory in the family visual memory repository.

[0110] Optionally, the user holding the target in the trajectory unit is determined based on the target-associated image information, including: Identify user images within the target-associated image information that have a spatial overlap with the target to be retrieved; The user corresponding to the target user image in each of the user images whose intersection-union ratio of the target image to be retrieved is greater than a preset threshold is determined as the user holding the target image.

[0111] In some embodiments of the invention, the location reasoning module 420 is specifically used to: extract the current target movement trajectory within the recorded trajectory, obtain the current video information of the target to be retrieved located in the home space within the data source according to the current target movement trajectory, determine the target home location within the current video information where the disappearance time of the target to be retrieved is greater than a threshold as the object disappearance location, and generate a first confidence level according to the time difference between the image acquisition time of the object disappearance location and the current time; Extract the user's movement trajectory within the recorded trajectory, extract user video information of the target to be retrieved in the home space of the target to be retrieved in the data source according to the user's movement trajectory, and generate the object carrying the inferred location and second confidence level based on the spatial position relationship between the target to be retrieved and the user in the current video information and the user's spatial position relationship according to the trajectory time series prediction model. Extract the historical target movement trajectory within the recorded trajectory and the family space topology information within the data source, establish a mapping relationship between the historical trajectory points within the historical target movement trajectory and the spatial location within the family space topology information, and generate the object habitual location and third confidence level of the target to be retrieved based on the spatiotemporal feature prediction model and the mapping relationship.

[0112] In some embodiments of the invention, the semantic result module 430 is specifically used to: compare a first confidence level of the object's disappearance location with a first threshold; if the first confidence level is greater than or equal to the first threshold, then determine the object's disappearance location as the final location. If the first confidence level is less than the first threshold, the second confidence level of the object carrying the inferred position is compared with the second threshold. If the second confidence level is greater than or equal to the second threshold, the inferred position of the object is determined as the final position. If the second confidence level is less than the second threshold, then the object's habitual position is determined as the final position; The weighted sum of the first confidence level, the second confidence level, and the third confidence level is set as the fusion weight of the final position; The preset semantic generation module is invoked to generate the semantic detection result of the target to be retrieved according to the final position and the fusion weight.

[0113] In some embodiments of the invention, the semantic result module 430 further includes: a verification unit, used to determine that the fusion weight is less than a third threshold, and to control the mobile monitoring device to perform video verification of the habitual location of the object.

[0114] In some embodiments of the invention, the verification feedback module 440 is specifically used to: extract the final location corresponding to the target to be retrieved within the semantic detection result, and determine the time point at which the target to be retrieved is located at the final location based on the historical monitoring video of the target to be retrieved within the data source; At least one frame of the historical surveillance video is extracted from the specified time point as the surveillance image evidence.

[0115] The object retrieval device provided in the embodiments of the present invention can execute the object retrieval method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of executing the method.

[0116] Example 6 Figure 7 This is a schematic diagram of the structure of an electronic device implementing the object retrieval method of this invention. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices (such as helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the invention described and / or claimed herein.

[0117] like Figure 7As shown, the electronic device 10 includes at least one processor 11 and a memory, such as a read-only memory (ROM) 12 or a random access memory (RAM) 13, communicatively connected to the at least one processor 11. The memory stores computer programs executable by the at least one processor. The processor 11 can perform various appropriate actions and processes based on the computer program stored in the ROM 12 or loaded from storage unit 18 into the RAM 13. The RAM 13 can also store various programs and data required for the operation of the electronic device 10. The processor 11, ROM 12, and RAM 13 are interconnected via a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0118] Multiple components in electronic device 10 are connected to I / O interface 15, including: input unit 16, such as keyboard, mouse, etc.; output unit 17, such as various types of displays, speakers, etc.; storage unit 18, such as disk, optical disk, etc.; and communication unit 19, such as network card, modem, wireless transceiver, etc. Communication unit 19 allows electronic device 10 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0119] Processor 11 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, digital signal processors (DSPs), and any suitable processor, controller, microcontroller, etc. Processor 11 performs the various methods and processes described above, such as object retrieval methods.

[0120] In some embodiments, the object retrieval method may be implemented as a computer program tangibly contained in a computer-readable storage medium, such as storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or installed on electronic device 10 via ROM 12 and / or communication unit 19. When the computer program is loaded into RAM 13 and executed by processor 11, one or more steps of the object retrieval method described above may be performed. Alternatively, in other embodiments, processor 11 may be configured to perform the object retrieval method by any other suitable means (e.g., by means of firmware).

[0121] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0122] Computer programs used to implement the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when executed by the processor, the computer programs cause the functions / operations specified in the flowcharts and / or block diagrams to be performed. The computer programs may be executed entirely on a machine, partially on a machine, or as a standalone software package, partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0123] In the context of this invention, a computer-readable storage medium can be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, apparatus, or device. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination thereof. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0124] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0125] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or middleware components (e.g., application servers), or frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), blockchain networks, and the Internet.

[0126] A computing system can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a hosting product within the cloud computing service system to address the shortcomings of traditional physical hosts and VPS services, such as high management difficulty and weak business scalability.

[0127] It should be understood that the various forms of processes shown above can be used, with steps reordered, added, or deleted. For example, the steps described in this invention can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this invention can be achieved, and this is not limited herein.

[0128] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. An object retrieval method, characterized in that, The method includes: Obtain the target to be retrieved corresponding to the object query request, and obtain the recorded trajectory of the target to be retrieved from the data source of the home monitoring system. The recorded trajectory includes the current target movement trajectory, the historical target movement trajectory, and the user movement trajectory. Based on the recorded trajectory, the object disappearance location, the object's usual location, and the object's estimated location are generated for the target to be retrieved; A semantic detection result is generated based on the object's disappearance location, the object's usual location, and the inferred location carried by the object. Obtain surveillance image evidence from the data source that matches the semantic detection result, and generate object query feedback results based on the semantic detection result and the surveillance image evidence.

2. The method according to claim 1, characterized in that, The data source includes a home vision system, a home robot's positioning and navigation system, and a home visual memory repository. The step of obtaining the recorded trajectory of the target from the data source of the home monitoring system includes: Read the real-time video stream of the home vision system, identify the current spatial position of the target to be retrieved in the real-time video stream based on the preset target detection module, and arrange the current spatial positions of the target to be retrieved in the real-time video stream according to the acquisition time of the target to be retrieved in the real-time video stream to form the current target movement trajectory; Find the map location of the target to be retrieved within the positioning and navigation system, and arrange the map location points in chronological order to form the current target's movement trajectory; Determine the semantic description information of the target to be retrieved, and extract at least one historical target movement trajectory of the target to be retrieved from the family visual memory repository according to the semantic description information; Obtain target-related image information of the target to be retrieved in the family visual memory repository, determine the user of the target to be retrieved based on the target-related image information, and obtain the user's movement trajectory in the family visual memory repository.

3. The method according to claim 2, characterized in that, The step of determining the user of the target to be retrieved based on the target-related image information includes: Identify user images within the target-associated image information that have a spatial overlap with the target to be retrieved; The user corresponding to the target user image in each of the user images whose intersection-union ratio of the target image to be retrieved is greater than a preset threshold is determined as the user holding the target image.

4. The method according to claim 1, characterized in that, The process of generating the object disappearance location, the object's usual location, and the object's inferred location based on the recorded trajectory includes: Extract the current target movement trajectory within the recorded trajectory, obtain the current video information of the target to be retrieved in the home space within the data source according to the current target movement trajectory, determine the target home location within the current video information whose disappearance time is greater than a threshold as the object disappearance location, and generate a first confidence level according to the time difference between the image acquisition time of the object disappearance location and the current time. Extract the user's movement trajectory within the recorded trajectory, extract user video information of the target to be retrieved in the home space of the target to be retrieved in the data source according to the user's movement trajectory, and generate the object carrying the inferred location and second confidence level based on the spatial position relationship between the target to be retrieved and the user in the current video information and the user's spatial position relationship according to the trajectory time series prediction model. Extract the historical target movement trajectory within the recorded trajectory and the family space topology information within the data source, establish a mapping relationship between the historical trajectory points within the historical target movement trajectory and the spatial location within the family space topology information, and generate the object habitual location and third confidence level of the target to be retrieved based on the spatiotemporal feature prediction model and the mapping relationship.

5. The method according to claim 1 or 4, characterized in that, The step of generating semantic detection results based on the object's disappearance location, the object's habitual location, and the inferred location carried by the object includes: The first confidence level of the object's disappearance location is compared with a first threshold. If the first confidence level is greater than or equal to the first threshold, the object's disappearance location is determined as the final location. If the first confidence level is less than the first threshold, the second confidence level of the object carrying the inferred position is compared with the second threshold. If the second confidence level is greater than or equal to the second threshold, the inferred position of the object is determined as the final position. If the second confidence level is less than the second threshold, then the object's habitual position is determined as the final position; The weighted sum of the first confidence level, the second confidence level, and the third confidence level is set as the fusion weight of the final position; The preset semantic generation module is invoked to generate the semantic detection result of the target to be retrieved according to the final position and the fusion weight.

6. The method according to claim 5, characterized in that, Also includes: If the fusion weight is determined to be less than a third threshold, the mobile monitoring device is controlled to perform video verification of the object's habitual location.

7. The method according to claim 1, characterized in that, The step of obtaining surveillance image evidence within the data source that matches the semantic detection result includes: Extract the final location of the target to be retrieved from the semantic detection results, and determine the time point when the target to be retrieved is located at the final location based on the historical monitoring video of the target to be retrieved in the data source; At least one frame of the historical surveillance video is extracted from the specified time point as the surveillance image evidence.

8. An object retrieval device, characterized in that, The device includes: The trajectory extraction module is used to obtain the target to be retrieved corresponding to the object query request, and to obtain the recorded trajectory of the target to be retrieved from the data source of the home monitoring system. The recorded trajectory includes the current target movement trajectory, the historical target movement trajectory, and the user movement trajectory. The location reasoning module is used to generate the object disappearance location, the object's usual location, and the object's estimated location based on the recorded trajectory. The semantic results module is used to generate semantic detection results based on the object's disappearance location, the object's habitual location, and the object's inferred location. The verification and feedback module is used to obtain monitoring image evidence that matches the semantic detection result within the data source, and generate object query feedback results based on the semantic detection result and the monitoring image evidence.

9. An electronic device, characterized in that, The electronic device includes: At least one processor; and A memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform the object retrieval method according to any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that cause a processor to execute the object retrieval method as described in any one of claims 1-7.

Images