Object searching method, scene information construction method and related equipment

Abstract

The embodiment of the invention provides an object searching method, a scene information construction method and related equipment, and the method comprises the steps: determining a secondary scene identifier in a main scene based on a scene image, and constructing the multi-level information of each object through obtaining the detailed information of the objects in the secondary scene, and the scene information database is constructed based on the multi-level information of the object, so that the scene information database can contain detailed description information of the object in the scene, and a basis is provided for providing a position for searching the object for a user based on the scene information database. The method comprises the steps of determining a target object to be searched by a user by responding to an object searching operation of the user; the multi-level information of the target object is determined in the preset scene information data set, and the position information of the target object is output to the user based on the multi-level information of the target object, so that the object required by the user is quickly determined, detailed information of the object is provided for the user, and the user is helped to quickly and accurately find the required object.

Description

Technical Field

[0001] The present application relates to the field of terminals, and in particular to an object search method, a scene information construction method and related equipment. Background Art

[0002] In various everyday scenarios, users struggle to quickly locate objects in a scene due to the wide variety of objects, a lack of systematic planning for their locations, or changes in their locations. For example, in a home setting, family members often struggle to locate household items, making it difficult to locate the items they need. Related object location solutions, such as attaching labels to each object and determining its location based on the label, are not only time-consuming and labor-intensive, but can also become ineffective due to labels falling off or being damaged, making it difficult for users to find the items they need in a timely manner. Summary of the Invention

[0003] In view of the above, it is necessary to provide an object search method, a scene information construction method and related equipment, which can solve the problem that related technologies are difficult to quickly and accurately prompt the user of the object location without relying on object labels.

[0004] In a first aspect, the present application provides an object search method, which includes: responding to a user's object search operation, determining a target object corresponding to the object search operation; determining multi-level information of the target object based on a preset scene information database, wherein the multi-level information is constructed based on a scene image, object information of objects in the scene, and positional relationships between multiple objects; and outputting the position information of the target object based on the multi-level information of the target object.

[0005] Through the above technical solution, the target object that the user wants to find can be determined by responding to the user's object search operation; by determining the multi-level information of the target object in a preset scene information database, the location information of the target object is output to the user based on the multi-level information of the target object, thereby quickly determining the object required by the user and providing the user with detailed information of the object, helping the user to quickly and accurately find the required object.

[0006] In one possible implementation, responding to the user's object search operation and determining the target object corresponding to the object search operation includes: responding to the user's object search operation performed through voice instructions, receiving and recognizing the voice instructions, and obtaining a recognition result, wherein the language instructions include preset keywords; and / or responding to the user's object search operation performed through interface interaction based on a preset user interface, receiving and recognizing the interface interaction data, and obtaining a recognition result; and determining the target object through the recognition result.

[0007] In one possible implementation, determining the target object through the recognition result includes: determining the object name in the recognition result, and determining the target object based on the object name; or determining object attribute information in the recognition result, and determining the target object based on the object attribute information.

[0008] In a possible implementation, the object attribute information includes one or more of object function, object appearance, object location, and object identification.

[0009] In a possible implementation, determining the multi-level information of the target object based on a preset scene information database includes: determining a matching object that matches the target object from the scene information database, and acquiring the multi-level information of the matching object.

[0010] The above technical solution can obtain interaction data corresponding to a user's object search operation through various methods. By analyzing and identifying this interaction data, the target object corresponding to the user's object search operation can be quickly and accurately determined. During the target object search process, multi-level information related to the target object can be efficiently retrieved and matched, enabling users to quickly obtain the required information in different scenarios.

[0011] In a possible implementation, the method further includes: outputting the location information of the target object in one or more of the following output modes: outputting the multi-level information using voice; and / or displaying the multi-level information on a preset graphical user interface.

[0012] Through the above technical solution, a variety of output methods (such as voice and graphical user interface display) can be used to output multi-level information of the target object, and flexible display forms (including text, images, animations, etc.) can be used to enhance the readability and accessibility of multi-level information. At the same time, it supports users to personalize the display form, thereby improving the user experience.

[0013] In one possible implementation, the method further includes: if the user has access rights, responding to the user's object search operation; if the user does not have access rights, prompting the user to perform a permission verification operation, and responding to the user's object search operation after the user passes the permission verification.

[0014] This technical solution introduces an access rights control mechanism that verifies whether a user has permission to access a specific database or object search application before responding to an object search operation. This ensures the legitimacy and security of information access, improving the security of information management and the standardization of user operations.

[0015] In a second aspect, the application provides a scene information construction method, which comprises: acquiring a scene image of a main scene, the main scene comprising a plurality of sub-scenes; identifying a sub-scene based on a first type of object in the scene image, and establishing a corresponding relationship between the scene image and an identifier of the sub-scene; identifying a second type of object in the scene image, and acquiring object information of the second type of object; determining multi-level information of the second type of object based on the identifier of the sub-scene corresponding to the scene image, the object information of the second type of object, and a positional relationship between the second type of objects; and constructing a scene information database based on the multi-level information.

[0016] Through the above technical solution, detailed information of various objects in a scene can be acquired based on a scene image, multi-level information of each object can be constructed, and a scene information database can be constructed based on the multi-level information of the objects, so that the scene information database can contain detailed description information of the objects.

[0017] In a possible implementation, the object information comprises an object name and object attribute information, and the method further comprises: determining the positional relationship between the second type of objects based on the object information of the second type of objects.

[0018] Through the above technical solution, multi-dimensional detailed information of objects in a scene image can be determined, thereby providing a basis for constructing a scene information database.

[0019] In a possible implementation, the method of acquiring the multi-level information of the second type of objects comprises: determining first-level information based on the identifier of the sub-scene corresponding to the scene image; determining second-level information based on the object information of the second type of objects; determining third-level information based on the positional relationship between the second type of objects; establishing an association between the first-level information, the second-level information, and the third-level information of the second type of objects, to obtain multi-level information of the second type of objects; and taking information corresponding to the multi-level information of the second type of objects as the multi-level information of the second type of objects.

[0020] Through the above technical solution, detailed information of the second type of objects can be collected, and multi-level information of the second type of objects can be constructed based on the sub-scene to which the second type of objects belong, the object information of the second type of objects, and the positional relationship between the second type of objects and other objects.

[0021] In a possible implementation, the method further comprises: in response to an information entry operation of a user, receiving multi-level information of a third type of object, wherein the multi-level information of the third type of object comprises multi-level information of a second type of object corresponding to the third type of object and object information of the third type of object; and constructing the scene information database based on the multi-level information of the third type of object.

[0022] Through the above technical solution, it is possible to collect information about third-category objects that are difficult to obtain through image recognition of scene images, so that the scene information database can include multi-level information about third-category objects, which can meet the subsequent user's information requests for third-category objects.

[0023] In one possible implementation, determining the positional relationship between the second-category objects based on the object information of the second-category objects includes: determining a bounding box of each second-category object, and determining the nearest neighbor object of each second-category object based on the bounding box; determining the positional relationship between each second-category object and the corresponding nearest neighbor object based on the positional relationship between the bounding box of each second-category object and the bounding box of the nearest neighbor object, and determining the positional relationship between the second-category objects based on the positional relationship between each second-category object and the corresponding nearest neighbor object.

[0024] Through the above technical solution, the positional relationship between objects can be accurately obtained through bounding box position analysis.

[0025] In a possible implementation, determining the positional relationship between the second-category objects based on the object information of the second-category objects further includes: inputting the scene image into a preset visual language model, and outputting the positional relationship between the second-category objects using the visual language model.

[0026] Through the above technical solution, the visual language model can be used to deeply understand the image content in the scene image, and the positional relationship between objects can be output in the form of natural language, which facilitates the construction of multi-level information for each second-category object.

[0027] In a possible implementation, the information storage format of the scene information database includes a text format.

[0028] Through the above technical solution, the scene information database is stored in text format, which can reduce memory pressure and facilitate the retrieval of information in subsequent processes.

[0029] In a possible implementation manner, the method further includes: updating the scene information database based on a preset update condition.

[0030] In one possible implementation, the updating of the scene information database based on preset update conditions includes any of the following updating methods: updating the scene information database according to a preset update time node or update cycle; updating the scene information database in response to a user's operation to update the scene information database.

[0031] Through the above technical solution, the scene information database can be updated by automatic update or interaction with the user to determine the update, so that the scene information database has the latest and most accurate object information and can meet the needs of continuous monitoring and analysis of object locations.

[0032] In a third aspect, the present application provides a terminal device comprising a memory and a processor: wherein the memory is used to store program instructions; the processor is used to read and execute the program instructions stored in the memory, and when the program instructions are executed by the processor, the terminal device executes the above-mentioned object search method and scene information construction method.

[0033] In a fourth aspect, the present application provides a chip coupled to a memory in a terminal device, and the chip is used to control a processor of the terminal device to execute the above-mentioned object search method and scene information construction method.

[0034] In a fifth aspect, the present application provides a computer storage medium storing program instructions. When the program instructions are executed on a terminal device, the processor of the terminal device executes the above-mentioned object search method and scene information construction method.

[0035] In addition, the technical effects brought about by the third to fifth aspects can be found in the descriptions of the methods of each design in the above method section, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 This is a software architecture diagram of a terminal device provided in one embodiment of the present application.

[0037] Figure 2 This is a flowchart of a scene information construction method provided in one embodiment of the present application.

[0038] Figure 3 This is an example diagram of a multi-level semantic scene graph provided in one embodiment of the present application.

[0039] Figure 4 This is a flowchart of an object search method provided by an embodiment of the present application.

[0040] Figure 5 This is an example diagram of a wake-up method for an object search application provided in one embodiment of the present application.

[0041] Figure 6 This is an example diagram of displaying multi-level information provided by an embodiment of the present application.

[0042] Figure 7 This is a flowchart of a scene information construction method provided for another embodiment of the present application.

[0043] Figure 8 This is a flowchart of a scene information construction method provided for another embodiment of the present application.

[0044] Figure 9 This is a flowchart of an object search method provided by another embodiment of the present application.

[0045] Figure 10 This is a hardware architecture diagram of a terminal device provided in one embodiment of the present application. DETAILED DESCRIPTION

[0046] The terms "first" and "second" involved in an embodiment of the present application are only used for descriptive purposes and are not to be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, the features defined as "first" and "second" may explicitly or implicitly include one or more features. In the description of the embodiments of the present application, words such as "exemplary" or "for example" are used to indicate examples, illustrations or explanations. Any embodiment or design scheme described as "exemplary" or "for example" in an embodiment of the present application should not be interpreted as being more preferred or more advantageous than other embodiments or design schemes. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a concrete way.

[0047] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which this application belongs. The terms used in the specification of this application are for the purpose of describing specific embodiments only and are not intended to limit this application. It should be understood that, unless otherwise specified in this application, " / " means or. For example, A / B can mean A or B. "And / or" in this application is merely a way to describe the association relationship of associated objects, indicating that three relationships can exist. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. "At least one" means one or more. "Multiple" means two or more than two. For example, at least one of a, b or c can mean: a, b, c, a and b, a and c, b and c, a, b and c. In the absence of conflict, the following embodiments and features in the embodiments can be combined with each other.

[0048] With technological advancements and rising living standards, the number of items in users' daily lives has increased dramatically. These items are not only diverse but also frequently move around due to usage habits, cleaning, and space renovations. This high degree of dynamism and disorder poses a significant challenge to users in quickly locating the items they need.

[0049] In an example, in a home scenario, from daily necessities to electronic devices, to various decorations and collections, family members are often troubled by the forgetting of the location of household items, and it is difficult to determine the location information of the required items.

[0050] In addition, in addition to the above-mentioned daily life scenarios familiar to the user, when the user enters a strange environment, such as an airport, a hospital, a large shopping center, a museum, or a library, and the like, it is also a challenge to quickly find the location of a specific object in a complex and variable public place.

[0051] Related object location positioning schemes, although to some extent alleviate the problem of location forgetting, but its limitations are increasingly obvious. For example, paste a label on each object, determine the object location according to the label of each object, in addition to the problem of label easy to fall off, damage, the use of labels is also limited by the surface material of the object, label aesthetics and the overall environmental coordination. In addition, with the increase of the number of objects, the maintenance cost of the label system will also rise sharply, including regular update of location information, replacement of damaged labels, etc., which will bring unnecessary burden.

[0052] With the rise of Internet of Things technology, wireless sensors, wireless radio frequency identification tags, Bluetooth beacons and other intelligent devices begin to be applied to the field of object location positioning. These technologies realize real-time monitoring and data transmission of object location by equipping objects with small tracking devices. However, the popularization of Internet of Things technology also faces some challenges, such as device cost, privacy protection, data security and the like.

[0053] How to effectively control the cost, protect the privacy and data security of the user while ensuring the convenience of the user is a problem that needs to be considered in the process of technical development in the field of object location positioning.

[0054] In order to solve the above problems, the embodiments of the present application provide a scene information construction method, which can obtain detailed information of a plurality of objects in a scene based on a scene image, construct multi-level information of each object and construct a scene information database based on the multi-level information of the object, so that the scene information database can contain detailed description information of the object. The embodiments of the present application also provide an object finding method, which can determine a target object to be found by a user by responding to the object finding operation of the user, determine the multi-level information of the target object in the preset scene information database, and output the location information of the target user to the user based on the multi-level information of the target object, so as to quickly determine the required object of the user, provide detailed information of the object for the user, and help the user to quickly and accurately find the required object.

[0055] The scene information construction method and the object search method are applied to various terminal devices including display devices, such as mobile phones, tablet computers, wearable devices, camera devices, computers, self-mobile devices, etc. The terminal device includes an application processor, which is used to run the operating system. Figure 1 The software structure of the terminal device is illustrated.

[0056] See Figure 1 The figure shows the software architecture of the terminal device provided in the embodiment of the present application. The layered architecture divides the software into several layers, each with a clear role and division of labor. The layers communicate with each other through software interfaces. For example, the Android system is divided into the application layer 101, the framework layer 102, the Android runtime and system library 103, the hardware abstraction layer 104, the kernel layer 105, and the hardware layer 106 from top to bottom.

[0057] The application layer 101 may include a series of application packages, such as camera, gallery, calendar, call, map, navigation, WLAN, Bluetooth, music, video, short message, device control service, etc.

[0058] The framework layer 102 provides an application programming interface (API) and programming framework for the applications in the application layer. The application framework layer includes some predefined functions. For example, the application framework layer may include a window manager, content provider, view system, telephony manager, resource manager, notification manager, etc.

[0059] The window manager manages window applications. It can obtain the display size, determine whether a status bar is present, lock the screen, and take screenshots. The content provider stores and retrieves data and makes it accessible to applications. This data can include video, images, audio, incoming and outgoing calls, browsing history and bookmarks, and the phone book. The view system includes visual controls, such as those for displaying text and images. The view system is used to build applications. The display interface can consist of one or more views. For example, the display interface for a text message notification icon can include a view for displaying text and a view for displaying images. The phone manager provides communication functions for the terminal device, such as managing call status (including connected and ended calls). The resource manager provides applications with various resources, such as localized strings, icons, images, layout files, and video files. The notification manager enables applications to display notifications in the status bar. These messages can be displayed briefly and then disappear automatically, without user interaction. For example, the notification manager is used to notify download completions and message reminders. The notification manager can also be a notification that appears in the system's top status bar in the form of an icon or scrolling text bar, such as a notification from an application running in the background, or a notification that appears on the screen in the form of a dialog window. For example, a text message may be displayed in the status bar, a notification sound may be emitted, the terminal device may vibrate, or an indicator light may flash.

[0060] The Android Runtime consists of a core library and a virtual machine. The Android Runtime is responsible for scheduling and managing the Android system. The core library consists of two parts: one for Java-based functions and the other for the Android core library.

[0061] The application layer 101 and the framework layer 102 run in a virtual machine. The virtual machine executes the Java files in the application layer and the framework layer as binary files. The virtual machine is used to perform functions such as object lifecycle management, stack management, thread management, security and exception management, and garbage collection.

[0062] The system library 103 may include multiple functional modules, such as a surface manager, a media library, a 3D graphics processing library (such as Open GLES), a 2D graphics engine (such as SGL), etc.

[0063] The surface manager is configured to manage the display subsystem and provide a fusion of 2D and 3D layers for multiple applications. The media library supports playback and recording of various commonly used audio, video formats, and static image files. The media library can support various audio and video encoding formats, such as MPEG4, H.264, MP3, AAC, AMR, JPG, PNG, etc. The 3D graphics processing library is configured to implement 3D graphics drawing, image rendering, synthesis, and layer processing, etc. The 2D graphics engine is a drawing engine for 2D drawing.

[0064] The hardware abstraction layer 104 runs in the user space, encapsulates the kernel layer driver, and provides a calling interface to the upper layer.

[0065] The kernel layer 105 is a layer between hardware and software. The kernel layer 105 at least includes display drivers, touch control drivers, audio drivers, and sensor drivers.

[0066] The kernel layer 105 is the core of the operating system of the terminal device, is the first layer of software expansion based on hardware, provides the most basic functions of the operating system, is the basis of the operating system, is responsible for managing the processes, memory, device drivers, files, and network systems of the system, and determines the performance and stability of the system. For example, the kernel layer can determine the operation time of an application program on a certain part of hardware.

[0067] The kernel layer 105 includes programs closely related to hardware, such as interrupt handling programs and device drivers, and includes basic, common, and high-frequency running modules, such as clock management modules and process scheduling modules, and includes critical data structures. The kernel layer can be set in the processor or fixed in the internal memory.

[0068] The hardware layer 106 includes the hardware of the terminal device, such as a display screen, a key, a camera, etc.

[0069] Referring to FIG. 1, a flowchart of a scene information construction method provided by an embodiment of the present application is shown. The scene information construction method is applied to a terminal device, and specifically includes the following flow. Figure 2

[0070] S201, a scene image of a main scene is acquired.

[0071] ​In some embodiments of the present application, the main scene may include multiple sub-scenes, wherein the main scene may represent a larger range of scenes, used to indicate a wider or more comprehensive range of scenes. The sub-scene may be the result of subdividing the main scene according to actual needs, and the subdivision may be any level of a multi-level division, or the result of a single division. In one example, if the main scene is the residence of user A, the sub-scene may be the kitchen, bedroom, study, etc. divided according to the function of the room. In another example, if the main scene is the residence of user A, considering the difference in geographical location, the sub-scene may be the residence of user A in different locations, for example, user A's residence zs1 at location DD1, and user A's residence zs2 at location DD2. In another example, if the main scene is the residence of user A, the sub-scene may be the kitchen, bedroom, study, etc. in user A's residence zs1 at location DD1.

[0072] In some embodiments of the present application, a secondary scene can be further subdivided into more specific sub-scenes based on actual needs. In one example, if the secondary scene is a kitchen, the sub-scenes may include the kitchen's cooking area, washing area, storage area, etc. In another example, if the secondary scene is user A's residence zs1 at location DD1, the sub-scenes may include residence zs1's living room, balcony, bedroom, kitchen, etc.

[0073] Through the above embodiments, a clear hierarchical relationship definition is constructed for actual application scenarios, which significantly enhances the organization, processing efficiency and accuracy of data, while improving the convenience of information retrieval, providing strong support for the construction, management and application of scene information databases, and ensuring that data can serve actual application needs efficiently and accurately.

[0074] In some embodiments of the present application, the method for acquiring the scene image of the main scene can be determined according to actual needs. In one example, if the actual scene is a home scene, the acquisition method may include but is not limited to a combination of one or more of the following methods: receiving a scene image taken by a user through a terminal device (such as a mobile phone, tablet, camera, etc.), receiving a scene image taken from a mobile device (a device with moving and shooting functions, such as a smart sweeping robot, a smart camera, an indoor small drone, etc.), and receiving a scene image input by the user. In one example, the scene image can be a single-channel image, such as a grayscale image; the scene image can also be a multi-channel image, such as an RGB image.

[0075] In some embodiments of the present application, a scene video of the primary scene may also be obtained, and a scene image may be obtained by extracting video frames from the scene video. It is understood that the scene image of the primary scene may include the scene image of the secondary scene. The scene images obtained in the embodiments of the present application are collected, processed, and used under the premise of complying with user privacy protection requirements.

[0076] Through the above embodiments, scene images can be acquired in a variety of ways, and all image collection is carried out under the premise of ensuring user privacy protection, which not only meets practical application needs but also respects user privacy.

[0077] S202 : Identify a sub-scene based on the first type of objects in the scene image, and establish a corresponding relationship between the scene image and the identifier of the sub-scene.

[0078] In some embodiments of the present application, a scene image may include scene images of multiple sub-scenes. In order to distinguish the sub-scenes, the sub-scenes can be identified based on the first-category objects in the sub-scenes, thereby determining the identification of the sub-scenes (such as name, number, etc.). Among them, the first-category objects include identification objects of the sub-scenes, which can be set according to actual needs. For example, if the sub-scene is a kitchen, the first-category objects may be cooking utensils, etc.; if the sub-scene is a bedroom, the first-category objects may be beds, wardrobes, etc.; if the sub-scene is a study, the first-category objects may be desks, bookshelves, etc. In other embodiments, the first-category objects may also be sub-scenes obtained by subdividing the sub-scenes. For example, if the sub-scene is a kitchen, the first-category objects may be the cooking area, washing area, storage area, etc. of the kitchen.

[0079] In one example, identifying a sub-scene based on a first-category object in the sub-scene may include: preprocessing the scene image, for example, the preprocessing may include grayscale conversion, image denoising, image contrast enhancement, etc. Feature extraction may be performed on the preprocessed scene image, for example, feature extraction may be performed on the preprocessed scene image based on image features of the first-category object (e.g., color, texture, shape, edge, etc.); based on the feature data obtained by feature extraction, a preset method (e.g., a deep learning model) is used to classify the feature data to determine whether the object corresponding to the feature data is a first-category object. If the object corresponding to the feature data is a first-category object, the identifier of the sub-scene corresponding to the first-category object is confirmed as the identifier of the sub-scene corresponding to the scene image.

[0080] In some embodiments of the present application, if the scene image is a video frame extracted from a scene video, it is also possible to determine that the identifiers of the sub-scenes corresponding to multiple consecutive video frames are the same name based on the timing information of the video frame. For example, the identifier of the sub-scene corresponding to the i-th video frame and the identifier of the sub-scene corresponding to the i+n-th video frame can be identified. If the identifier of the sub-scene corresponding to the i-th video frame is consistent with the identifier of the sub-scene corresponding to the i+n-th video frame, it is determined that the identifiers of the sub-scenes of n+1 consecutive video frames from the i-th video frame to the i+n-th video frame are the same, and are all identifiers of the sub-scene corresponding to the i-th video frame. Wherein, i and n represent preset positive integers, which can be set according to actual needs. For example, n is 5, and i+n is less than or equal to the total number of video frames. This application does not impose specific restrictions on this.

[0081] In one example, the identifier of the secondary scene may be a kitchen. In another example, the identifier of the secondary scene may further include the name of the upper-level scene (eg, main scene) to which the secondary scene belongs. For example, the identifier of the secondary scene may be the kitchen of residence ZS1.

[0082] Through the above embodiment, the sub-scene corresponding to the scene image can be determined by the identification object of the sub-scene, and a corresponding relationship can be established between the scene image and the identification of the sub-scene, so as to realize the division of the scene image according to the sub-scene, which facilitates the construction of multi-level information of each object in the scene image.

[0083] S203: Identify the second type of objects in the scene image and obtain object information of the second type of objects.

[0084] In some embodiments of the present application, the second category of objects includes objects that can be identified through images. For example, the second category of objects may include objects that are not stored or not completely obscured. The second category of objects may also be objects of a pre-defined specific category. It is understood that there may be a corresponding relationship between the second category of objects and the first category of objects. For example, the second category of objects may include the first category of objects, and the second category of objects may have a positional relationship with the first category of objects. For example, the second category of objects may include a desk in the first category of objects, and the second category of objects may be a water cup placed on the desk in the first category of objects.

[0085] In some embodiments of the present application, the second-category objects in the scene image can be traversed and object information of the second-category objects can be obtained. If the scene image includes multiple time-sequential video frames, it can be determined whether the second-category objects in each video frame have already been identified in other video frames. For example, if a water glass SB1 has already been identified and its object information has been obtained in video frame SPZ1, then when the water glass SB1 is identified in video frame SPZ2, the step of obtaining object information for the water glass SB1 can be skipped, thereby avoiding repeated information acquisition for the second-category objects.

[0086] In some embodiments of the present application, the object information of the second category of objects may include an object name, and the object name may be used to indicate the category of the object. The method for identifying the object name of the second category of objects in the scene image may refer to the method for identifying the first category of objects. In one example, the object name may include multiple synonymous object names. For example, seating, chair, armchair, and wooden chair may be object names belonging to the same object. In another example, the object name may also include multi-level names. For example, the first-level name of an object may be clothing, the second-level name may be women's clothing, the third-level name may be skirt, the fourth-level name may be sleeveless long skirt, etc.

[0087] In some embodiments of the present application, the object information of the second category of objects may include object attribute information, which may be used to indicate information in multiple dimensions of the object, thereby providing a detailed description of the object. The object attribute information may include, but is not limited to, one or more of the following: object function, object appearance (e.g., color, shape, size, texture, material, etc.), object location, and object identification.

[0088] In some embodiments of the present application, an object function may describe a service, operation, or purpose that an object can perform or provide, and the object function of an object may be determined based on the object's name. For example, the object function of a teacup may include being used for drinking tea, and the object function of a coffee cup may include being used for drinking coffee.

[0089] In some embodiments of the present application, the appearance of an object may include a comprehensive description of the visual performance of the object, including but not limited to one or more combinations of the following: color, shape, size, texture, material, etc.

[0090] In some embodiments of the present application, the object position may include the specific position or coordinates of the object in space. For example, the object position may be the coordinates in the coordinate system corresponding to the scene image. For example, according to a preset rule, the coordinate system corresponding to each scene image may be determined. For example, the coordinate system corresponding to the scene image may be established with the lower left corner of the scene image as the origin, the horizontal direction as the X-axis, and the vertical direction as the Y-axis. When identifying a second type of object, a bounding box of the second type of object is generated, and the object position of the second type of object in the scene image is determined based on the coordinates of the center point of the bounding box in the scene image coordinate system.

[0091] In some embodiments of the present application, the object identifier may include information or symbols for uniquely identifying the object, which can be used to avoid confusion between similar objects. For example, the bounding boxes of the second type of objects can be numbered, and the corresponding object identifier can be determined based on the number of the bounding box.

[0092] Through the above embodiments, the multi-dimensional detailed information of each object in the scene image can be determined, providing a basis for constructing a scene information database.

[0093] S204 : Determine multi-level information of the second category objects based on the identifier of the sub-scene corresponding to the scene image, the object information of the second category objects, and the positional relationship between the second category objects.

[0094] In some embodiments of the present application, the multi-level information of the second category objects includes an identifier of a sub-scene corresponding to the scene image, object information of the second category objects, and positional relationships between the second category objects.

[0095] In some embodiments of the present application, the method for obtaining the positional relationship between the second category objects includes but is not limited to a combination of one or more of the following methods: determining the positional relationship between the second category objects based on the positional relationship between the bounding boxes of the second category objects; and obtaining the positional relationship between the second category objects based on a visual language model.

[0096] In some embodiments of the present application, the positional relationship between the second-category objects is determined based on the positional relationship between the bounding boxes of the second-category objects. The positional relationship between the second-category objects can be determined based on the object information (e.g., object position) of the second-category objects. For example, the bounding box of each second-category object can be determined, and the nearest neighbor object of each second-category object can be determined based on the bounding box. Based on the positional relationship between the bounding box of each second-category object and the bounding box of the nearest neighbor object, the positional relationship between each second-category object and the corresponding nearest neighbor object is determined, and the positional relationship between the second-category objects is determined based on the positional relationship between each second-category object and the corresponding nearest neighbor object.

[0097] In some embodiments of the present application, the nearest neighbor of the second-category object and the positional relationship between the second-category object and the corresponding nearest neighbor can be determined based on a variety of parameters. For example, the various parameters may include, but are not limited to, one or more combinations of the following: the degree of overlap between the bounding boxes of the second-category objects, the distance between the coordinates of the center points of the bounding boxes, the angle between the center points of the bounding boxes and the horizontal or vertical axis, etc. This application does not impose specific limitations on this.

[0098] In some embodiments of the present application, the positional relationship between the second-category objects is acquired based on a visual language model. A scene image (e.g., including bounding boxes of the second-category objects) can be input into a preset visual language model, and the positional relationship between the second-category objects can be output using the visual language model. The visual language model can be a pre-trained Vision-Language Model (VLM) that can simultaneously process and understand both visual (image) and language (text) modal information.

[0099] For example, visual language models can include but are not limited to image region description models (such as Region GPT (Generative Pre-trained Transformer), Dense Cap, GPT4ROI (Region-Of-Interest), Shikra), visual question answering models (such as Intern GPT, Det GPT), visual relationship detection models (Visual Relation Detection), scene graph generation models (Scene Graph Generation), etc.

[0100] In one example, a prompt word can be set for the visual language model, which can be used to prompt the visual language model to perform a corresponding function. For example, the prompt word can describe the attributes of each object within the bounding box (such as function, color, material, etc.) and its relationship with surrounding objects.

[0101] In one example, the positional relationship between the second objects may be: glasses are on the desk, with a glasses case and a mouse nearby. In another example, the positional relationship between the second objects may be: a mobile phone is above a book on a small table next to the bed.

[0102] Through the above embodiments, the positional relationship between the second-category objects can be flexibly and accurately obtained through methods such as bounding box position analysis and visual language model technology, which can deeply understand the image content in the scene image. The positional relationship between objects can be output in the form of natural language, which facilitates the construction of multi-level information for each second-category object.

[0103] In some embodiments of the present application, first-level information can be determined based on the identifier of the secondary scene corresponding to the scene image; second-level information can be determined based on the object information of the second-category objects; third-level information can be determined based on the positional relationship between the second-category objects; associations can be established between the first-level information, second-level information, and third-level information of the second-category objects to obtain a multi-level relationship between the second-category objects; and information corresponding to the multi-level relationship can be used as the multi-level information of the second-category objects. In another example, the second-category objects can also include zero-level information, which can include the identifier of the parent scene to which the secondary scene belongs (e.g., the identifier of the main scene).

[0104] In one example, the multi-level information of the second category of objects can be: zero-level information: residence ZS1; first-level information: bedroom WS1; second-level information: a red round-mouthed coffee cup that can be used for drinking coffee, numbered BZ1; third-level information: located on a small table next to the bed, with glasses, headphones, and mobile phones nearby. In another example, the multi-level information of the second category of objects can also include information described in natural language. The multi-level information of the second category of objects can be: for example, a red round-mouthed coffee cup is in the bedroom WS1 of residence ZS1, specifically located on a small table next to the bed, with glasses, headphones, mobile phones and other items nearby. In another example, the multi-level information can also be presented as follows: Figure 3 The multi-level semantic scene graph shown.

[0105] Through the above embodiment, detailed information of the second-category object can be collected, and multi-level information of the second-category object can be constructed based on information such as the sub-scene to which the second-category object belongs, the object information of the second-category object itself, and the positional relationship between the second-category object and other objects. The semantic information of the second-category object can be obtained in a manner consistent with natural language description, which facilitates the user to understand the multi-level information of the second-category object. The multi-level information of the second-category object can also be presented in the form of a multi-level semantic scene graph, which facilitates the user to quickly determine the object information based on the hierarchical relationship from top to bottom.

[0106] S205: Build a scene information database based on the multi-level information.

[0107] In some embodiments of the present application, the scene information database may include multi-level information of the second category of objects. In addition, the scene information database may also include multi-level information of the third category of objects. The third category of objects may include objects that are difficult to determine through image recognition, and the third category of objects may have a corresponding relationship with the second category of objects (or the first category of objects). For example, the correspondence between the third category of objects and the second category of objects may include, but is not limited to: the third category of objects are stored in the corresponding second category of objects, the third category of objects are blocked by the second category of objects, etc. For example, the third category of objects may be an ID card stored in a drawer of a desk, a cup stored in a cupboard, a shoe blocked by a bed, etc.

[0108] In some embodiments of the present application, for third-category objects, a method can be used to respond to a user's information entry operation and receive multi-level information about the third-category object; and a scene information database can be constructed based on the multi-level information about the third-category object. For example, a variety of information entry methods can be provided to the user for the user to perform information entry operations, including but not limited to: providing the user with a graphical user interface, responding to the user's input operation in the graphical user interface, recognizing the user's input text to determine the multi-level information about the third-category object; and responding to the user's voice input operation, recognizing the user's input voice to obtain the multi-level information about the third-category object. The graphical user interface provided to the user can be the application interface of an application integrated with the scene information construction method provided in the embodiments of the present application.

[0109] In some embodiments of the present application, an icon corresponding to the second-category object may be displayed to the user in a graphical user interface, and in response to the user clicking on the icon, multi-level information about the second-category object may be displayed. In response to the user's editing operation in the display interface for the multi-level information about the second-category object, the user may enter information about the third-category object corresponding to the second-category object. In one example, an interactive method such as voice or text displayed in a graphical user interface may be used to sequentially inquire about the name of the object to be recorded, the identifier of the sub-scene in which the object resides, object attribute information, and the positional relationship between the object and other objects. The user's voice or typed input may be received and parsed into text data, and the parsed text data may be added to the scene information database.

[0110] In some embodiments of the present application, the multi-level information of the third-category object includes the multi-level information of the second-category object corresponding to the third-category object, and the object information of the third-category object (e.g., object name, object attribute information). In one example, the multi-level information of the third-category object may include: the multi-level information of the second-category object corresponding to the third-category object: residence ZS1 (zero-level information of the second-category object); bedroom WS1 (first-level information of the second-category object); wooden table ZZ1 with a rectangular tabletop (second-level information of the second-category object); wooden table ZZ1 is close to the window, with items such as a computer, mouse, books, and keyboard on it, and wooden table ZZ1 has three drawers (third-level information of the second-category object); and identity card SFZ1 is stored in the middle drawer of wooden table ZZ1 (object information of the third-category object).

[0111] Through the above embodiments, it is possible to collect information about third-category objects that is difficult to obtain through image recognition of scene images, so that the scene information database can include multi-level information about third-category objects, which can meet the subsequent user's information requests for third-category objects.

[0112] In some embodiments of the present application, the scene information database may further include multi-level information on a fourth category of objects. The fourth category of objects may include objects capable of autonomous movement. For example, the fourth category of objects may include, but are not limited to, animals, self-moving devices, etc. For example, the fourth category of objects may include pets, sweepers, etc. The multi-level information on the fourth category of objects may include, but is not limited to, one or more of the following combinations: an identifier of the sub-scene to which the fourth category of objects belongs, object information of the fourth category of objects (e.g., object name, object attribute information), and a positional relationship between the fourth category of objects and the second category of objects (or first category of objects).

[0113] In one example, object information of a fourth-category object may be received in response to a user input operation. The positional relationship between the fourth-category object and the second-category object (or the first-category object) may be: a vacuum cleaner is located under a coffee table (a second-category object (or the first-category object)); and a cat is located on a cat climbing frame (a second-category object (or the first-category object)).

[0114] In some embodiments of the present application, for the fourth category of objects, multiple methods can be used to construct their multi-level information. For example, this can include, but is not limited to, a combination of one or more of the following methods: (a) Identifying the fourth category of objects based on scene images and obtaining object information for the fourth category of objects, and constructing their multi-level information based on the object information of the fourth category of objects. The specific method can refer to the method for obtaining multi-level information for the second category of objects. The scene image can be a scene image captured by a terminal device (e.g., an intelligent monitoring device) or a scene image captured by a camera corresponding to the fourth category of objects. For example, if the fourth category of objects is a pet, the corresponding camera can be a surveillance camera worn by the pet; if the fourth category of objects is a self-moving device, the corresponding camera can be a camera of the self-moving device, etc. Continuous location monitoring of the fourth category of objects can be achieved based on multiple scene images with a time sequence. (b) Determining the multi-level information of the fourth category of objects based on the location information of the intelligent device corresponding to the fourth category of objects (e.g., a wireless sensor, a wireless radio frequency identification tag, a Bluetooth beacon, etc.).

[0115] In one example, the multi-level information of the fourth category object may include one or more combinations of the following: an identifier of the sub-scene to which the fourth category object belongs: the bedroom WS1 of the residence ZS1, object information of the fourth category object (such as object name, object attribute information): the sweeping machine SDJ1, brand PP1; the positional relationship between the fourth category object and the second category object (or the first category object): the sweeping machine SDJ1 is located under the small table beside the bed.

[0116] Through the above embodiment, the information of the fourth category of objects that can move autonomously can be collected, so that the scene information database can include multi-level information of the fourth category of objects, which can meet the subsequent user's information request requirements for the fourth category of objects.

[0117] In some embodiments of the present application, the information storage format of the scene information database includes but is not limited to text format. By using text format to store the scene information database, memory pressure can be reduced and subsequent retrieval of information therein can be facilitated.

[0118] In some embodiments of the present application, the scene information database can also be uploaded to a server or other device in communication connection with the terminal device, thereby relieving the application pressure of the terminal device, and also enabling the scene information database to be shared to other terminal devices through the server. The server can be a stand-alone server, a server cluster composed of multiple servers, or a cloud server capable of cloud computing.

[0119] In some embodiments of the present application, the objects in the scene can change due to various reasons (e.g., appearance of new objects, disappearance of old objects, movement of object positions, etc.). In order to have the latest and most accurate object information in the scene information database and to meet the needs of continuous monitoring and analysis, the scene information database can be updated based on preset update conditions. In an example, for different types of objects, different update conditions can be used to update their multi-level information, thereby updating the scene information database. For example, updating the scene information database includes but is not limited to deleting, adding, modifying, etc. data in the scene information database.

[0120] In some embodiments of the present application, the preset update condition can be a preset update time node or an update period, and the scene information database can be automatically updated according to the update condition. In an example, the update time node can be a fixed time point, for example, the 20th of each month. In another example, according to actual needs, every k hours of time node can be used as the update time node, where k can be a user-set value. The update condition is only an example and is not limited to this in actual application.

[0121] In some embodiments of the present application, since the fourth type of object can also cause the position of other objects to change during autonomous movement, the update time node can be determined according to the state of the fourth type of object. In an example, if the fourth type of object is a self-moving device, the time node when the self-moving device is in a preset state can be used as the update time node. For example, if the fourth type of object is a user, the preset state can include but is not limited to the user's sleep state, the state when the user leaves the main scene (or the secondary scene), etc. If the fourth type of object is a self-moving device, the preset state can include but is not limited to the stationary state, the charging state, and the full power state of the self-moving device. The state of the fourth type of object can be determined according to multiple scene images with time sequence.

[0122] In an example, the updated scene image can be obtained at the preset update time node, and the multi-level information of the second type of object (or the first type of object, the fourth type of object) can be obtained according to the updated scene image. In addition, the position information of the intelligent device of the fourth type of object can also be obtained at the preset update time node, thereby obtaining the multi-level information of the fourth type of object.

[0123] By setting preset update conditions, multi-level information of objects of the second category (or the first category), the fourth category, and so on can be automatically updated.

[0124] In some embodiments of the present application, the user may be prompted whether to update the scene information database, and the scene information database may be updated in response to the user's operation of updating the scene information database.

[0125] In one example, interactive methods such as voice announcements or graphical user interface text display can be used to prompt the user whether to update the scene information database; in response to the user's determination to update the scene information database, the scene information database is updated by collecting and analyzing updated scene images, receiving user input, and other operations.

[0126] In some embodiments of the present application, the user may be prompted at a preset update time to determine whether to update the scene information database. For example, the terminal device may send a prompt message to the user terminal via the server, and prompt the user on the graphical user interface of the user terminal whether to update the scene information database.

[0127] Through the above embodiments, the scene information database can be updated by automatic updating or updating through interaction with the user, so that the scene information database has the latest and most accurate object information and can meet the needs of continuous monitoring and analysis of object locations.

[0128] In some embodiments of the present application, the scene information database may also include the distribution of each object's location, determined based on the object's multi-level information. In one example, the scene information database may include the object's historical location, where the timestamp corresponding to the historical location may be, for example, the time when the scene image was acquired. By analyzing the object's historical location, the distribution of the object's location can be determined, including but not limited to the usual placement of objects (e.g., second-category objects) and the movement patterns of objects (e.g., fourth-category objects).

[0129] In another example, subsequent applications can be performed based on the distribution of the locations of objects according to actual needs. For example, these applications may include but are not limited to: optimizing the spatial layout within a residence based on the usual placement of objects to improve space utilization; optimizing merchandise display methods and marketing strategies based on the sales rate of merchandise at the merchandise display locations in retail stores; and monitoring the movement of important items in museums, libraries, and other places to determine abnormal behavior.

[0130] Through the above embodiments, the space usage pattern within the scene can be determined through the scene information database, and the spatial layout of the scene can be optimized, the user experience can be improved, and the scene security monitoring can be enhanced.

[0131] In some embodiments of the present application, the scene information database may also include associations between objects. In one example, the similarity between objects can be determined based on data such as object names or object attribute information, and associations can be established between objects whose similarity exceeds a preset similarity threshold. For example, it may be determined that items WP1 and WP2 belonging to the same brand PP1 have the same function and similar appearance. Therefore, an association can be established between items WP1 and WP2 of brand PP1.

[0132] Through the above embodiments, it is possible to deeply analyze the multi-dimensional information of objects in the scene information database, such as the object names and attributes, accurately evaluate and identify objects with high similarity, and establish association relationships between objects with high similarity, thereby realizing deep integration and intelligent interconnection of information, enriching the connotation of the scene information database, and providing more comprehensive and accurate support for subsequent data analysis, user recommendations and personalized services.

[0133] The scene information construction method provided in the embodiment of the present application can collect detailed information of various objects in the scene based on multiple methods, construct multi-level information of each object and construct a scene information database based on the multi-level information of the object, so that the scene information database can contain detailed description information of the object.

[0134] The above embodiment introduces a scene information construction method and constructs a scene information database. In order to provide information prompts to users based on the scene information database, for example, to help users determine the multi-level information of an object through the scene information database to quickly locate the object, the embodiment of the present application also provides an object search method. Figure 4 FIG. 1 is a flow chart of an object search method according to an embodiment of the present invention. The object search method is applied in a terminal device and specifically includes the following process.

[0135] S401 , responding to a user's object search operation, and determining a target object corresponding to the object search operation.

[0136] In some embodiments of the present application, an object search application may be pre-installed in the terminal device. Figure 2As shown in the flow, the object search application can provide the user with the positioning and search service for the specified object based on the pre-constructed scene information database. The object search application can be a program provided by the embodiments of the present application and integrated with the instruction code for implementing the object search method. The object search application can be a program with a voice assistant function, and the object search application can also include all the functions of the information acquisition program in the above embodiments. The object search application can call the scene information database in the above embodiments. In an example, if the terminal device is a computer open to all users in a bookstore, the object search application corresponding to the bookstore can be pre-installed in the computer, so as to facilitate the user to determine the position of the required book through the object search application.

[0137] In some embodiments of the present application, the terminal device can also download and install the object search application in response to the user operation. In an example, if the terminal device is a user's mobile phone, the object search application can be installed in the user's terminal device in response to the user's download and installation operation of the object search application.

[0138] In some embodiments of the present application, the terminal device can set the wake-up method of the object search application. Based on the set wake-up method of the object search application, the object search application can be woken up, and the object search application can be used to respond to the user's object search operation.

[0139] In some embodiments of the present application, different terminal devices can have different wake-up methods of the object search application. Taking the mobile phone terminal as an example, the wake-up method of the object search application can include but not limited to one or more methods as follows: (1) the icon corresponding to the object search application can be displayed on the desktop of the terminal device, and the wake-up of the object search application can be realized by responding to the user's click operation on the icon. For example Figure 5 As shown, the example diagram of the wake-up method of the object search application provided by the embodiments of the present application is shown. The wake-up of the object search application can be realized by responding to the user's click operation on the icon 501. (2) One or more shortcut keys can also be set for the object search application, and the wake-up of the object search application can be realized by responding to the user's key operation on the shortcut key. For example Figure 5As shown, the shortcut keys may be a volume increase key 502 and a volume decrease key 503. The key operation of the shortcut keys may be pressing the volume increase key 502 and the volume decrease key 503 at the same time, or clicking the volume increase key 502 twice in a row within 1 second, etc. (3) The object search application may also be set to run continuously in the background in response to the user's operation, and the object search application may be awakened by responding to the user's preset voice command. The preset voice command may be a voice command containing a preset keyword, and the preset keyword may guide the voice assistant to understand the user's intention. For example, the preset keyword may be a nickname of the object search application set by the user, such as "little housekeeper", etc., or it may be a preset keyword in the voice command corresponding to the object search operation, such as "help me find", "help me find", "provide for me", "please tell me", "please help me find", etc. The application interface of the awakened object search application may be displayed on the display screen of the terminal device.

[0140] In some embodiments of the present application, a user's object search operation may include an operation for searching for object information, where the object corresponding to the user's object search operation may be an object within the user's current scene. For example, if user A is currently in bookstore SD1, the book "book1" corresponding to user A's object search operation may be an object within the current bookstore SD1. The object corresponding to the user's object search operation may also not be an object within the user's current scene. For example, if user B is currently in company GS1, the water cup corresponding to user B's object search operation may be an object within user B's residence ZS1.

[0141] In some embodiments of the present application, a terminal device (e.g., an object search application) may provide a user with multiple input methods, including but not limited to a combination of one or more of the following methods: voice input, text input, specific gesture input, handwriting input, brain-computer interface input, eye tracking input, key input, etc. The user may perform an object search operation in a variety of ways, and the terminal device may respond to the user's object search operation in a variety of ways. By receiving interaction data (or input data, such as voice data) between the user and the terminal device during the object search operation, the terminal device may identify and determine the target object corresponding to the object search operation.

[0142] In one example, the terminal device can respond to the object search operation performed by the user through a voice command, receive and recognize the voice command, and obtain a recognition result, wherein the voice command contains preset keywords (or keywords, key sentences, etc.); and determine the target object through the recognition result. In one example, the terminal device can collect the user's voice data. If the preset keyword is recognized in the collected voice data, it can be determined that the voice data is a voice command. After recognizing the preset keyword in the voice command, the terminal device can recognize the voice data after the preset keyword in the voice command, obtain the corresponding recognition result, and determine the target object through the recognition result. For example, the preset keyword can include but is not limited to a combination of one or more of the following keywords: "help me find", "help me find", "provide for me", "please tell me", "please help me find", "location", "where", etc.

[0143] In one example, a terminal device may receive and recognize data from a user interface interaction performed based on a preset user interface to perform an object search operation, thereby obtaining a recognition result. For example, the preset user interface may be a graphical user interface of an object search application, and the user interface interaction may include, but is not limited to, a combination of one or more of the following interaction methods: text input, specific gesture input, handwriting input, voice input, and inputting options by clicking a preset control to enter a menu.

[0144] In some embodiments of the present application, a terminal device may determine a target object based on the recognition result corresponding to the user's object search operation. The recognition result may include, but is not limited to, the object name and object attribute information. Object attribute information includes, but is not limited to, one or more of the following: object function, object appearance, object location, and object identification. For example, please refer to the description in S203.

[0145] In some embodiments of the present application, the object name in the recognition result can be determined and the target object can be determined based on the object name; or the object attribute information in the recognition result can be determined and the target object can be determined based on the object attribute information. In another example, the target object can be determined by combining the object name in the recognition result with the object attribute information.

[0146] In one example, if the user's interaction data with the terminal device during an object search operation is voice data, the voice data can be converted into text data using a voice recognition method, and the recognition result can be determined based on the text data. For example, the text data determined based on the user's voice data can be "Please help me find a toy," "Please help me find my glasses," "Where is the watering can?", etc.

[0147] In one example, if the recognition result is "Please help me find a toy," it can be determined that the object name is "toy" and the object attribute is "for entertainment, games, playing, etc.", so the target object can be determined to be "a toy for entertainment, games, playing, etc.". In another example, if the recognition result is "Where is the kettle for watering flowers," it can be determined that the object name is "kettle" and the object attribute information is "for watering flowers," so the target object can be determined to be "a kettle for watering flowers."

[0148] In some embodiments of the present application, a user can open an object search application and perform an object search operation, which can be implemented through a series of menus provided by the object search application. For example, the user clicks on the search control in the object search application and enters the information of the target object to be found through text input or voice input. In addition, the object search application can also directly provide menus corresponding to multiple scenes for the user to view and click. The user can click on the menu corresponding to the "residence" scene, and then further enter the menu of the "bedroom" scene. The object search application can directly display a directory of all objects in the "bedroom" scene. The directory may include the name of the object. If the user clicks on the name of one of the objects, the object interface can be entered, presenting the scene image of the clicked object, or the location information corresponding to the clicked object, etc.

[0149] Through the above embodiments, the interaction data corresponding to the user's object search operation can be obtained in various ways, and the target object corresponding to the user's object search operation can be quickly and accurately determined based on the analysis and identification of the interaction data.

[0150] In some embodiments of the present application, during actual application, it is also possible to determine whether the user performing the object search operation has access rights before responding to the user's object search operation. Access rights may include access rights to the scene information database or access rights to the object search application. For example, if the user has access rights, the user's object search operation is responded to; if the user does not have access rights, the user is prompted to perform a permission verification operation. After the user passes the permission verification, the user's object search operation is responded to.

[0151] In some embodiments of the present application, if user B uses terminal device B1 to perform an object search operation in user A's residence ZS1, the object search application installed in terminal device B1 can be used to determine whether user B has access rights. For example, user A is a user with permission management qualifications. User A can pre-set users with access rights through the object search application installed in terminal device A1, and terminal device A1 can send information about users with access rights (such as device information of terminal device B2, mobile phone number of user B, etc.) to the server. If user B performs an object search operation through the object search application of terminal device B1, the object search application of terminal device B1 can send user B's information to the server. The server can compare user B's information with the information of users with access rights to obtain a comparison result, which can indicate whether user B has access rights. The server can send the comparison result to the object search application of terminal device B1, so that the object search application determines whether user B has access rights.

[0152] In some embodiments of the present application, if a user does not have access rights, the user may be prompted to perform a permission verification operation. For example, the object search application may prompt the user to perform a permission verification operation using voice, text, images, etc. The permission verification operation may include sending a request message to a user with permission management qualifications or performing a security verification. After the user passes the permission verification, the user's object search operation may be responded to.

[0153] Through the above embodiment, an access rights control mechanism can be introduced. Before responding to a user's object search operation, the user's permission to access a specific database or object search application is verified. If the user does not have permission, they are prompted to perform permission verification operations, such as sending a request to an administrator or completing security verification. This ensures the legitimacy and security of information access, thereby improving the security of information management and the standardization of user operations.

[0154] S402: Determine multi-level information of the target object based on a preset scene information database.

[0155] In some embodiments of the present application, a preset scene information database may include a scene information database constructed using the scene information construction method provided in an embodiment of the present application. The scene information database may include multi-level information of multiple objects. The multi-level information may be constructed based on a scene image, object information of objects in the scene, and positional relationships between multiple objects. The multi-level information of an object may include, but is not limited to, one or more of the following combinations: an identifier of the sub-scene corresponding to the object, object information of the object, and positional relationships between the object and other objects.

[0156] In some embodiments of the present application, a matching object matching the target object can be determined from the scene information database, and multi-level information of the matching object can be obtained. The matching object can include one or more objects corresponding to the target object, and the multi-level information of the matching object can include, but is not limited to, one or more combinations of the following: an identifier of a sub-scene corresponding to the matching object, object information of the matching object, and a positional relationship between the matching object and other objects.

[0157] In an example, if the target object is a "toy for entertainment, game, playing, etc.", a matching object matching the target object can be retrieved from the scene information database, for example, the matching object can be a "jigsaw puzzle" or a "game console". For example, the multi-level information of the matching object "game console" can be that the game console is located on a bedside table in the bedroom WS1 of the residence ZS1, and there are also a tablet computer, a table lamp and other articles around.

[0158] In some embodiments of the present application, if it is determined that there is no matching object matching the target object in the scene information database, one or more combinations of the following operations can be performed: prompting the user to re-execute the object finding operation; issuing an abnormality warning; determining an object having an association relationship with the target object as a matching object matching the target object, and obtaining multi-level information of the matching object.

[0159] In an example, the prompt information can be displayed in the user interface provided by the object finding application to prompt the user to re-execute the object finding operation. The object finding application can also prompt the user to re-execute the object finding operation through voice output, and can also prompt the user to change the words when re-executing the object finding operation.

[0160] In another example, the user can be prompted that the target object does not exist in the scene information database, so as to remind the user of the finding exception. For example, the target object corresponding to the object finding operation performed by the user in the retail store LSD1 is the article WP1 of the brand PP1, and the retail store LSD1 does not have the article WP1 of the brand PP1 in the corresponding scene information database. The user can be prompted that the article WP1 of the brand PP1 does not exist in the retail store LSD1.

[0161] In yet another example, based on the above example, the article WP2 of the brand PP1 having an association relationship with the article WP1 of the brand PP1 can be determined as a matching object matching the target object, and multi-level information of the article WP2 of the brand PP1 can be obtained.

[0162] Through the above-described embodiments, multi-level information related to a target object can be efficiently retrieved and matched during the target object query process, enabling users to quickly obtain the required information in different scenarios. When direct matching fails, multiple alternative solutions are provided, such as prompting the user to retry the operation, issuing an exception warning, or recommending alternative objects based on related relationships. These measures not only improve the user experience but also enhance the system's fault tolerance and practicality.

[0163] S403: Outputting the location information of the target object based on the multi-level information of the target object.

[0164] In some embodiments of the present application, the primary purpose of a user's object search operation may be to determine the location of a target object. This location information can be output to the user by outputting multi-level information about the target object, facilitating the user's quick locating of the target object. The target object location information can be output in one or more of the following ways: outputting the multi-level information using voice; and / or displaying the multi-level information in a pre-set graphical user interface.

[0165] In one example, for example Figure 6 As shown, it is an example diagram of displaying multi-level information provided by an embodiment of the present application. Among them, multiple forms can be used to display multi-level information in a preset graphical user interface, and multi-level information of corresponding forms can also be displayed to the user in response to the user's selection of the display form. For example, the multiple forms include but are not limited to a combination of one or more of the following forms: text, image, animation. Among them, the multi-level information in the form of an image can be a multi-level semantic scene graph, or a scene image, or a scene image converted into a line graph, and the object position is displayed in a highlighted manner within the scene image or line graph. The multi-level information in the form of an animation can be an animation that uses direction indicators such as arrows in the above-mentioned line graph to indicate the route from entering the scene corresponding to the scene image to reaching the location of the target object.

[0166] Through the above embodiments, a variety of output methods (such as voice and graphical user interface display) can be used to output multi-level information of the target object, and flexible display forms (including text, images, animations, etc.) can be used to enhance the readability and accessibility of the multi-level information. At the same time, users are supported to personalize the display form, thereby improving the user experience.

[0167] The object search method provided in the embodiment of the present application can determine the target object that the user wants to find by responding to the user's object search operation; by determining the multi-level information of the target object in a preset scene information database, the location information of the target object is output to the user based on the multi-level information of the target object, thereby quickly determining the object required by the user and providing the user with detailed information of the object, helping the user to quickly and accurately find the required object.

[0168] See Figure 7 As shown, it is a flow chart of the scene information construction method provided by another embodiment of the present application. In the scene information acquisition stage, a multi-layer semantic scene graph can be initialized, and then an RGB image sequence of the scene (such as a scene graph) is collected, and a location recognition model is used to identify the location corresponding to the current scene graph (such as the room name is bedroom) based on the identification items in the image (such as a bed in a bedroom). Thereafter, a target detection model can be used to obtain a detection box (such as a bounding box) for each object (or object of the target type) in the current scene graph, and a description of the object attributes in each detection box and a description of the relationship with the surrounding objects can be obtained based on the visual language model. Based on the object description and location information, the objects in the current scene graph are associated and the object information is fused, and the scene graph is updated to execute the above process for the next scene graph.

[0169] Through the above embodiments, the place name corresponding to the scene graph can be determined based on the place recognition model, and the detection frame of the object in the current scene graph can be obtained using the target detection model. Based on the detection frame, the attribute description information and position relationship information of the corresponding object can be obtained using the visual language model, thereby realizing the acquisition of a multi-level semantic scene graph (e.g., multi-level information) for the objects in the scene graph. In particular, the user does not need to actively register the appearance information of the object for item search. Instead, the object information can be automatically extracted and memorized based on the RGB image using pre-trained target detection, place recognition, and image description models, which can reduce the difficulty of user use.

[0170] See Figure 8 As shown, it is a flowchart of a scene information construction method provided by another embodiment of the present application. A timestamp can be set for a scene image with a time sequence (for example, the timestamp of the previous scene image of the scene image with a timestamp of t is t-1), and a place recognition model is used to identify the place name of the scene image with a timestamp of t, and the place name is used as information of information level 1. The target detection model can be used to obtain the detection frame of the object in the scene image with a timestamp of t, and the item category and identification of the object are obtained as information of information level 2. The scene image with a timestamp of t having a detection frame can be input into a visual language model, and the prompt word can be input into the visual language model. The item attributes of the item and the description of the relationship with the surrounding items are obtained through the visual language model as information of information level 3. The information of the above multiple information levels can be associated and fused, and the information corresponding to the scene image with a timestamp of t-1 can also be fused in this process to obtain a multi-level semantic scene graph.

[0171] Through the above-described embodiments, a multi-level semantic scene graph can be constructed to capture scene information from multiple dimensions, including spatial and temporal dimensions. Using location recognition and visual language models to extract the location information of each item and its relationship to surrounding items helps users recall the location of items. Scene memory information, including the target item, its relationship to surrounding items, and its location, can be stored in text format, for example, without the need to store images of all items, reducing storage overhead and increasing query speed.

[0172] See Figure 9 The figure shows a flow chart of an object search method provided by another embodiment of the present application. The method can read the user's voice input data, parse the voice recognition data to obtain the name or attribute of the object to be searched, and search for the target object corresponding to the object to be searched by reading the scene memory (e.g., data obtained in the scene information acquisition phase). The multi-level information of the target object in the scene memory is displayed to the user using voice or a display screen (e.g., a graphical interactive interface).

[0173] The above embodiment eliminates the need for additional image capture and allows users to search for the name of the item being searched by simply searching the stored item information. This reduces user complexity and increases query speed. Providing information about the item being searched and its surroundings in user-friendly language helps users quickly recall and find the item.

[0174] The present application also provides a terminal device 100, see Figure 10 As shown, the terminal device 100 can be a mobile phone, a tablet computer, a desktop computer, a laptop computer, a handheld computer, a notebook computer, an ultra-mobile personal computer (UMPC), a netbook, a cellular phone, a personal digital assistant (PDA), an augmented reality (AR) device, a virtual reality (VR) device, an artificial intelligence (AI) device, a wearable device, an in-vehicle device, a smart home device and / or a smart city device. The embodiments of the present application do not impose any special restrictions on the specific type of the terminal device 100.

[0175] The terminal device 100 may include a processor 110, an external memory interface 120, an internal memory 121, a Universal Serial Bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, an earphone interface 170D, a sensor module 180, a button 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, etc. The sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, an air pressure sensor 180C, a magnetic sensor 180D, an acceleration sensor 180E, a distance sensor 180F, a proximity light sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, etc.

[0176] It should be understood that the structure illustrated in the embodiments of the present invention does not constitute a specific limitation on the terminal device 100. In other embodiments of the present application, the terminal device 100 may include more or fewer components than shown, or may combine or separate certain components, or arrange the components differently. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0177] The processor 110 may include one or more processing units. For example, the processor 110 may include an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural network processing unit (NPU). Different processing units may be independent devices or integrated into one or more processors.

[0178] The controller can generate operation control signals according to the instruction operation code and timing signal to complete the control of instruction fetching and execution.

[0179] Processor 110 may also include a memory for storing instructions and data. In one embodiment of the present application, the memory in processor 110 is a cache memory. The memory can store instructions or data that have just been used or are being recycled by processor 110. If processor 110 needs to use an instruction or data again, it can directly call it from the memory. This avoids repeated accesses, reduces processor 110 latency, and thus improves system efficiency.

[0180] In one embodiment of the present application, the processor 110 may include one or more interfaces. The interfaces may include an Inter-integrated Circuit (I2C) interface, an Inter-integrated Circuit Sound (I2S) interface, a Pulse Code Modulation (PCM) interface, a Universal Asynchronous Receiver / Transmitter (UART) interface, a Mobile Industry Processor Interface (MIPI), a General-Purpose Input / Output (GPIO) interface, a Subscriber Identity Module (SIM) interface, and / or a Universal Serial Bus (USB) interface.

[0181] The I2C interface is a bidirectional synchronous serial bus that includes a serial data line (SDA) and a serial clock line (SCL). In one embodiment of the present application, the processor 110 may include multiple I2C buses. The processor 110 may be coupled to the touch sensor 180K, charger, flash, camera 193, etc. through different I2C bus interfaces. For example, the processor 110 may be coupled to the touch sensor 180K through the I2C interface, so that the processor 110 and the touch sensor 180K communicate through the I2C bus interface, thereby realizing the touch function of the terminal device 100.

[0182] The I2S interface can be used for audio communication. In one embodiment of the present application, the processor 110 can include multiple I2S buses. The processor 110 can be coupled to the audio module 170 via the I2S bus to enable communication between the processor 110 and the audio module 170. In one embodiment of the present application, the audio module 170 can transmit audio signals to the wireless communication module 160 via the I2S interface, enabling the function of answering calls through a Bluetooth headset.

[0183] The PCM interface can also be used for audio communication, sampling, quantizing and encoding analog signals. In one embodiment of the present application, the audio module 170 and the wireless communication module 160 can be coupled via a PCM bus interface. In one embodiment of the present application, the audio module 170 can also transmit audio signals to the wireless communication module 160 via the PCM interface, thereby realizing the function of answering calls via a Bluetooth headset. Both the I2S interface and the PCM interface can be used for audio communication.

[0184] The UART interface is a universal serial data bus used for asynchronous communication. The bus can be a bidirectional communication bus. It converts the data to be transmitted between serial communication and parallel communication. In one embodiment of the present application, the UART interface is generally used to connect the processor 110 and the wireless communication module 160. For example, the processor 110 communicates with the Bluetooth module in the wireless communication module 160 via the UART interface to implement Bluetooth functionality. In one embodiment of the present application, the audio module 170 can transmit audio signals to the wireless communication module 160 via the UART interface, enabling the function of playing music through a Bluetooth headset.

[0185] The MIPI interface can be used to connect the processor 110 to peripheral devices such as the display screen 194 and the camera 193. MIPI interfaces include the Camera Serial Interface (CSI) and the Display Serial Interface (DSI). In one embodiment of the present application, the processor 110 and the camera 193 communicate via the CSI interface to implement the camera function of the terminal device 100. The processor 110 and the display screen 194 communicate via the DSI interface to implement the display function of the terminal device 100.

[0186] The GPIO interface can be configured through software. The GPIO interface can be configured as a control signal or a data signal. In one embodiment of the present application, the GPIO interface can be used to connect the processor 110 to the camera 193, the display 194, the wireless communication module 160, the audio module 170, the sensor module 180, etc. The GPIO interface can also be configured as an I2C interface, an I2S interface, a UART interface, a MIPI interface, etc.

[0187] The USB interface 130 is an interface that complies with USB standards and may be a Mini USB interface, a MicroUSB interface, a USB Type-C interface, or the like. The USB interface 130 can be used to connect a charger to charge the terminal device 100, or to transfer data between the terminal device 100 and peripheral devices. It can also be used to connect headphones to play audio. The interface can also be used to connect to other terminal devices 100, such as AR devices.

[0188] It is understood that the interface connection relationship between the modules illustrated in the embodiment of the present invention is merely an illustrative illustration and does not constitute a structural limitation on the terminal device 100. In other embodiments of the present application, the terminal device 100 may also adopt a different interface connection method from the above embodiment, or a combination of multiple interface connection methods.

[0189] The charging management module 140 is configured to receive charging input from a charger. The charger can be either a wireless charger or a wired charger. In some wired charging embodiments, the charging management module 140 can receive charging input from the wired charger via the USB interface 130. In some wireless charging embodiments, the charging management module 140 can receive wireless charging input via the wireless charging coil of the terminal device 100. While charging the battery 142, the charging management module 140 can also provide power to the terminal device 100 via the power management module 141.

[0190] The power management module 141 is used to connect the battery 142, the charging management module 140, and the processor 110. The power management module 141 receives input from the battery 142 and / or the charging management module 140, and provides power to the processor 110, the internal memory 121, the display 194, the camera 193, and the wireless communication module 160. The power management module 141 can also be used to monitor parameters such as battery capacity, battery cycle count, and battery health status (leakage, impedance). In some other embodiments, the power management module 141 can also be set in the processor 110. In other embodiments, the power management module 141 and the charging management module 140 can also be set in the same device.

[0191] The wireless communication function of the terminal device 100 can be implemented through the antenna 1, the antenna 2, the mobile communication module 150, the wireless communication module 160, the modem processor and the baseband processor.

[0192] Antenna 1 and Antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in terminal device 100 can be used to cover a single or multiple communication frequency bands. Different antennas can also be reused to improve antenna utilization. For example, antenna 1 can be reused as a diversity antenna for a wireless local area network. In other embodiments, the antennas can be used in conjunction with a tuning switch.

[0193] The mobile communication module 150 can provide solutions for wireless communications including 2G / 3G / 4G / 5G applied to the terminal device 100. The mobile communication module 150 may include at least one filter, a switch, a power amplifier, a low noise amplifier (LNA), etc. The mobile communication module 150 can receive electromagnetic waves from the antenna 1, and filter, amplify, and process the received electromagnetic waves, and transmit them to the modulation and demodulation processor for demodulation. The mobile communication module 150 can also amplify the signal modulated by the modulation and demodulation processor, and convert it into electromagnetic waves for radiation through the antenna 1. In one embodiment of the present application, at least some of the functional modules of the mobile communication module 150 can be set in the processor 110. In one embodiment of the present application, at least some of the functional modules of the mobile communication module 150 can be set in the same device as at least some of the modules of the processor 110.

[0194] The modem processor may include a modulator and a demodulator. The modulator is used to modulate the low-frequency baseband signal to be transmitted into a medium-high frequency signal. The demodulator is used to demodulate the received electromagnetic wave signal into a low-frequency baseband signal. The demodulator then transmits the demodulated low-frequency baseband signal to the baseband processor for processing. After being processed by the baseband processor, the low-frequency baseband signal is passed to the application processor. The application processor outputs a sound signal through an audio device (not limited to the speaker 170A, the receiver 170B, etc.) or displays an image or video through the display screen 194. In one embodiment of the present application, the modem processor may be an independent device. In other embodiments, the modem processor may be independent of the processor 110 and be set in the same device as the mobile communication module 150 or other functional modules.

[0195] The wireless communication module 160 can provide a solution for wireless communication including Wireless Local Area Networks (WLAN) (e.g., Wireless Fidelity (Wi-Fi) network), Bluetooth (BT), Global Navigation Satellite System (GNSS), Frequency Modulation (FM), Near Field Communication (NFC), Infrared (IR) technology, etc. applied to the terminal device 100. The wireless communication module 160 can be one or more devices that integrate at least one communication processing module. The wireless communication module 160 receives electromagnetic waves via the antenna 2, frequency-modulates and filters the electromagnetic wave signals, and transmits the processed signals to the processor 110. The wireless communication module 160 can also receive signals to be transmitted from the processor 110, frequency-modulate them, amplify them, and radiate them as electromagnetic waves via the antenna 2.

[0196] In one embodiment of the present application, the antenna 1 of the terminal device 100 is coupled to the mobile communication module 150, and the antenna 2 is coupled to the wireless communication module 160, so that the terminal device 100 can communicate with the network and other devices through wireless communication technology. The wireless communication technology may include Global System for Mobile Communications (GSM), General Packet Radio Service (GPRS), Code Division Multiple Access (CDMA), Wideband Code Division Multiple Access (WCDMA), Time-Division Code Division Multiple Access (TD-SCDMA), Long Term Evolution (LTE), BT, GNSS, WLAN, NFC, FM, and / or IR technology. The GNSS may include a Global Positioning System (GPS), a Global Navigation Satellite System (GLONASS), a Beidou Navigation Satellite System (BDS), a Quasi-Zenith Satellite System (QZSS) and / or a Satellite Based Augmentation System (SBAS).

[0197] The terminal device 100 implements display functions through a GPU, display screen 194, and an application processor. The GPU is a microprocessor for image processing that connects the display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 110 may include one or more GPUs that execute program instructions to generate or modify display information.

[0198] The display screen 194 is configured to display images, videos, and the like. The display screen 194 includes a display panel. The display panel can be a Liquid Crystal Display (LCD), an Organic Light-Emitting Diode (OLED), an Active-Matrix Organic Light Emitting Diode (AMOLED), a Flex Light-Emitting Diode (FLED), a Miniled, a Microled, a Micro-OLED, a Quantum Dot Light Emitting Diodes (QLED), or the like. In an embodiment of the present application, the terminal device 100 can include one or N display screens 194, where N is a positive integer greater than 1.

[0199] The terminal device 100 can implement the photographing function through the ISP, the camera 193, the video codec, the GPU, the display screen 194, and the application processor.

[0200] The ISP is configured to process the data fed back by the camera 193. For example, when taking a photo, the shutter is opened, the light is transmitted to the camera photosensitive element through the lens, the light signal is converted into an electrical signal, and the camera photosensitive element transmits the electrical signal to the ISP for processing to convert it into an image visible to the naked eye. The ISP can also optimize the algorithm for the noise, brightness, and skin color of the image. The ISP can also optimize the exposure, color temperature, and other parameters of the shooting scene. In an embodiment of the present application, the ISP can be arranged in the camera 193.

[0201] The camera 193 is configured to capture still images or videos. The object generates an optical image through the lens and projects it onto the photosensitive element. The photosensitive element can be a Charge Coupled Device (CCD) or a Complementary Metal-Oxide-Semiconductor (CMOS) phototransistor. The photosensitive element converts the light signal into an electrical signal, and then transmits the electrical signal to the ISP to convert it into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. The DSP converts the digital image signal into an image signal in a standard RGB, YUV, or the like format. In an embodiment of the present application, the terminal device 100 can include one or N cameras 193, where N is a positive integer greater than 1.

[0202] The digital signal processor is used to process digital signals. In addition to processing digital image signals, it can also process other digital signals. For example, when the terminal device 100 selects a frequency point, the digital signal processor is used to perform Fourier transform on the frequency point energy.

[0203] Video codecs are used to compress or decompress digital video. Terminal device 100 may support one or more video codecs. This allows terminal device 100 to play or record videos in various encoding formats, such as Moving Picture Experts Group (MPEG) 1, MPEG2, MPEG3, and MPEG4.

[0204] The NPU is a neural network (NN) computing processor. Drawing on the structure of biological neural networks, such as the transmission patterns between neurons in the human brain, it rapidly processes input information and can continuously self-learn. The NPU enables intelligent cognitive applications in the terminal device 100, such as image recognition, face recognition, information prompts, voice recognition, and text comprehension.

[0205] The internal memory 121 may include one or more random access memories (RAM) and one or more non-volatile memories (NVM).

[0206] Random access memory may include static random-access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM, for example, the fifth generation DDRSDRAM is generally referred to as DDR5SDRAM), etc.

[0207] Non-volatile memory may include disk storage devices and flash memory.

[0208] The flash memory can include NOR FLASH, NAND FLASH, 3D NAND FLASH, etc. according to the operation principle, can include Single-Level Cell (SLC), Multi-Level Cell (MLC), Triple-Level Cell (TLC), Quad-Level Cell (QLC), etc. according to the storage unit potential order, and can include Universal Flash Storage (UFS), embedded Multi Media Card (eMMC), etc. according to the storage specification.

[0209] The random access memory can be directly read and written by the processor 110, can be used to store executable programs (such as machine instructions) of an operating system or other programs running, and can also be used to store data of users and application programs, etc.

[0210] The non-volatile memory can also store executable programs and store data of users and application programs, etc., and can be loaded in advance into the random access memory for direct reading and writing by the processor 110.

[0211] The external memory interface 120 can be used to connect an external non-volatile memory, to realize the storage capacity expansion of the terminal device 100. The external non-volatile memory communicates with the processor 110 through the external memory interface 120, to realize the data storage function. For example, files such as music and video are saved in the external non-volatile memory.

[0212] The internal memory 121 or the external memory interface 120 is used to store one or more computer programs. The one or more computer programs are configured to be executed by the processor 110. The one or more computer programs include a plurality of instructions, which when executed by the processor 110, can implement the screen display detection method executed on the terminal device 100 in the above-mentioned embodiments, to realize the screen display detection function of the terminal device 100.

[0213] The terminal device 100 can realize the audio function through the audio module 170, the speaker 170A, the receiver 170B, the microphone 170C, the earphone interface 170D, and the application processor, etc. For example, music playing, recording, etc.

[0214] The audio module 170 is used to convert digital audio information into analog audio signal output, and is also used to convert analog audio input into digital audio signals. The audio module 170 can also be used to encode and decode audio signals. In one embodiment of the present application, the audio module 170 can be provided in the processor 110, or some functional modules of the audio module 170 can be provided in the processor 110.

[0215] The speaker 170A, also called a "speaker", is used to convert audio electrical signals into sound signals. The terminal device 100 can listen to music or listen to hands-free calls through the speaker 170A.

[0216] The receiver 170B, also called a "handset", is used to convert audio electrical signals into sound signals. When the terminal device 100 receives a call or voice message, the user can hear the voice by placing the receiver 170B close to the ear.

[0217] Microphone 170C, also known as "microphone" or "microphone", is used to convert sound signals into electrical signals. When making a call or sending a voice message, the user can speak by putting their mouth close to the microphone 170C to input the sound signal into the microphone 170C. The terminal device 100 can be provided with at least one microphone 170C. In other embodiments, the terminal device 100 can be provided with two microphones 170C, which can not only collect sound signals but also realize noise reduction function. In other embodiments, the terminal device 100 can also be provided with three, four or more microphones 170C to realize sound signal collection, noise reduction, and can also identify the source of sound, realize directional recording function, etc.

[0218] The headphone jack 170D is used to connect a wired headphone and can be a USB interface 130 or a 3.5mm Open Mobile Terminal Platform (OMTP) standard interface or a Cellular Telecommunications Industry Association of the USA (CTIA) standard interface.

[0219] The buttons 190 include a power button, a volume button, etc. The buttons 190 may be mechanical buttons or touch buttons. The terminal device 100 may receive key inputs and generate key signal inputs related to user settings and function control of the terminal device 100.

[0220] Motor 191 can generate vibration prompts. Motor 191 can be used for incoming call vibration prompts, and can also be used for touch vibration feedback. For example, touch operations acting on different applications (such as taking pictures, audio playback, etc.) can correspond to different vibration feedback effects. For touch operations acting on different areas of the display screen 194, motor 191 can also correspond to different vibration feedback effects. Different application scenarios (for example: time reminders, receiving messages, alarm clocks, games, etc.) can also correspond to different vibration feedback effects. The touch vibration feedback effect can also support customization.

[0221] The indicator 192 may be an indicator light, which may be used to indicate the charging status, power level changes, messages, missed calls, notifications, etc.

[0222] The SIM card interface 195 is used to connect a SIM card. The SIM card can be connected to and disconnected from the terminal device 100 by inserting or removing it from the SIM card interface 195. The terminal device 100 can support one or N SIM card interfaces, where N is a positive integer greater than one. The SIM card interface 195 can support Nano SIM cards, Micro SIM cards, SIM cards, and the like. Multiple cards can be inserted into the same SIM card interface 195 at the same time. The multiple cards can be of the same or different types. The SIM card interface 195 can also be compatible with different types of SIM cards. The SIM card interface 195 can also be compatible with external memory cards. The terminal device 100 interacts with the network through the SIM card to implement functions such as calls and data communications. In one embodiment of the present application, the terminal device 100 uses an eSIM, i.e., an embedded SIM card. The eSIM card can be embedded in the terminal device 100 and cannot be separated from the terminal device 100. An embodiment of the present application also provides a computer storage medium, in which computer instructions are stored. When the computer instructions are executed on the terminal device 100, the terminal device 100 executes the above-mentioned related method steps to implement the method in the above-mentioned embodiment.

[0223] The embodiments of the present application also provide a computer program product. When the computer program product is run on a computer, the computer is caused to execute the above-mentioned related steps to implement the method in the above-mentioned embodiment.

[0224] In addition, an embodiment of the present application also provides a device, which can specifically be a chip, component or module, and the device may include a connected processor and memory; wherein the memory is used to store computer-executable instructions, and when the device is running, the processor can execute the computer-executable instructions stored in the memory to enable the chip to execute the methods in the above-mentioned method embodiments.

[0225] Among them, the terminal device, computer storage medium, computer program product or chip provided in this embodiment are all used to execute the corresponding methods provided above. Therefore, the beneficial effects that can be achieved can refer to the beneficial effects in the corresponding methods provided above, and will not be repeated here.

[0226] Through the description of the above implementation methods, technical personnel in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional modules is used as an example. In actual applications, the above-mentioned functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0227] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of modules or units is only a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0228] Units described as separate components may or may not be physically separate, and components shown as units may be one physical unit or multiple physical units, that is, they may be located in one place or distributed in multiple places. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0229] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0230] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on this understanding, the technical solution of the embodiment of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for enabling a device (which can be a single-chip microcomputer, chip, etc.) or a processor to execute all or part of the steps of the various embodiments of the present application. The aforementioned storage medium includes various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0231] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application and are not intended to limit the present application. Although the present application has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present application may be modified or replaced by equivalents without departing from the spirit and scope of the technical solutions of the present application.

Claims

1. An object search method, applied to a terminal device, characterized in that: The method comprises: In response to a user's object search operation, determining a target object corresponding to the object search operation; Determining, based on a preset scene information database, multi-level information of the target object, wherein the multi-level information is constructed based on a scene image, object information of objects in the scene, and positional relationships between multiple objects; Based on the multi-level information of the target object, position information of the target object is output.

2. The object search method according to claim 1, characterized in that: In response to the user's object search operation, determining the target object corresponding to the object search operation includes: In response to the object search operation performed by the user through a voice instruction, receiving and recognizing the voice instruction to obtain a recognition result, wherein the language instruction includes a preset keyword; and / or, In response to an object search operation performed by the user through an interface interaction based on a preset user interface, receiving and identifying data of the interface interaction to obtain an identification result; The target object is determined according to the recognition result.

3. The object search method according to claim 2, characterized in that: Determining the target object through the recognition result includes: Determine the object name in the recognition result, and determine the target object according to the object name; or, Determine object attribute information in the recognition result, and determine the target object according to the object attribute information.

4. The object search method according to claim 3, characterized in that: The object attribute information includes one or more of object function, object appearance, object location, and object identification.

5. The object search method according to claim 1, wherein: The determining of the multi-level information of the target object based on a preset scene information database includes: A matching object that matches the target object is determined from the scene information database, and multi-level information of the matching object is obtained. The object search method according to claim 1, wherein: The method further includes: outputting the location information of the target object in one or more of the following output modes: Outputting the multi-level information using speech; and / or The multi-level information is displayed on a preset graphical user interface.

7. The object search method according to claim 1, characterized in that: The method further comprises: If the user has access rights, respond to the user's object search operation; If the user does not have access rights, the user is prompted to perform a rights verification operation. After the user passes the rights verification, the user's object search operation is responded to.

8. A scene information construction method, applied to a terminal device, characterized in that: The method comprises: Acquire a scene image of a main scene, where the main scene includes multiple sub-scenes; identifying a secondary scene based on the first type of object in the scene image, and establishing a correspondence between the scene image and an identifier of the secondary scene; Identifying a second type of object in the scene image, and acquiring object information of the second type of object; determining multi-level information of the second-category objects based on an identifier of a secondary scene corresponding to the scene image, object information of the second-category objects, and a positional relationship between the second-category objects; Based on the multi-level information, a scene information database is constructed.

9. The scene information construction method according to claim 8, characterized in that: The object information includes object names and object attribute information. The method further includes: determining the positional relationship between the second-category objects based on the object information of the second-category objects.

10. The scene information construction method according to claim 8, characterized in that: The method for obtaining multi-level information of the second type of object includes: Determining first-level information based on an identifier of a sub-scene corresponding to the scene image; determining second-level information based on the object information of the second-category object; determining third-level information based on the positional relationship between the second-category objects; Establishing associations among the first-level information, the second-level information, and the third-level information of the second-category objects to obtain a multi-level relationship of the second-category objects; The information corresponding to the multi-level relationship is used as the multi-level information of the second type of object.

11. The scene information construction method according to claim 8, characterized in that: The method further comprises: In response to the user's information entry operation, receiving multi-level information of a third-category object; wherein the multi-level information of the third-category object includes the multi-level information of the second-category object corresponding to the third-category object and the object information of the third-category object; The scene information database is constructed based on the multi-level information of the third category of objects.

12. The scene information construction method according to claim 9, characterized in that: The determining the positional relationship between the second-category objects based on the object information of the second-category objects includes: Determine a bounding box for each second-category object, and determine a nearest neighbor object for each second-category object based on the bounding box; Based on the positional relationship between the bounding box of each second-category object and the bounding box of the nearest neighbor object, the positional relationship between each second-category object and the corresponding nearest neighbor object is determined, and the positional relationship between the second-category objects is determined according to the positional relationship between each second-category object and the corresponding nearest neighbor object.

13. The scene information construction method according to claim 9, characterized in that: The determining the positional relationship between the second-category objects based on the object information of the second-category objects further includes: The scene image is input into a preset visual language model, and the positional relationship between the second-category objects is output using the visual language model.

14. The scene information construction method according to claim 8, characterized in that: The information storage format of the scene information database includes a text format.

15. The scene information construction method according to claim 8, characterized in that: The method further includes: updating the scene information database based on a preset update condition.

16. The scene information construction method according to claim 15, characterized in that: The updating of the scene information database based on the preset update condition includes any of the following updating methods: Updating the scene information database according to a preset update time node or update cycle; In response to a user's operation of updating the scene information database, the scene information database is updated.

17. A terminal device, characterized in that: The terminal device includes a memory and a processor: The memory is used to store program instructions; The processor is used to read and execute the program instructions stored in the memory. When the program instructions are executed by the processor, the terminal device executes the object search method as described in any one of claims 1 to 7 or the scene information construction method as described in any one of claims 8 to 16.

18. A computer storage medium, characterized in that The computer storage medium stores program instructions, which, when executed on a terminal device, enable a processor of the terminal device to execute the object search method according to any one of claims 1 to 7 or the scene information construction method according to any one of claims 8 to 16.

Images