Article searching method, computing device, Internet of Things system, equipment and storage medium
By registering the place and item information in the item query system and using sensors to collect images for item search, the problems of high cost of item search and power limit in the prior art are solved, and efficient and applicable item positioning and privacy protection are achieved.
Patent Information
- Application Number
- CN202411048762.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-31
- Publication Date
- 2024-11-26
AI Technical Summary
In the prior art, a tracker is required when an item is left behind, which is costly and limited by battery power, and cannot be used for the search of all items.
By registering the location and item information in the item query system, and using sensors to collect images for item search, it can achieve item positioning without additional tracker installation, saving costs and avoiding battery power limitations.
It realizes effective search of items, is suitable for any item, is easy to operate, has strong expansion, and ensures user privacy and security.
Smart Images

Figure CN119027926A_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to artificial intelligence technology and communication technology, and in particular to an object search method, a computing device, an Internet of Things system, a device and a storage medium. Background Art
[0002] In daily life, users often leave items in their daily places. For example, when a user is on a business trip or at work, he or she may leave items such as wallets and keys in a corner of the house; or when a user drives home from get off work, he or she may leave items such as headphones and computers in the car, etc.
[0003] In the related art, a tracker (tag) with a built-in Bluetooth chip, Global Navigation Satellite System (GNSS) or other communication unit can be installed on each item, and the item can be found by locating the tracker. In this implementation, a tracker needs to be installed for each item and the tracker needs to be firmly bound to the item, which is costly and not feasible for some items. In addition, it needs to rely on the battery power of the tracker, which limits the search for the item. Summary of the invention
[0004] In order to solve the above technical problems, the embodiments of the present disclosure provide an item search method, a computing device, an Internet of Things system, a device and a storage medium to improve the item search effect.
[0005] In one aspect of the present disclosure, there is provided an item search method, which is applied to an item query system, and the method comprises:
[0006] Determine first item information and first sensor information corresponding to the first event; wherein the first sensor information includes sensor information of a first sensor, and the first sensor includes a sensor corresponding to at least one registered place, and the registered place is a place registered in the item query system;
[0007] Acquire a first image captured by the first sensor;
[0008] A first item search is performed based on the first item information and the first image to obtain a first search result.
[0009] Another aspect of the present disclosure provides a computing device, which is applied to an item query system, and the computing device includes:
[0010] A first determination module, configured to determine first item information and first sensor information corresponding to a first event; wherein the first sensor information includes sensor information of a first sensor, and the first sensor includes at least one sensor corresponding to a registered place, and the registered place is a place registered in the item query system;
[0011] A first acquisition module, used to acquire a first image captured by the first sensor;
[0012] The item search module is used to perform a first item search based on the first item information and the first image to obtain a first search result.
[0013] Another aspect of the present disclosure provides an Internet of Things system, comprising at least one sensor and at least one end-side device; wherein the at least one sensor is deployed in at least one place corresponding to the Internet of Things system, a first end-side device among the at least one end-side device comprises a computing unit, a second end-side device among the at least one end-side device comprises an information input unit, a third end-side device among the at least one end-side device comprises an information output unit, and the first end-side device, the second end-side device, and the third end-side device respectively comprise one or more terminal devices; each of the sensors and each of the terminal devices respectively comprises a communication unit, the at least one sensor is communicatively connected to the first end-side device through the communication unit, and other terminal devices among the at least one end-side device except the first end-side device are communicatively connected to the first end-side device through the communication unit.
[0014] Another aspect of the embodiments of the present disclosure provides an electronic device, including:
[0015] processor;
[0016] a memory for storing instructions executable by the processor;
[0017] The processor is used to read the executable instructions from the memory and execute the instructions to implement the object search method described in any one of the above embodiments.
[0018] According to another aspect of the embodiments of the present disclosure, a computer-readable storage medium is provided, wherein the storage medium stores a computer program, and when the computer program is executed, the method for searching for items described in any of the above embodiments of the present disclosure is executed.
[0019] According to another aspect of the embodiments of the present disclosure, a computer program product is provided. When an instruction processor in the computer program product executes, the method for searching for an item described in any of the above embodiments is implemented.
[0020] Based on the embodiment of the present disclosure, a technical solution for searching for items based on visual technology is provided. The item query system determines the first item information and first sensor information corresponding to the first event. The first sensor information includes sensor information of the first sensor corresponding to at least one registered place, wherein the registered place is a place registered in the item query system. Then, the first image captured by the first sensor is obtained. Then, the first item search is performed based on the first item information and the first image to obtain a first search result, thereby achieving the search for the item. The technical solution for searching for items in the embodiment of the present disclosure does not require the installation of additional electronic tools such as trackers for the items, and the electronic tools are firmly bound to the items, saving the cost required for this. It can be applied to the search for any item, has strong scalability, and is easy to operate. Moreover, it is not limited by the battery power of the electronic tools, avoiding the limitation on the search for items, and can achieve effective search for items, thereby effectively improving the search effect for items.
[0021] The technical solution of the present invention is further described in detail below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS
[0022] Figure 1 It is a structural diagram of an exemplary embodiment of the present disclosure.
[0023] Figure 2 The diagram is an example of an embodiment of the present disclosure being configured as an item query system.
[0024] Figure 3 It is a flowchart of an exemplary embodiment of the object search method disclosed in the present invention.
[0025] Figure 4 It is an exemplary flow chart of determining first item information and first sensor information in another exemplary embodiment of the item search method disclosed in the present invention.
[0026] Figure 5 It is an exemplary flowchart of searching for a first item in another exemplary embodiment of the item search method disclosed in the present invention.
[0027] Figure 6 It is a schematic diagram of an exemplary process of performing place registration in an embodiment of the present disclosure.
[0028] Figure 7 It is a schematic diagram of an exemplary process of registering items in an embodiment of the present disclosure.
[0029] Figure 8 It is an application flow diagram of another exemplary embodiment of the object search method in the embodiment of the present disclosure.
[0030] Fig. 9It is a flowchart of another exemplary embodiment of the object search method disclosed in the present invention.
[0031] Fig.10 It is an application flow diagram of another exemplary embodiment of the object search method in the embodiment of the present disclosure.
[0032] Fig.11 It is an exemplary flowchart of determining first item information and first sensor information in another exemplary embodiment of the item search method disclosed in the present invention.
[0033] Fig.12 It is a schematic diagram of an exemplary flow chart of searching for a first item in another exemplary embodiment of the item search method disclosed in the present invention.
[0034] Fig.13 It is another exemplary flowchart of searching for a first item in yet another exemplary embodiment of the item search method disclosed herein.
[0035] Fig.14 It is another exemplary flowchart of searching for a first item in another exemplary embodiment of the item searching method disclosed in the present invention.
[0036] Fig.15 It is a schematic diagram of an application flow of another exemplary embodiment of the object search method in the embodiment of the present disclosure.
[0037] Fig.16 It is another exemplary flowchart of searching for a first item in another exemplary embodiment of the item searching method disclosed in the present invention.
[0038] Fig.17 It is another exemplary flowchart of searching for a first item in another exemplary embodiment of the item search method disclosed in the present invention.
[0039] Fig.18 It is an exemplary flow chart of determining first item information and first sensor information in another exemplary embodiment of the item search method disclosed in the present invention.
[0040] Fig.19 It is a structural diagram of an exemplary embodiment of a computing device disclosed in the present invention.
[0041] Fig. 20 It is a structural diagram of another exemplary embodiment of the computing device disclosed in the present invention.
[0042] Fig.21 It is a structural diagram of another exemplary embodiment of the computing device disclosed in the present invention.
[0043] Fig. 22 FIG. 4 is a schematic diagram of the structure of another exemplary embodiment of the computing device disclosed in the present invention.
[0044] Fig.23 It is a structural diagram of another exemplary embodiment of the computing device disclosed in the present invention.
[0045] Fig.24 is a structural diagram of an electronic device provided by an exemplary embodiment of the present disclosure. DETAILED DESCRIPTION
[0046] To explain the present disclosure, example embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present disclosure, rather than all of the embodiments. It should be understood that the present disclosure is not limited to the example embodiments.
[0047] It should be noted that the relative arrangement of components and steps, the numerical expressions and numerical values set forth in these embodiments do not limit the scope of the present disclosure unless specifically stated otherwise.
[0048] It should also be understood that in the embodiments of the present disclosure, “plurality” may refer to two or more than two, and “at least one” may refer to one, two, or more than two.
[0049] It should also be understood that any component, data or structure mentioned in the embodiments of the present disclosure can generally be understood as one or more, unless explicitly limited or otherwise indicated in the context.
[0050] In addition, the term "and / or" in the present disclosure is only a description of the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in the present disclosure generally indicates that the associated objects before and after are in an "or" relationship.
[0051] It should also be understood that the description of the various embodiments in the present disclosure focuses on the differences between the various embodiments, and the same or similar aspects thereof can be referenced to each other, and for the sake of brevity, they will not be described one by one.
[0052] Technologies, methods, and equipment known to ordinary technicians in the relevant art may not be discussed in detail, but where appropriate, the technologies, methods, and equipment should be considered as part of the specification.
[0053] SUMMARY OF THE DISCLOSURE
[0054] In daily life, users often leave items in their daily places. For example, in some scenarios, users may leave wallets, keys and other items in a corner of their home when they are on a business trip or at work; or in other scenarios, users may leave headphones, computers and other items in their vehicles when they drive home from get off work; and so on.
[0055] In a related technology, a tracker with a built-in Bluetooth chip, GNSS or other communication unit can be installed on each item, and the item can be found by locating the tracker. If the item is nearby, the tracker can be controlled to play a sound. In this implementation, since a tracker needs to be installed for each item, the tracker is expensive, resulting in high costs; and the tracker needs to be firmly bound to the item, which makes the installation complicated and not feasible for some items; in addition, it depends on the battery power of the tracker. When the battery of the tracker is exhausted, the location of the corresponding item cannot be found, thereby limiting the search for the item.
[0056] Exemplary Systems
[0057] Figure 1 Schematic diagram of the structure of the unified exemplary embodiment of the present disclosure. Figure 1 As shown, the Internet of Things system of the embodiment of the present disclosure includes at least one sensor 11 and at least one end-side device 12.
[0058] Among them, at least one sensor 11 is deployed in at least one place corresponding to the Internet of Things system. In specific applications, the sensors deployed in each place can be separately deployed sensors or sensors on existing devices in each place. For example, when the place corresponding to the Internet of Things system is a vehicle, a ship, a flight equipment (such as an airplane, an aircraft), a smart terminal and other types of manned mobile devices, the sensor deployed in the mobile device can be a camera of the driver monitoring system (Driver Monitoring System, DMS) in the mobile device, a camera of the passenger monitoring system (Occupancy Monitoring System, OMS), a camera of the in-cabin monitoring system (In-cabin Monitoring System, IMS), etc. The embodiment of the present disclosure does not limit the source and deployment method of the sensor.
[0059] The first end-side device 120 in at least one end-side device 12 includes a computing unit for providing computing power. In some optional examples, the computing unit may have a certain computing power and may provide processing of at least one type of information such as images, videos, audio, text, etc., which is not limited in the embodiments of the present disclosure. With the rapid development of artificial intelligence (AI) technology and the continuous improvement of AI chip computing power, mobile terminals and in-vehicle infotainment systems (i.e., in-vehicle infotainment, IVI) equipped with AI chips can provide powerful AI computing capabilities and be used as the first end-side device 120, becoming an AI computing center as an end-side AI device.
[0060] The second terminal device 121 in at least one terminal device 12 includes an information input unit for implementing an information input function. In some optional examples, the information input unit can implement various forms of information input functions by collecting images, videos, audios, receiving text input by users, etc., which is not limited in the embodiments of the present disclosure.
[0061] The third terminal device 122 in at least one terminal device 12 includes an information output unit for implementing an information output function. In some optional examples, the information output unit can output at least one type of information such as images, videos, audio, sounds, text, dynamic signals (such as vibration, etc.), and light signals.
[0062] The first terminal device 120, the second terminal device 121, and the third terminal device 122 may include one or more terminal devices respectively. Figure 1 The figure only exemplarily shows the implementation of the first end-side device 120, the second end-side device 121, and the third end-side device 122 including one terminal device. For the first end-side device 120, the second end-side device 121, and the third end-side device 122 including multiple terminal devices, reference can be made to the implementation.
[0063] Each sensor in the at least one sensor 11 and each terminal device in the at least one terminal device 12 respectively include a communication unit, and the at least one sensor 11 is communicatively connected to the first terminal device 120 through the communication unit, and the other terminal devices in the at least one terminal device 12 except the first terminal device 120 are communicatively connected to the first terminal device 120 through the communication unit. In some optional examples, the communication units in each sensor and each terminal device can communicate through at least one of various communication networks such as a mobile communication network (such as 5G, 4G, etc.) and a wireless local area network (Wireless Fidelity, WiFi).
[0064] Based on this embodiment, a vision-based Internet of Things system is provided. The Internet of Things system can be obtained by interconnecting sensors and terminal devices on the end side, including sensors deployed in corresponding places, and terminal devices with computing capabilities, information input functions and information output functions. The system can collect and compute scene data for each place within the scope of the Internet of Things system, circulate the data within the scope of the Internet of Things system, and interact with users. There is no need to install additional electronic tools such as trackers for items, and the system is not limited by the battery power of electronic tools. Application processing based on user needs can be achieved within the scope of the corresponding place, such as target query based on scene data, etc. The Internet of Things system can add more end-side devices (including sensors, terminal devices) or change the connected end-side devices as needed, and has strong scalability and easy operation. The Internet of Things system includes terminal devices with computing capabilities, which can be used as a computing center to make full use of idle end-side computing resources to provide the computing power required by the Internet of Things system, without the need to invest additional computing resources, thereby saving costs.
[0065] Optionally, in another exemplary embodiment of the Internet of Things system of the present disclosure, a fourth end-side device 123 in at least one end-side device 12 includes an information storage unit for implementing a storage function, and the fourth end-side device 123 may include one or more terminal devices.
[0066] In some possible implementations, the terminal devices included in the first terminal device 120, the second terminal device 121, the third terminal device 122, and the fourth terminal device 123 may be the same, different, or partially the same. For example, a mobile phone terminal, a personal computer (PC), an IVI, etc. that includes a computing unit, an input unit, an output unit, and a storage unit at the same time can be used as the first terminal device 120, the second terminal device 121, the third terminal device 122, and the fourth terminal device 123 at the same time; a smart watch that includes an input unit and an output unit can be used as the second terminal device 121 and the third terminal device 122 at the same time.
[0067] In some possible implementations, at least one sensor 11 in the IoT system of the disclosed embodiment may include at least one of the following types of sensors: visual sensors (such as cameras), laser radars, laser rangefinders, etc., wherein the cameras may be monocular cameras, binocular cameras, or multi-cameras, etc., and the disclosed embodiment does not limit the type of sensors. When the sensor is a visual sensor, image acquisition can be performed directly for the corresponding place; when the sensor is a laser radar, point cloud data can be acquired for the corresponding place, and three-dimensional reconstruction can be performed based on the acquired point cloud data to obtain a three-dimensional scene; when the sensor is a laser rangefinder, distance measurement can be performed for the corresponding place, and a depth image can be generated based on the distance measurement data. In the disclosed embodiment, the visual sensor and its acquired data image are used as an example for explanation. For other types of sensors and their acquired data, reference can be made to the implementation, and the disclosed embodiment will not be repeated.
[0068] In some possible implementations, at least one terminal device 12 in the IoT system of the disclosed embodiment may include at least one of the following types of terminal devices: PC, mobile phone terminal, handheld or laptop device, wearable device (such as smart watch, smart bracelet, augmented reality (AR) glasses, etc.), smart home appliances (such as smart speakers, smart TVs, smart alarm clocks, etc.), IVI (such as car speakers, car displays, etc.), etc. The disclosed embodiment does not limit the type of terminal devices.
[0069] In some possible implementations, at least one end-side device 12 in the IoT system of the present disclosure embodiment includes a terminal device that a user or a family has permission to use. At least one location corresponding to the IoT system includes at least one of at least one mobile device and at least one fixed location that a user or a family has permission to use.
[0070] In some optional examples, the mobile devices may include, for example, but are not limited to, at least one of vehicles, ships, flying equipment (such as airplanes, aircraft), smart terminals and other types of manned equipment. The number of mobile devices in at least one mobile device may be one or more. When at least one mobile device includes multiple mobile devices, the types of the multiple mobile devices may be the same or different, or may be partially the same. For example, at least one mobile device may include 2 vehicles, 1 airplane and 2 ships. The disclosed embodiment does not limit the type and number of mobile devices included in at least one place corresponding to the Internet of Things system.
[0071] In some optional examples, the fixed places may include, for example, but are not limited to, at least one of various places such as residences, business activity places, offices, entertainment places, and fitness places. The number of fixed places in at least one fixed place may be one or more. When at least one fixed place includes multiple fixed places, the types of the multiple fixed places may be the same or different, or may be partially the same. For example, at least one fixed place may include 2 residences and 1 office place. The disclosed embodiment does not limit the type and number of fixed places included in the at least one place corresponding to the Internet of Things system.
[0072] With the rapid development of AI technology and the continuous improvement of AI chip computing power, many end-side devices such as users' mobile terminals, IVI, PCs, etc. are equipped with AI chips, which can provide powerful AI computing capabilities. As the first end-side device, it becomes an AI computing center. Based on this embodiment, the IoT system is constructed by the end-side devices that the user or family has the right to use, and the sensors deployed in the places where the user or family has the right to use, thereby forming a family-wide IoT system, which can realize the scene data collection, AI computing processing, information input and output, and data interconnection and intercommunication within the system for the family-centered scene. Therefore, it is possible to realize the positioning of any object based on the AI algorithm within the scope of the IoT system. The IoT system can also be called a family-based IoT system or a system for positioning everything. In the system for positioning everything based on the family, many end-side devices such as users' mobile terminals, IVI, PCs, etc. equipped with AI chips are used to provide AI computing capabilities, make full use of idle AI computing resources, build a family AI computing center, and provide the computing capabilities required by the IoT system without the need to invest additional computing resources, saving costs.
[0073] In some possible applications, the IoT system of the embodiment of the present disclosure may be configured as an item query system to search for items within at least one location corresponding to the item query system. In this application scenario, a terminal device in the first terminal device 120 is configured as a computing device to implement the item search method of each embodiment of the present disclosure. In this application scenario, the computing device may also be referred to as an item search device.
[0074] When the Internet of Things system of the embodiment of the present disclosure is configured as an item query system, the storage unit in the fourth terminal side device 123 can be used to store at least one of the sensor acquisition data involved in the item query system (for example, images acquired by at least one sensor 11), the registration place information table, the registered item information table, the alarm item information, the log file, the registered user information table, the use permission information of the item query system, and other related information. Among them, the registered user information table includes the user registration information of at least one user registered in the item query system, and the registration place information table of any registered user (hereinafter referred to as the first user) includes the registration place information of at least one registration place registered by the first user in the item query system, and the registration place information of each registration place includes the place information of each registration place and the sensor information corresponding to each registration place. The registered item information table of the first user includes the item registration information of at least one registered item registered by the user in the item query system, and the item registration information of each registered item includes the item information of each registered item and the first place list; wherein, the item information of each registered item includes at least one of the image and item description information of each registered item, and when the item information of each registered item includes an image, the item registration information of each registered item also includes feature information obtained by extracting features from the image; the first place list of each registered item includes the place information of the second place where each registered item is allowed to be left behind. The alarm item information corresponding to any place is the information of the alarm item registered by the corresponding user and not allowed to be left behind in the corresponding registered place.
[0075] In a specific implementation, when the fourth end-side device 123 includes multiple terminal devices, the above-mentioned relevant information can be stored in one of the terminal devices included in the fourth end-side device 123 according to user settings, or can be stored in each terminal device included in the fourth end-side device 123 in a dispersed manner according to a preset method. For example, in some optional examples, the above-mentioned related information can be stored in each terminal device included in the fourth terminal device 123 according to the type; or, in other optional examples, each part of the above-mentioned related information can be stored in each terminal device included in the fourth terminal device 123 according to the access efficiency, for example, the images collected by the sensors deployed in each place are stored in the fourth terminal device located in each place, and the other information in the above-mentioned related information is stored in other fourth terminal devices; or, in some other optional examples, the above-mentioned related information can also be stored in each terminal device included in the fourth terminal device 123 in a manner of combining type and access efficiency; or, in some other optional examples, the above-mentioned related information can also be stored in each terminal device included in the fourth terminal device 123 in a manner of combining type and access efficiency; or, in some other optional examples, the above-mentioned related information can also be stored in each terminal device included in the fourth terminal device 123 according to the user setting. The embodiments of the present disclosure do not limit the specific manner in which the fourth terminal device stores information.
[0076] In specific applications, the stored data in the fourth end-side device 123 can be cleaned up in a preset manner to improve the storage resource utilization of the fourth end-side device 123. For example, in some optional examples, the sensor collection data in the fourth end-side device 123 that has been stored for a period greater than a preset period (e.g., 7 days) can be deleted regularly to free up storage space; in other optional examples, when the information stored in the fourth end-side device 123 is greater than the preset storage capacity (e.g., 100G), the sensor collection data with an earlier storage time in the fourth end-side device 123 can be deleted in order of storage time to limit the information stored in the fourth end-side device 123 to within the preset storage capacity range, thereby avoiding the inability to store new sensor collection data due to insufficient capacity of the end-side device. This non-disclosed embodiment does not limit the specific method for cleaning the data stored in the fourth end-side device 123.
[0077] In some possible implementations, a user (hereinafter referred to as the second user) may construct an Internet of Things system and configure it as an item query system, wherein the second user and the first user may be the same user or different users. The second user or family has the right to use multiple terminal devices, some of which include a computing unit, and all of which include an information input unit, an information output unit, and a storage unit. Among them, the system menu of the terminal device including the computing unit may include menu items for building an Internet of Things system and configuring it as related information for an item query system, and / or an application (APP) for building an Internet of Things system and configuring it as related information for an item query system may be installed. The second user may choose to build and configure the relevant information of the item query system through the system menu or the APP. The embodiment of the present disclosure does not limit the specific implementation method of building and configuring the relevant information of the item query system.
[0078] In an optional example, after the second user logs in to multiple terminal devices with his user account, the multiple terminal devices can be associated based on the user account; or, the second user can also associate multiple terminal devices based on a short-range communication method (such as Bluetooth communication). The embodiment of the present disclosure does not limit the association method of multiple terminal devices of the same user. The second user can obtain the device information of the multiple terminal devices associated with the second user through the system menu or APP and set up multiple terminal devices, for example, select which terminal devices to build an item query system, which terminal device including a computing unit is configured or designated as a computing device to implement the item search method of the embodiment of the present disclosure, which terminal device including a storage unit is designated as a storage device for relevant information of the item query system, and set the use permission information of the item query system (including user accounts with use permissions, etc.), etc., thereby completing the construction of the Internet of Things system and the configuration of the item query system information.
[0079] After the second user sets the usage permission information of the item query system, he can register as a user in the item query system to add the user registration information of the second user to the registered user information table of the item query system, register for a place to add the registered place information of his registered place to the registered place information table of the item query system, and register for an item to add the item registration information of his registered item to the registered item information table of the item query system.
[0080] After the second user sets the usage permission information of the item query system, he or she may also share the usage permission information with his or her family members (for example, the first user), and the family members may register as a user in the item query system based on the usage permission information, add the user registration information of the first user to the registered user information table of the item query system, register for a place, add the registered place information of his or her registered place to the registered place information table of the item query system, and register an item, add the item registration information of his or her registered item to the registered item information table of the item query system.
[0081] After the first user or the second user completes user registration in the item query system, they can set their alarm setting information, such as alarm device, alarm form (such as sound, text, light signal, vibration), etc., so that when the item query system detects that the registered item of the first user or the second user is lost, it can send an alarm message to them with its alarm setting information. Taking the second user as an example, in some optional examples, the item query system can generate alarm setting information in the user registration information of the second user based on the alarm setting request after receiving the alarm setting request sent by the second user. The alarm setting information includes an alarm device and an alarm form, wherein the alarm device includes one or more terminal devices in the item query system that the second user or family has the right to use. When the second user does not set the alarm setting information, the default alarm mode set by default in the item query system can be used as the alarm setting information of the second user, and the embodiments of the present disclosure do not limit this.
[0082] In the related art, when searching for items by installing a tracker on the item, the location data of the tracker will be uploaded to the cloud network of the service provider, which poses a risk of user privacy leakage. The item query system implemented by the IoT system of this embodiment is interconnected by the terminal device and sensor that the user or family has the right to use, thereby forming an item query system in the family scope. The item query system can realize the search of items based on visual technology, without the need to install additional electronic tools such as trackers for the items, and to firmly bind the electronic tools to the items, saving the cost required for investment, and can be applied to the search of any items. It has strong scalability and is easy to operate; and it is not limited by the battery power of the electronic tool, avoiding the limitation of the item search, and can realize the effective search of the items, thereby effectively improving the item search effect. In addition, the item query system uses one of the terminal devices as a computing device to provide the computing resources required for item search, and the required computing models are deployed in the private devices in the family scope. The configuration information of the entire item query system and the data involved in the item query process are only circulated between the private terminal devices in the family scope, without uploading to the cloud. Compared with the related art, it effectively guarantees the privacy security of the user.
[0083] Figure 2 FIG. 1 is an example diagram of an item query system according to an embodiment of the present disclosure. Figure 2 As shown, in this example, after the IoT system is configured as an item query system, a user (hereinafter referred to as the first user) has registered places in the item query system including a residence 13 and a vehicle 14, and the visual sensors 110 and 111 in at least one sensor 11 are respectively deployed in the living room and balcony of the residence 13; the visual sensor 112 in at least one sensor 11 is deployed in the cockpit of the vehicle 14. At least one end-side device 12 includes a PC, a mobile terminal, an IVI, a smart watch, and a smart speaker that the first user or his family has the right to use, wherein the PC, the mobile terminal, and the IVI respectively include a computing unit and a storage unit, which can be used as a first end-side device 120 and a fourth end-side device 123, and the IVI is configured as a computing device and is designated as a storage device for relevant information involved in the item query system; the PC, the mobile terminal, the IVI, the smart watch, and the smart speaker respectively include an information input unit and an information output unit, which can be used as a second end-side device 121 and a third end-side device 122. Each sensor and each end device in the item query system respectively includes a communication unit, and supports communication with each other through 5G, WiFi, and the like.
[0084] Based on Figure 2 In an application example of the item query system shown, a left-behind alarm for registered items can be implemented. For example, when the visual sensor 112 in the cabin of the vehicle 14 detects that the first user has left the vehicle 14, a notification message of the first user leaving the vehicle 14 is sent to the IVI, and the IVI obtains the alarm item information corresponding to the vehicle 14, and obtains the image captured by the visual sensor 112. Assuming that the alarm item indicated by the alarm item information is a headset, the headset is searched based on the alarm item information corresponding to the vehicle 14 and the image captured by the visual sensor 112. If the image corresponding to the headset is found from the image captured by the visual sensor 112, it means that the first user left the headset in the vehicle 14 after getting off the vehicle, and the wearable device (mobile phone terminal or smart watch) of the first user is instructed to send an alarm message to the first user in the form of voice, text, picture, etc.
[0085] Based on Figure 2In another application example of the item query system shown, item search can be implemented. The first user initiates an item search request in the form of voice, text, picture, etc. through the human-computer interaction interface (as an information input unit) in the mobile terminal, which carries item description information, such as "Please help me find the blue work badge"; the mobile terminal forwards the item search request to the IVI; the IVI calls all visual sensors in the item query system to collect images and obtains three images; and based on the first item description information, it sequentially traverses the three images to see whether the image corresponding to the blue work badge is included; if the image corresponding to the blue work badge is not included in the three images, a null value or a search result message indicating that the blue work badge is not found can be returned; if the image corresponding to the blue work badge is included in any of the three images (assuming that the image collected by the visual sensor 110), a search result message can be returned in the form of text, voice, image, etc. through the human-computer interaction interface (as an information output unit) in the mobile terminal, and the search result message includes an image (image collected by the visual sensor 110) marked with an object detection frame and / or position coordinates of the blue work badge, and position description information in the form of text, voice, etc. (for example, the blue work badge is on the sofa in the living room, etc.).
[0086] Based on Figure 2 In another application example of the item query system shown, tracking of registered items can be achieved. After the item is registered in the item query system, IVI obtains the item registration information of the registered item, calls all the visual sensors in the item query system to collect images, obtains three images, and determines the target registration location where the registered item is currently located based on the item registration information of the registered item and the three images, and then calls the visual sensor corresponding to the target registration location to perform image acquisition and target detection and position tracking for the registered item regularly (for example, once every 2 minutes), and records the tracked position. If the location of the registered item changes, the frequency of image acquisition and target detection is increased (for example, once every 20 seconds), and the collected image sequence is analyzed by the analysis model, and the analysis results are stored in the log file of the item query system. If the registered item is not visible afterwards, the possible location or possible destination of the registered item (for example, it is put into a bag) can be determined by the analysis results, increasing the possibility of finding the registered item.
[0087] Exemplary Methods
[0088] Figure 3 The flowchart of an exemplary embodiment of the method for searching for items disclosed in the present invention is shown in FIG. The method for searching for items disclosed in the exemplary embodiment is applied to an item query system and can be implemented by an electronic device (e.g., a computing device with computing capabilities, a terminal device configured as a computing device, etc.) in the application system. Figure 3As shown, the object search method of this embodiment includes the following steps:
[0089] Step 21, determining first object information and first sensor information corresponding to a first event.
[0090] The first sensor information includes sensor information of the first sensor, and the first sensor includes a sensor corresponding to at least one registered place, wherein the registered place is a place registered in the item query system. The first sensor may include one or more sensors, wherein the number of sensors corresponding to different registered places in at least one registered place may be the same or different, and each registered place may correspond to one sensor or multiple sensors, which is not limited in the embodiments of the present disclosure.
[0091] Each sensor information is used to uniquely identify a sensor in the item query system. The sensor information may include at least one of the following information: the name, number, corresponding place, and position in the corresponding place of the sensor. For example, in an optional example, a sensor information is: camera-0098-residence 1-living room 002, indicating that the sensor identified is a camera with a number of 0098, which is deployed in the living room of residence 1 and is the 002nd camera in the living room of residence 1.
[0092] The first event in the embodiment of the present disclosure is specifically an event that can trigger item search. The first event can be set according to actual needs and can be updated according to needs. For example, in some possible implementations, the first event may include but is not limited to at least one of the following: the first user leaves the first place, an item search request sent by the first user is received, new item registration information is detected in the item query system, etc., and the embodiment of the present disclosure does not limit this. The first user can be any user registered in the item query system, and the embodiment of the present disclosure does not limit this.
[0093] Step 22: Acquire a first image captured by a first sensor.
[0094] When the first sensor includes one sensor, the first image may include one image or multiple images. When the first sensor includes multiple sensors, the first image may include one image or multiple images collected by each sensor in the first sensor, which is not limited in the embodiments of the present disclosure.
[0095] Step 23: search for the first item based on the first item information and the first image to obtain a first search result.
[0096] Based on the disclosed embodiment, a technical solution for searching for items based on visual technology is provided, by determining the first item information and the first sensor information corresponding to the first event, the first sensor information includes the sensor information of the first sensor corresponding to at least one registered place, wherein the registered place is a place registered in the item query system, and then the first image captured by the first sensor is obtained, and then the first item search is performed based on the first item information and the first image to obtain the first search result, thereby realizing the search for the item. This embodiment does not need to install additional electronic tools such as trackers for the items, and does not need to firmly bind the electronic tools to the items, thus saving the cost required for investment, and can be applied to the search of any items, with strong scalability and easy operation; and it is not limited by the battery power of the electronic tools, thus avoiding the limitation on the search for items, and can realize the effective search for items, thereby effectively improving the search effect for items.
[0097] In some possible implementations, the first event is the first user leaving the first place. Figure 4 FIG. 1 is an exemplary flow chart of determining first item information and first sensor information in another exemplary embodiment of the item search method disclosed in the present invention. Another exemplary embodiment of the item search method is an embodiment corresponding to the event that the first user leaves the first place. Figure 4 As shown, in Figure 3 Based on the embodiment shown, in this embodiment, step 21 may include:
[0098] Step 2110, in response to detecting that the first user leaves the first place, obtaining alarm item information corresponding to the first place as first item information, and obtaining sensor information corresponding to the first place as first sensor information.
[0099] The first user in this embodiment may be any user registered in the item query system, and this embodiment of the present disclosure does not impose any limitation on this.
[0100] In this embodiment, the first place is one of the at least one registered places registered by the first user in the item query system, and the alarm item information includes information of at least one alarm item, which is an item registered by the first user and is not allowed to be left in the corresponding registered place.
[0101] Through this step 2110, when it is detected that the first user leaves the first place, information of at least one alarm item registered by the first user and not allowed to be left in the first place can be obtained, and sensor information of the sensor corresponding to the first place (i.e., deployed in the first place) can be obtained, so as to obtain images collected by the sensor corresponding to the first place, and combine the alarm item information to find out whether there are any items left in the first place that are not allowed to be left.
[0102] and Figure 4 Correspondingly to the illustrated embodiment, in some optional examples, in step 22, a first image captured by the first sensor can be obtained, and the first image can include one or more images most recently captured by each sensor in the first sensor; or, each sensor in the first sensor can be called to perform image capture, and each sensor can capture one or more images respectively to obtain a first image.
[0103] Based on this embodiment, when the first user leaves the first place, images collected by the sensor corresponding to the first place can be obtained to find out whether there are any items left in the first place that are not allowed to be left behind.
[0104] In some possible implementations, before step 2110, whether the first user leaves the first place may also be determined in the following manner:
[0105] Step 2100: Acquire a second image captured by the first sensor according to the first cycle.
[0106] The second image may include one or more images most recently captured by each sensor in the first sensor.
[0107] Among them, when the value of the first period is small, the user leaving the first place can be detected more timely, but it is necessary to detect the second image obtained in each first period, which requires more computing resources; when the value of the first period is large, relatively fewer computing resources are required, but it may not be possible to detect the user leaving the first place in time. Therefore, in actual applications, the first user can set the value of the first period according to actual needs. For example, the value of the first period can be set to 5 minutes, 10 minutes, etc., and the value of the first period can be modified according to actual needs. The embodiment of the present disclosure does not limit the value of the first period and the modification method.
[0108] Step 2101: Perform a first detection on a first user based on a second image, and determine whether the first user has left a first place based on the obtained first detection result.
[0109] If the first detection result indicates that the image of the first user is detected from the second image, it can be determined that the first user is in the first place, that is, has not left the first place; otherwise, if the first detection result indicates that the image of the first user is not detected from the second image, it can be determined that the first user is not in the first place, that is, has left the first place.
[0110] Optionally, the first detection for the first user may be face detection, skeleton detection, etc. for the first user. In some optional examples, face detection and skeleton detection for the first user may be performed on the second image based on the face image (hereinafter referred to as the registered face image) and skeleton image (hereinafter referred to as the registered skeleton image) provided by the first user when registering in the item query system, and whether the image of the first user is detected from the second image is determined based on whether there is a face corresponding image with a similarity higher than a preset similarity threshold with the registered face image or a skeleton corresponding image with a similarity higher than a preset similarity threshold with the registered skeleton image in the second image. The embodiments of the present disclosure do not limit the specific detection method of the first detection.
[0111] Based on this embodiment, it is possible to detect whether the first user has left the first place by periodically acquiring images collected by sensors corresponding to the first place, so as to timely discover that the first user has left the first place.
[0112] Figure 5 FIG. 2 is an exemplary flow chart of searching for a first item in another exemplary embodiment of the item search method disclosed in the present invention. Figure 5 As shown, in Figure 3 or Figure 4 Based on the embodiment shown, in this embodiment, step 23 may include:
[0113] Step 2310, in response to the first item information including item description information and feature information of the first item, target detection is performed on the first image based on the item description information of the first item to obtain a first candidate region (Region Proposal, RP).
[0114] In this embodiment, the first item is an item (such as a handbag) corresponding to the first item information, that is, an alarm item corresponding to the first place. The item description information of the first item may include words used to describe the name, color, purpose and other related information of the first item, such as "blue work badge", "red wallet" and so on. The feature information of the first item is feature information obtained by extracting features from the image of the first item.
[0115] The first candidate area in this embodiment is a candidate area obtained by performing object detection on the first image based on the object description information of the first object. The first candidate area may include one candidate area or multiple candidate areas. If the first candidate area includes multiple candidate areas, steps 2311-2312 are performed for each candidate area included in the first candidate area.
[0116] In some possible implementations, in step 2310, the region-word alignment capability learned in advance from image-text pair data by an open vocabulary object detection (OVD) algorithm can be used to perform object detection on the first image based on the vocabulary in the item description information of the first item to obtain a first candidate region, which may include an entity corresponding to the candidate region (i.e., a detection box, bounding box) and a phrase-level description thereof.
[0117] Alternatively, in some other possible implementations, in step 2310, a visual grounding algorithm can also be used to perform target detection on the first image based on the object description information of the first object to obtain a first candidate area. The first candidate area can include an entity corresponding to the candidate area (i.e., a detection box) and can optionally include an attribute description about the entity (i.e., a description of the content in the detection box).
[0118] Alternatively, in some other possible implementations, in step 2310, a visual grounding model (VGM) that integrates the OVD algorithm and the visual positioning algorithm can be used to directly frame the first object from the first image based on the first object information, that is, determine the object detection frame of the first object from the first image. In a specific implementation, the VGM can be implemented using an encoder-decoder structure, wherein the encoder can include a text encoder, an image encoder, and a visual encoder. The text encoder can extract features from the input text (e.g., the item description information of the first item) to obtain text features, the image encoder can extract features from the input feature information (e.g., the feature information of the first item) to obtain image features, and the visual encoder can extract features from the input image (e.g., the first image) to obtain visual features (e.g., the first candidate feature information). The features extracted by the encoder are output to the decoder, and the decoder outputs the object detection frame of the first item based on the features output by the encoder.
[0119] In step 2310, if no candidate region is obtained by performing target detection on the first image, steps 2311-2312 are not performed, and a first search result in which the first object is not found is obtained.
[0120] Step 2311, extract image features from the first candidate area to obtain first candidate feature information.
[0121] Step 2312: Based on the similarity between the first candidate feature information and the feature information of the first object, determine whether the first image includes an image corresponding to the first object, and obtain a first search result.
[0122] Specifically, when the similarity between the first candidate feature information and the feature information of the first object is greater than a first preset threshold, the first candidate area is considered to be similar to the image corresponding to the first object, and it is determined that the first image includes the image corresponding to the first object; when the similarity between the first candidate feature information and the feature information of the first object is not greater than the first preset threshold, the first candidate area is considered to be dissimilar to the image corresponding to the first object, and it is determined that the first image does not include the image corresponding to the first object. The first preset threshold can be determined according to actual needs and can be updated as needed, and the embodiments of the present disclosure are not limited to this.
[0123] When the first candidate area includes multiple candidate areas, if the similarity between the first candidate feature information corresponding to any candidate area in the first candidate area and the feature information of the first item is greater than a first preset threshold, the first search result is that the first item is found. Otherwise, if the similarity between the first candidate feature information corresponding to all candidate areas in the first candidate area and the feature information of the first item is not greater than the first preset threshold, the first search result is that the first item is not found.
[0124] If the first image includes an image corresponding to the first object, the first search result may include: the first image annotated with an object detection frame (i.e., a detection frame included in the first candidate area) and / or position coordinates of the first object, and at least one of the position description information of the first object. If the first image does not include an image corresponding to the first object, the first search result may be a null value.
[0125] Optionally, before step 2310, the specific information included in the first item information can be determined. When the first item information includes item description information and feature information of the first item, when target detection is performed from the first image based on the item description information of the first item to obtain the first candidate area, it can be considered that a suspected target is detected, and then the feature information of the first item is compared with the features of the first candidate area to further confirm whether the suspected target is the first item (for example, a handbag). That is, the first item search is performed based on the item description information and feature information of the first item at the same time, and it is determined whether the first image includes an image corresponding to the first item, thereby improving the accuracy of the first item search.
[0126] Optionally, see Figure 5 In another exemplary embodiment of the object search method, step 23 may further include:
[0127] Step 2313: In response to the first object information including the object description information of the first object but not including the feature information of the first object, target detection is performed on the first image based on the object description information of the first object to obtain a second candidate region.
[0128] The second candidate region in this embodiment is a candidate region obtained by performing object detection on the first image based on the object description information of the first object. The second candidate region may include one candidate region, may include multiple candidate regions, or may be empty, that is, no candidate region is obtained.
[0129] In some optional examples, step 2313 may utilize the OVD algorithm, the visual positioning algorithm, or the VGM that integrates the OVD algorithm and the visual positioning algorithm to perform target detection on the first image based on the item description information of the first item. For details, please refer to the relevant introduction of the above embodiments, which will not be repeated here.
[0130] When this embodiment is implemented using a VGM of an Encoder-Decoder structure, the text features can be extracted from the input text (e.g., the item description information of the first item) by a text encoder, and the visual features can be extracted from the input image (e.g., the first image) by a visual encoder. The features extracted by the encoder are output to the decoder, and the decoder outputs an object detection frame of the first item based on the features output by the encoder. For details, please refer to the relevant introduction of the above embodiment, which will not be repeated here.
[0131] Step 2314, obtaining a first search result based on the second candidate area.
[0132] Specifically, if no candidate region is obtained by performing target detection on the first image based on step 2313, that is, the candidate region is empty, step 2314 is not performed, and a first search result in which the first object is not found is obtained. If one or more candidate regions are obtained by performing target detection on the first image based on step 2313, a first search result in which the first object is found is obtained.
[0133] Based on this embodiment, if the first item information includes item description information of the first item but does not include feature information of the first item, the first item can be searched directly using the item description information of the first item.
[0134] Optionally, see Figure 5 In another exemplary embodiment of the object search method, step 23 may further include:
[0135] Step 2315 , in response to the first object information including feature information of the first object but excluding object description information of the first object, target detection is performed on the first image to obtain a third candidate region.
[0136] In operation 2315, any target detection algorithm may be used to perform target detection on the first image to obtain a third candidate region.
[0137] The third candidate region in this embodiment is a candidate region obtained by performing target detection on the first image. The third candidate region may include one candidate region or multiple candidate regions. If the third candidate region includes multiple candidate regions, steps 2316-2317 are performed for each candidate region included in the third candidate region.
[0138] In step 2315, if no candidate region is obtained by performing target detection on the first image, steps 2316-2317 are not executed, and a first search result in which the first object is not found is obtained.
[0139] Step 2316: extract image features from the third candidate region to obtain third candidate feature information.
[0140] Step 2317: Based on the similarity between the third candidate feature information and the feature information of the first object, determine whether the first image includes an image corresponding to the first object, and obtain a first search result.
[0141] In some optional examples, the specific implementation of step 2317 can refer to the implementation of step 2312, which will not be described here. Based on this embodiment, if the first item information includes the feature information of the first item but does not include the item description information of the first item, the feature information of the first item can be directly used to search the first image for the first item.
[0142] Optionally, see Figure 4 In some possible implementations, based on the above Figure 4-Figure 5 After any embodiment obtains the first search result, the method may further include:
[0143] Step 24: In response to the first search result that the first item is found, sending an alarm message to the first user to prompt that the first item is left in the first place.
[0144] In some optional examples, an alarm message may be sent to a terminal device of the first user in the item query system, and the terminal device may display the alarm message to the first user in a manner preset by the first user or in a manner set by default in the item query system.
[0145] In some optional examples, an alarm message can be sent to the first user in a corresponding alarm form through the alarm device specified by the first user according to the alarm setting information pre-set by the first user, such as the alarm device, the alarm form (such as sound, text, light signal, vibration), etc. When the first user has not set the alarm setting information, the default alarm mode set by default in the item query system can be used to send the alarm message to the first user. For example, if it is detected that the smart watch of the first user is in a wearing state, the alarm message can be sent to the first user by controlling the vibration of the smart watch, or by playing a prompt message through the voice of the first user's mobile phone terminal, etc. The embodiment of the present disclosure does not limit the specific method of sending the alarm message to the user.
[0146] Based on this embodiment, after the first object is found in the first image, an alarm message may be sent to the first user to promptly remind the first user that the first object is left in the first place.
[0147] Based on the above Figure 4-Figure 5 After any embodiment obtains the first search result, if the first search result is that the first item is not found, the corresponding alarm operation shown in step 24 is not performed.
[0148] Optionally, in some possible implementations, the item query system of the embodiment of the present disclosure may include at least one of a registered place information table and a registered item information table of the first user.
[0149] The registration place information table of the first user may include the registration place information of at least one registration place registered by the first user in the item query system. The registration place information of each registration place may include the place information of each registration place and the sensor information corresponding to each registration place. The place information of each registration place is used to uniquely identify a registration place in the item query system. The place information may include at least one of the name, number, and location of the place. For example, in an optional example, the place information of a registration place is: Residence 1-Living Room, indicating that the place identified is the living room of Residence 1.
[0150] The registered item information table of the first user may include item registration information of at least one registered item registered by the first user in the item query system. The item registration information of each registered item may include item information of each registered item and a first place list. The item information of each registered item may include at least one of an image and item description information of each registered item. When the item information of each registered item includes an image, the item registration information of each registered item may also include feature information obtained by extracting features from the image. The first place list of each registered item includes the place information of the second place where each registered item is allowed to be left, and therefore, the first place list may also be referred to as a place whitelist.
[0151] Based on this embodiment, the alarm item information corresponding to the first place in the above step 2110 can be determined based on the first user's registered place information table and the first place list in the registered item information table. Figure 7 Description of the illustrated embodiment.
[0152] Optionally, in Figure 4-Figure 5 On the basis of any embodiment, the steps of performing place registration and item registration in the item query system may also be included. The two may be performed simultaneously or in any order, and the embodiments of the present disclosure do not limit this.
[0153] Figure 6 FIG. 2 is a flow chart of an exemplary embodiment of performing place registration in the embodiment of the present disclosure. Figure 6 As shown, in some possible implementations, taking the first user as an example, the place registration can be performed in the item query system in the following manner:
[0154] Step 31: receiving a first registration request sent by a first user, wherein the first registration request includes registration location information of a fifth location, and the registration location information of the fifth location includes location information of the fifth location and sensor information corresponding to the fifth location.
[0155] The first registration request is a request for requesting to register a place in the item query system.
[0156] The fifth place in this embodiment is the place requested to be registered by the first user (e.g., the living room of residence 1, the balcony of residence 1, the car cabin, etc.), that is, the place to be registered, which is a place that has not yet been registered in the item query system. The fifth place may include one place, indicating that the first user requests to register one place in the item query system this time; or the fifth place may also include multiple places, indicating that the first user requests to batch register multiple places in the item query system this time, and the embodiment of the present disclosure does not limit this.
[0157] In some optional examples, the first user may send a first registration request through a terminal device of the first user in the item query system.
[0158] Step 32, adding the registered place information of the fifth place to the registered place information table of the first user, thereby realizing the registration of the fifth place.
[0159] After the fifth place is registered in the item query system, it is transformed from a place to be registered into a registered place of the first user in the item query system. Based on this embodiment, the user's place registration in the item query system is realized.
[0160] Based on this embodiment, the first user's place registration in the item query system is realized. The first user can register for places in the item query system where networked sensors are deployed according to the network coverage of sensors (such as cameras) in the item query system, so as to subsequently perform item left behind alarm or item search for the registered places.
[0161] Figure 7 FIG. 1 is a flow chart of an exemplary embodiment of item registration in the present disclosure. Figure 7 As shown, in some possible implementations, taking the first user as an example, the item can be registered in the item query system in the following manner:
[0162] Step 41: receiving a second registration request sent by a first user, wherein the second registration request includes item information of a second item, and the item information of the second item includes at least one of an image of the second item and item description information.
[0163] The first registration request is a request for requesting item registration in the item query system.
[0164] The second item in this embodiment is an item that the first user requests to register (for example, a wallet, work badge, handbag, lipstick, earphone, mobile phone terminal, etc.), that is, an item to be registered, which is an item that has not yet been registered in the item query system. The second item can be the same item as the first item, or it can include the first item. The second item can include one item, indicating that the first user requests to register one item in the item query system this time; or the second item can also include multiple items, indicating that the first user requests to batch register multiple items in the item query system this time, and the disclosed embodiment does not limit this.
[0165] The item description information may include any information used to describe the attributes of the item, such as but not limited to information used to describe the name, color, size, etc. of the item, and may be sent after being input by the user through a terminal device including an information input unit (such as a human-computer interaction interface) in the item query system. For example, in some optional examples, the item description information is "blue work badge", "light-colored lipstick", etc. The image of the second item may be acquired and sent by the user through a terminal device (such as a mobile phone terminal, IVI, etc.) with image acquisition capability and including an information input unit.
[0166] In some optional examples, the first user may send the second registration request through the terminal device of the first user in the item query system.
[0167] Step 42: Add the item registration information of the second item to the first user's registered item information table, where the item registration information of the second item includes the item information of the second item.
[0168] Step 43, in response to the item information of the second item including an image of the second item, feature extraction is performed on the image of the second item to obtain feature information of the second item, and the feature information of the second item is added to the item registration information of the second item, thereby achieving registration of the second item.
[0169] Steps 41 and 43 of this embodiment may also utilize the OVD algorithm, the visual positioning algorithm, or the VGM that integrates the OVD algorithm and the visual positioning algorithm to perform visual positioning of the image of the second object based on the object description information of the second object, and then perform feature extraction on the image of the second object to obtain feature information of the second object.
[0170] After the second item is registered in the item query system, it is transformed from an item to be registered into a registered item of the first user in the item query system.
[0171] Based on this embodiment, the registration of items by the first user in the item query system is realized. The first user can register items that need to be queried or left for alarm in the item query system as needed.
[0172] In the embodiment of the present disclosure, during the registration process of an item (referred to as the second item), a place whitelist (referred to as the first place list in the embodiment of the present disclosure) can be directly set for the second item, that is, a list of places where the second item is allowed to be left. Alternatively, a place whitelist can be set for the second item at any time after the registration of the second item is completed, and the embodiment of the present disclosure does not limit this.
[0173] Optionally, taking setting a place whitelist after the second item is registered as an example, the place whitelist can be set for the second item in the following manner:
[0174] Step 44: receiving a first place list setting request sent by the first user for the second item, wherein the first place list setting request includes location information of a second place where the second item is allowed to be left.
[0175] The first place list setting request is a request for setting a first place list in the item query system.
[0176] The second place can also be called a white list place. When the second object is left at the second place, no alarm will be initiated.
[0177] In some optional examples, the first user may send a first place list setting request through a terminal device of the first user in the item query system.
[0178] Step 45, adding the location information of the second location to the first location list in the item registration information of the second item.
[0179] Based on this embodiment, a location whitelist setting for items is implemented. The first user can set a location whitelist for the second item as required. For example, when the first user's registered locations include residence and private vehicle, if the second item (such as a wallet) is allowed to be left at the residence but not in the vehicle, the residence can be set as a whitelist location.
[0180] Optionally, after setting a place whitelist for the second item, based on the first user's registered place information table and the second item's place whitelist, alarm item information corresponding to a non-whitelist place (also called an alarm place) can be generated for the second item in the item query system, so that an alarm is triggered when the second item is left at a non-whitelist place, to remind the first user that the second item is left at the alarm place. The non-whitelist place is a registered place where the second item is not allowed to be left. In some possible implementations, see Figure 7 After step 45, the alarm item information corresponding to the alarm location may be generated for the second item in the following manner:
[0181] Step 46, based on the first user's registered location information table and the second item's first location list, determine a second location list for the second item, the second location list including location information of at least one alarm location, wherein the alarm location is a location where the second item is not allowed to be left behind.
[0182] In step 46, all the registered places included in the registered place information table of the first user, except the places in the first place list of the second object, may be obtained as the alarm places in the second place list of the second object.
[0183] Step 47, adding the information of the second article to the alarm article information corresponding to each alarm place in the second place list, that is, adding the information of the second article as one piece of alarm article information to the alarm article information corresponding to each alarm place.
[0184] The information of the second item includes at least part of the item registration information of the second item, for example, it may include at least one of the image, item description information and feature information of the second item. The alarm item information corresponding to each alarm location may include information of at least one registered item of the first user in the item query system.
[0185] Based on this embodiment, the alarm item information corresponding to each alarm location can be generated. For example, when the first user's registered locations include residence and private vehicle, if the second item (such as a wallet) is allowed to be left at the residence but not at the vehicle, after the first user sets the residence as a whitelist location, the item query system can add the second item (such as a wallet) to the corresponding alarm item information of the vehicle.
[0186] Alternatively, in the embodiment of the present disclosure, during the registration of the second item, a non-place whitelist (i.e., the second place list) may be directly set for the second item, i.e., a list of places where the item is not allowed to be left. Alternatively, a non-place whitelist may be directly set for the second item at any time after an item is registered, and the embodiment of the present disclosure does not limit this.
[0187] Optionally, see Figure 7 , a non-place whitelist (i.e., a second place list) may be set for the second item in the following manner:
[0188] Step 48: receiving a second place list setting request sent by the first user for the second item, wherein the second place list setting request includes location information of at least one alarm location, wherein the alarm location is a location where the second item is not allowed to be left.
[0189] When the second object is left at any alarm location, an alarm may be initiated to remind the first user that the second object is left at the alarm location.
[0190] Step 49: Add the location information of the at least one alarm location to the second location list in the item registration information of the second item.
[0191] Based on this embodiment, the first user can directly set a non-place whitelist for the second item as required. For example, when the first user's registered places include residence and private vehicle, if the second item (such as a wallet) is allowed to be left at the residence but not in the vehicle, the first user can directly set the vehicle as the alarm place for the second item (such as a wallet) without setting a place whitelist for the second item (such as a wallet).
[0192] Optionally, after setting a second place list for the second item, the item registration information of the first user registered in the item query system corresponding to each alarm place in the second place list can be obtained to determine the alarm item information corresponding to each alarm place, that is, at least part of the item registration information of each registered item by the first user in the item query system corresponding to each alarm place is obtained, such as at least one of the image, item description information and feature information of each registered item, as the alarm item information corresponding to each alarm place.
[0193] Figure 8 FIG. 1 is a schematic diagram of an application flow of another exemplary embodiment of the method for finding items in the embodiment of the present disclosure. This application embodiment can realize an alarm for items left behind. Figure 8 As shown above Figure 4-Figure 7 An application of another exemplary embodiment of the object search method shown in the figure, taking the target object (corresponding to the first object and the second object in the above embodiment) as a handbag and the sensor as a camera as an example, the application embodiment includes the following steps:
[0194] A11, target item registration: the user takes an image of the target item (handbag) through a camera on a terminal device (such as a mobile terminal or IVI) in the item query system, inputs the name of the target item "handbag" as item description information and selects the image of the target item through the human-computer interaction interface on the terminal device, and sends an item registration request (i.e., the above-mentioned second registration request) to the computing device in the item query system, which includes the image and name of the target item; the computing device calls VGM to perform feature extraction on the image of the target item to obtain feature information of the target item, and adds the item registration information of the target item in the user's registered item information table, including the name, image and feature information of the target item, thereby realizing the target item registration;
[0195] A12, location whitelist setting: After the user selects the whitelist location of the target item from the pre-registered registered locations (e.g., living room, balcony, car cabin) through the human-computer interaction interface, the user sends a first location list setting request to the computing device, and the computing device adds the location information of the whitelist location to the location whitelist (i.e., the above-mentioned first location list) in the object registration information of the target item, thereby implementing the location whitelist setting. For example, when the registered locations include the living room, balcony, and car cabin, if the target item is allowed to be left in the living room but not allowed to be left on the balcony or in the car cabin, the living room is set as the whitelist location. When the target item is left in the living room, no alarm will be initiated;
[0196] A13, alarm item information setting: the computing device determines that other places in the user's registered places except the whitelist places are alarm places, and adds the information of the target item (including name, image and feature information) to the alarm item information corresponding to each alarm place. For example, if the whitelist place of the handbag is the living room, the information of the handbag is added to the alarm item information of the balcony and the car cabin respectively;
[0197] A14, target object search: when the user leaves any alarm location A (assuming that the user gets off the car and leaves the car cabin), the computing device senses that the user has left the car cabin through the camera corresponding to the car cabin (for example, the DMS and OMS cameras in the car cabin), and calls the DMS and OMS cameras in the car cabin to collect images. Since the car cabin may correspond to information of multiple alarm objects, based on all the alarm object information corresponding to the alarm location and the images collected by the DMS and OMS cameras, the VGM is called to perform target detection on each alarm object one by one. If a suspected alarm object is found, the image features of the candidate area where the suspected alarm object is located are extracted and compared with the feature information in the item registration information of the alarm object to further confirm whether it is the alarm object; if the alarm object is not found, the subsequent process is not executed, that is, no alarm is triggered;
[0198] A15, send an alarm message: If it is confirmed to be an alarm item after feature comparison, it is determined that the alarm item is found, and an alarm message is immediately sent to the user's mobile phone terminal, wearable device and other terminal devices in the item query system to remind the user that an alarm item (such as headphones) is left in the car cabin.
[0199] In this application embodiment, steps A14-A15 are operations triggered by the event that the user leaves any alarm location, and steps A11-A13 are optional operations that can be executed in advance.
[0200] Fig. 9 FIG. 1 is a flow chart of another exemplary embodiment of the method for searching for items disclosed in the present invention. Fig. 9As shown in the above Figure 3-Figure 7 On the basis of any embodiment, the embodiment of the present disclosure may further include:
[0201] Step 51: Acquire a third image captured by the second sensor.
[0202] The second sensor includes a sensor corresponding to the sensor information in the first user's registered place information table. Since the first user's registered place information table may include multiple registered places, and each registered place may correspond to multiple sensors, in step 51, the sensors corresponding to all the sensor information in the first user's registered place information table may be obtained at once as the second sensor, and steps 51-52 of this embodiment may be executed to determine the third place where the first item is located; or, the sensors corresponding to part of the sensor information in the first user's registered place information table (for example, sensor information corresponding to a preset number of registered places) may be obtained in sequence as the second sensor, and steps 51-52 of this embodiment may be executed to determine the third place where the first item is located. If the third place where the first item is located cannot be determined, the sensor corresponding to the next part of the sensor information may be obtained as the second sensor, and steps 51-52 of this embodiment may be executed until the third place where the first item is located is determined.
[0203] When the second sensor includes one sensor, the third image may include one image or multiple images. When the second sensor includes multiple sensors, the third image may include one image or multiple images collected by each sensor in the second sensor, which is not limited in the embodiments of the present disclosure.
[0204] Step 52: searching for the first object based on the object registration information of the first object and the third image to determine the third place where the first object is located.
[0205] Among them, the item registration information of the first item includes the item information of the first item. In step 52, the first item can be searched based on the item information in the item registration information of the first item and the third image. The specific implementation method can refer to the implementation method of searching the first item based on the first item information and the first image in the above embodiments, which will not be repeated here.
[0206] In step 52, after determining the third image including the image corresponding to the first object, the third image including the image corresponding to the first object is used as the target image, and the place corresponding to the second sensor that captured the target image can be determined as the third place where the first object is located.
[0207] Step 53, according to the second cycle, calling the third sensor corresponding to the third field to perform image acquisition, and based on the acquired first image sequence and the object registration information of the first object, performing target detection and position tracking on the first object to obtain a first position tracking result.
[0208] The first image sequence may include a plurality of frames of images acquired by the third sensor according to the second period and arranged based on the image acquisition time sequence.
[0209] The second period can be set according to actual needs and can be updated as needed. In actual applications, for items whose positions change frequently (such as mobile phone terminals), a smaller second period (such as 2 minutes) can be set to achieve timely tracking of the items; and for items whose positions change less frequently (such as wallets), a larger second period (such as 20 minutes) can be set, and so on. The embodiments of the present disclosure do not limit this.
[0210] After determining the third place where the first object is located, through step 53, the third sensor corresponding to the third place can be specifically called to collect images according to the second cycle, and based on the object information in the object registration information of the first object, an arbitrary object detection algorithm is used to perform target detection on the image collected by the third sensor to track the position of the first object according to the second cycle. The first position tracking result can be a first image sequence marked with an object detection frame of the first object.
[0211] Among them, the implementation method of performing target detection on the image collected by the third sensor based on the item information of the first item can refer to the implementation method of searching for the first item based on the first item information and the first image in the above embodiments, which will not be repeated here.
[0212] Step 54, based on the first position tracking result, in response to a change in the position of the first object, according to a third cycle, calling a third sensor to perform image acquisition, and based on the acquired second image sequence and the object registration information of the first object, performing target detection and position tracking on the first object.
[0213] The third period is shorter than the second period. The third period can be set according to actual needs, for example, can be set to 10 seconds, and can be updated as needed.
[0214] The second image sequence may include multiple frames of images collected by the third sensor according to the third period and arranged based on the image collection time sequence. Through this step 54, when the position of the first object changes, the position tracking period of the first object can be reduced and the position tracking frequency can be increased, so as to achieve effective tracking of the position of the first object. For details, please refer to the implementation method of step 53, which will not be repeated here.
[0215] Step 55 : In response to not detecting the first object in the most recent frame of image acquired by the third sensor, acquiring a third image sequence.
[0216] The third image sequence includes: at least one frame of image acquired by the third sensor and acquired earlier than the most recent frame of image.
[0217] Step 56: Use the analysis model to perform event analysis on the third image sequence to obtain an event analysis result of the first object, and store the event analysis result of the first object in a log file of the object query system.
[0218] The event analysis results may include the possible destination of the first object (for example, put into a bag) or the possible location (for example, on a sofa), and may also include a third image sequence marked with an object detection frame of the first object, which is not limited in this embodiment of the present disclosure.
[0219] The analysis model can be a pre-trained deep learning model, such as a convolutional neural network (CNN) model, a recurrent neural network (RNN) model, a long short term memory network (LSTM), etc., which is not limited in the embodiments of the present disclosure. In some optional examples, multiple sample image sequences can be obtained, each sample image sequence includes a sample object image and is annotated with the destination information of the sample object. The initial deep learning model is trained in a supervised or unsupervised manner using the multiple sample image sequences. The analysis model obtained by training can predict the destination or position of the target in the image sequence for the input image sequence.
[0220] Based on steps 55-56, during the process of target detection and position tracking of the first object, if it is found that the first object is not tracked, that is, the first object is invisible, event analysis can be performed on at least one frame of image before the first object becomes invisible to analyze the possible whereabouts or possible location of the first object.
[0221] Based on this embodiment, the positioning and position tracking of the registered item is realized. When the registered item is not visible, event analysis can be performed on the image sequence before the registered item becomes visible, and the obtained event analysis results can be stored in a log file. When the registered item is not found subsequently, the possible whereabouts or possible location of the registered item can be determined based on the event analysis results, thereby increasing the possibility of finding the item, thereby improving the efficiency and effect of item search.
[0222] Fig.10FIG. 1 is a schematic diagram of an application flow of another exemplary embodiment of the object search method in the embodiment of the present disclosure. This application embodiment can realize target tracking for registered objects. Fig.10 As shown above Fig. 9 An application of another exemplary embodiment of the object search method shown in the figure, taking the target object (corresponding to the first object and the second object in the above embodiment) as a work badge and the sensor as a camera as an example, the application embodiment includes:
[0223] A21, target item registration: the user takes an image of the target item (work badge) through a camera on a terminal device (such as a mobile terminal or IVI) in the item query system, inputs the name of the target item "work badge" as the item description information through the human-computer interaction interface of the terminal device, selects the image of the target item, and sends an item registration request (i.e., the above-mentioned second registration request) to the computing device in the item query system, which includes the image and name of the target item; the computing device calls VGM to perform feature extraction on the image of the target item to obtain feature information of the target item, and adds the item registration information of the target item to the user's registered item information table, including the name, image and feature information of the target item, thereby realizing the target item registration;
[0224] A22, search for the target location where the target item is located: the computing device obtains images captured by cameras corresponding to all registered locations of the user in the item query system. Assuming that all registered locations of the user include the living room, balcony, and car cabin, the computing device obtains images captured by cameras in the living room, balcony, and car cabin, and calls VGM to perform target detection based on the item registration information of the target item and the images captured by cameras in the living room, balcony, and car cabin, and searches for the target item by feature comparison to determine the target location where the target item is located (i.e., the third location mentioned above), which is assumed to be the living room;
[0225] A23, target detection and position tracking: the computing device periodically (for example, once every 2 minutes) calls the images captured by the camera in the living room, calls the VGM to perform target detection based on the images captured by the camera in the living room and the object registration information of the target object, and performs position tracking by feature comparison, and records the target position; if it is found that the position of the target object has changed, triggering a key event response, the computing device shortens the target detection and position tracking cycle, increases the tracking frequency, and saves the image sequence (i.e., the third image sequence) collected during the tracking process after the key event response is triggered;
[0226] A24, event analysis: when the target object is not visible (i.e., the target is not detected), the computing device invokes the analysis model to perform event analysis on the third image sequence, obtains the event analysis result of the target object, including the possible location or destination of the target object (e.g., it is put into a bag), and stores it in the log file of the object query system;
[0227] A25, search for target items based on user requests: When a user initiates a search request for a target item through a human-computer interaction interface on a terminal device in the item query system, the computing device can search for the target item by searching for the image captured by the camera in the user's registered place and querying the log file to obtain the search result of the target item, and output it to the user through text, voice or image, etc. Combined with the query log file, the possibility of finding the target item is increased, which helps to improve the search efficiency of the target item.
[0228] In this application embodiment, steps A22-A24 are operations triggered by the event of target item registration completion, and step A25 is an operation triggered by the event of receiving an item search request sent by a user.
[0229] In some other possible implementations, the first event is receiving an item search request sent by a first user, where the item search request includes the first item information. Fig.11 FIG. 1 is an exemplary flow chart of determining the first item information and the first sensor information in another exemplary embodiment of the item search method disclosed in the present invention. Another exemplary embodiment of the item search method is an embodiment corresponding to the event of receiving an item search request sent by a first user. Fig.11 As shown, in Figure 3 Based on the embodiment shown, in this embodiment, step 21 may include:
[0230] Step 2120, in response to receiving the item search request sent by the first user, obtaining first item information from the item search request, and obtaining sensor information corresponding to at least one registration location registered by the first user in the item query system as first sensor information.
[0231] In some optional examples, the first user can initiate an item search request in at least one form of voice, text, picture, etc. through a human-computer interaction interface of a terminal device in the item query system, such as "Where is my wallet?", "Please help me find my blue work badge", "Please help me find the lipstick in the picture", etc., to request the item query system to search for the item.
[0232] Based on this embodiment, when an item search request sent by the first user is received, the item information to be found and the sensor corresponding to the corresponding registration place can be determined to obtain the image captured by the sensor and search for the item within the corresponding registration place in combination with the item information.
[0233] In some optional examples, in step 2120, obtaining sensor information corresponding to at least one registration place registered by the first user in the item query system as the first sensor information may include: in response to the item search request including the first place information, obtaining sensor information corresponding to the first place information as the first sensor information; or, in response to the item search request not including the place information, obtaining sensor information corresponding to at least one registration place registered by the first user in the item query system as the first sensor information, the at least one registration place may be all the registration places registered by the first user in the item query system, or may be some registration places selected from all the registration places registered by the first user in the item query system in a preset manner, such as selecting in sequence.
[0234] and Fig.11 Corresponding to the illustrated embodiment, in some optional examples, in step 22, in response to the item search request including the first time information, which is the time information used as the search condition, at least one frame of image corresponding to the first time information can be obtained from the historical images collected by the first sensor as the first image, for example, at least one frame of image collected by each sensor in the first sensor at the first time information is obtained as the first image, or at least one frame of image collected by each sensor in the first sensor at at least one adjacent collection time before the first time information is obtained as the first image, etc. In response to the item search request not including the time information, the first image collected by the first sensor is obtained, and the first image can include one or more images collected by each sensor in the first sensor at the most recent time; or, the first sensor can be called to collect images, and each sensor in the first sensor collects one or more images to obtain the first image.
[0235] Fig.12 FIG. 1 is an exemplary flow chart of searching for a first item in another exemplary embodiment of the item search method disclosed in the present invention. Fig.12 As shown, in Figure 3 Based on the illustrated embodiment, in another exemplary embodiment of the object search method, step 23 may include:
[0236] Step 2320, in response to the first item information including the first item image and the first item description information, feature extraction is performed on the first item image to obtain first feature information; and target detection is performed on the first image based on the first item description information to obtain a fourth candidate area.
[0237] In some possible implementations, step 2320 may utilize an OVD algorithm, a visual positioning algorithm, or a VGM that integrates the OVD algorithm and the visual positioning algorithm to perform target detection on the first image based on the first object description information. For details, please refer to the relevant introduction of the above embodiments, which will not be repeated here.
[0238] When this embodiment is implemented using a VGM of an Encoder-Decoder structure, a text encoder can be used to extract features from an input text (e.g., description information of the first object) to obtain text features, an image encoder can be used to extract features from an input image (e.g., image of the first object) to obtain image features, and a visual encoder can be used to extract features from an input image (e.g., first image) to obtain visual features (e.g., fourth candidate feature information). The features extracted by the encoder are output to the decoder, and the decoder outputs an object detection frame of the first object based on the features output by the encoder.
[0239] The fourth candidate region in this embodiment is a candidate region obtained by performing object detection on the first image based on the object description information of the first object. The fourth candidate region may include one candidate region or multiple candidate regions. If the fourth candidate region includes multiple candidate regions, steps 2321-2322 are performed respectively for each candidate region included in the fourth candidate region.
[0240] In step 2320, if no candidate region is obtained by performing target detection on the first image, steps 2321-2322 are not performed, and a first search result in which the first object is not found is obtained.
[0241] Step 2321, extract image features from the fourth candidate region to obtain fourth candidate feature information.
[0242] Step 2322: Based on the similarity between the fourth candidate feature information and the first feature information, determine whether the first image includes the first object image corresponding to the first object information, obtain a first search result, and output the first search result.
[0243] In some optional examples, the specific implementation of step 2322 can refer to the implementation method of the above-mentioned step 2312, which will not be repeated here.
[0244] Based on this embodiment, when the first object information includes the first object image and the first object description information, when target detection is performed from the first image based on the first object description information to obtain the fourth candidate area, it can be considered that a suspected target is detected, and then the first feature information of the first object image is compared with the features of the fourth candidate area to further confirm whether the suspected target is the first object. At the same time, the first object search is performed based on the object description information and image of the first object to determine whether the first image includes an image corresponding to the first object, thereby improving the accuracy of the first object search.
[0245] Optionally, see Fig.12 In another exemplary embodiment of the object search method, step 23 may further include:
[0246] Step 2323: In response to the first item information including the first item description information but not including the first item image, determine whether the registered item information table of the first user includes feature information corresponding to the first item description information.
[0247] In response to the first user's registered item information table including the second characteristic information corresponding to the first item description information, execute steps 2324-2326; otherwise, in response to the first user's registered item information table not including the characteristic information corresponding to the first item description information, execute step 2327.
[0248] Step 2324: Perform object detection on the first image based on the first object description information to obtain a fifth candidate region.
[0249] In some optional examples, step 2324 can be implemented using the OVD algorithm, the visual positioning algorithm, or the VGM that integrates the OVD algorithm and the visual positioning algorithm. For details, please refer to the relevant introduction of the above embodiments, which will not be repeated here.
[0250] The fifth candidate region may include one candidate region or multiple candidate regions. If the fifth candidate region includes multiple candidate regions, steps 2325-2326 are performed for each candidate region included in the fifth candidate region.
[0251] Step 2325: extract image features from the fifth candidate region to obtain fifth candidate feature information.
[0252] Step 2326, based on the similarity between the fifth candidate feature information and the second feature information, determine whether the first image includes the first object image corresponding to the first object information, obtain a first search result, and output the first search result.
[0253] The specific implementation method of step 2326 can refer to the relevant implementation method in the above embodiment, which will not be repeated here.
[0254] Step 2327, performing target detection on the first image based on the first object description information, determining whether the first image includes the first object image corresponding to the first object information based on the obtained sixth candidate area, obtaining a first search result, and outputting the first search result.
[0255] In some optional examples, step 2327 may utilize the OVD algorithm, the visual positioning algorithm, or the VGM that integrates the OVD algorithm and the visual positioning algorithm to perform target detection on the first image based on the first object description information. For details, please refer to the relevant introduction of the above embodiments, which will not be repeated here.
[0256] In step 2327, if no candidate region is obtained by performing object detection on the first image, a first search result indicating that the first object is not found is obtained.
[0257] Based on this embodiment, when the first item information only includes the first item description information, it can be determined whether the first user's registered item information table includes feature information corresponding to the first item description information. If the feature information corresponding to the first item description information is included, the first item description information and the feature information can be combined to simultaneously search for the first item, and determine whether the first image includes an image corresponding to the first item, thereby improving the accuracy of searching for the first item. If the feature information corresponding to the first item description information is not included, the first item search is directly performed based on the first item description information, and it is determined whether the first image includes an image corresponding to the first item, thereby achieving search for the first item.
[0258] Optionally, see Fig.12 In another exemplary embodiment of the object search method, step 23 may further include:
[0259] Step 2328: In response to the first item information including the first item image but not including the first item description information, feature extraction is performed on the first item image to obtain first feature information.
[0260] Step 2329: perform object detection on the first image to obtain a seventh candidate region.
[0261] In operation 2329, any target detection algorithm may be used to perform target detection on the first image to obtain a seventh candidate region.
[0262] The seventh candidate region in this embodiment is a candidate region obtained by performing target detection on the first image. The seventh candidate region may include one candidate region or multiple candidate regions. If the seventh candidate region includes multiple candidate regions, steps 2330-2331 are performed for each candidate region included in the seventh candidate region.
[0263] Step 2330: extract image features from the seventh candidate region to obtain seventh candidate feature information.
[0264] Step 2331, based on the similarity between the seventh candidate feature information and the first feature information, determine whether the first image includes the first object image corresponding to the first object information, obtain a first search result, and output the first search result.
[0265] The specific implementation method of step 2331 can refer to the relevant implementation method in the above embodiment, which will not be repeated here.
[0266] Based on this embodiment, when the first item information only includes the first item image but does not include the first item description information, it can be determined by feature comparison whether the first image includes an image corresponding to the first item, thereby realizing the search for the first item.
[0267] In some possible implementations, after determining whether the first image includes the first item image corresponding to the first item information in steps 2322, 2326, 2327, and 2331, the following may also be included:
[0268] In response to the first image not including the first object image, at least one frame of image is obtained from the historical images collected by the first sensor as the fourth image. In an optional example, at least one frame of image collected by each sensor in the first sensor within a preset time period before the collection moment of the first image can be obtained from the historical images collected by each sensor in the first sensor as the fourth image. For example, 6 frames of images collected by each sensor in the first sensor within 3 hours before the collection moment of the first image are obtained as the fourth image; then, the first object search is performed based on the first object information and the fourth image to obtain a first search result. The specific implementation method can refer to the implementation in the above-mentioned embodiment of the present disclosure, and will not be repeated here.
[0269] Based on this embodiment, when the first object is not found in the first image, the historical cache images collected by each sensor in the first sensor may be called to perform object query to increase the possibility of finding the first object.
[0270] Fig.13 FIG. 2 is another exemplary flow chart of searching for a first item in another exemplary embodiment of the item search method disclosed in the present invention. Fig.13 As shown, in Fig.12 Based on the illustrated embodiment, if the first search result is that the first item is not found, after outputting the first search result, the following may also be included:
[0271] Step 2332: Receive query auxiliary information sent by the first user.
[0272] The query auxiliary information is relevant information used to assist in searching for the first item, that is, prompt information used for searching for the first item, and may include at least one of location information and time information.
[0273] Among them, the time information in the query auxiliary information is the time information used to assist the search, such as around 3 pm, hereinafter referred to as the second time information; the place information in the query auxiliary information is the place information used to assist the search, such as in the car, on the sofa at home, etc., hereinafter referred to as the fourth place information.
[0274] In some optional examples, the first user may send query auxiliary information through the terminal device of the first user in the item query system.
[0275] In response to the query auxiliary information including the second time information and the fourth place information, execute steps 2333-2335; in response to the query auxiliary information including the second time information but not including place information, execute steps 2336-2338; in response to the query auxiliary information including the fourth place information but not including time information, execute steps 2339-2341.
[0276] Step 2333, obtaining a fourth sensor corresponding to the fourth location information.
[0277] Among them, the fourth place corresponds to the fourth place information in at least one registered place registered by the first user in the item query system.
[0278] In step 2333, the sensor identified by the sensor information corresponding to the fourth location information may be obtained from the registered location information table of the first user as the fourth sensor. The fourth sensor may include one or more sensors.
[0279] Step 2334: Acquire at least one frame of image corresponding to the second time information from the images captured by the fourth sensor as the fifth image.
[0280] For example, at least one frame of image captured by each sensor in the fourth sensor at the second time information is obtained as the fifth image, or at least one frame of image captured by each sensor in the fourth sensor at at least one adjacent capture moment before the second time information is obtained as the fifth image, and so on. The embodiments of the present disclosure are not limited to this.
[0281] Step 2335: search for the first item based on the first item information and the fifth image to obtain a second search result.
[0282] The specific implementation method can refer to the implementation method of searching for the first item based on the first item information and the first image in the above embodiments, which will not be described in detail here.
[0283] Step 2336, obtaining a first sensor corresponding to at least one registration location registered by the first user in the item query system.
[0284] The specific implementation method can refer to the implementation method of step 2120 in the above embodiment, which will not be repeated here.
[0285] Step 2337: Acquire at least one frame of image corresponding to the second time information from the images captured by the first sensor as a sixth image.
[0286] For example, at least one frame of image captured by each sensor in the first sensor at the second time information is obtained as the sixth image, or at least one frame of image captured by each sensor in the first sensor at at least one adjacent capture moment before the second time information is obtained as the sixth image, and so on. The embodiments of the present disclosure are not limited to this.
[0287] Step 2338: search for the first item based on the first item information and the sixth image to obtain a second search result.
[0288] The specific implementation method can refer to the implementation method of searching for the first item based on the first item information and the first image in the above embodiments, which will not be described in detail here.
[0289] Step 2339, obtaining the fourth sensor corresponding to the fourth location information.
[0290] The specific implementation method can refer to the implementation method of step 2333 in the above embodiment, which will not be repeated here.
[0291] Step 2340: Acquire, from the images acquired by the fourth sensor, a plurality of frames of images acquired within the first historical time period as the seventh image.
[0292] The first historical time period can be set according to actual needs. When the first time information is included in the item search request, the first historical time period can be a preset time period before the first time information, such as within 1 hour before the first time information; when the first time information is not included in the item search request, the first historical time period can be a preset time period adjacent to the capture time of the first image, such as within 2 hours before the capture time of the first image, and so on. The present disclosed embodiment does not impose any restrictions on this.
[0293] Step 2341: search for the first item based on the first item information and the seventh image to obtain a second search result.
[0294] The specific implementation method can refer to the implementation method of searching for the first item based on the first item information and the first image in the above embodiments, which will not be described in detail here.
[0295] Based on this embodiment, when the first object is not found, the query auxiliary information sent by the user can be used as a prompt to narrow the search scope and focus on searching related historical images that may include the first object, which helps reduce the consumption of computing resources required for the search and increases the possibility of finding the first object.
[0296] In some optional examples, the first search result or the second search result can be sent to the terminal device of the first user in the item query system, and the terminal device displays the first search result or the second search result to the first user through a human-computer interaction interface (as an information output unit).
[0297] Optionally, see Fig.13 After obtaining the second search result, if the second search result is that the first item is found, the second search result can be output. If the first search result or the second search result is that the first item is not found, it can also include:
[0298] Step 2342: Query whether the log file of the item query system includes the event analysis result of the first item.
[0299] In response to the event analysis result including the first item in the log file, step 2343 is executed; in response to the event analysis result not including the first item in the log file, step 2344 is executed.
[0300] Step 2343, determine the destination of the first item based on the event analysis result of the first item, obtain a destination determination result, and output the destination determination result.
[0301] Step 2344, output the second search result.
[0302] Based on this embodiment, when the first item is not found, if the log file includes an event analysis result of the first item, the whereabouts of the first item can be determined based on the event analysis result of the first item, thereby improving the success rate of item search.
[0303] In some possible implementations, in the above Figure 12-13In any of the embodiments shown, if the first search result and the second search result are to find the first object from the corresponding image, then the corresponding first search result and the second search result may include the location information of the first object, and the location information of the first object may include at least one of the following: a corresponding image marked with an object detection frame and / or location coordinates of the first object, and location description information of the first object. The location description information of the first object is used to describe the location of the first object. For example, taking the first search result as an example, if the first search result is to find the first object from the first image, then the first search result may include the location information of the first object, specifically including at least one of the following: a first image marked with an object detection frame and / or location coordinates of the first object, and location description information of the first object (for example, on the sofa in the living room).
[0304] Based on this embodiment, the location information of the first object may be displayed in the form of an image, location description information, or a combination thereof, so that the user may intuitively know the place and / or location where the first object exists.
[0305] Fig.14 FIG. 2 is another exemplary flow chart of searching for a first item in another exemplary embodiment of the item search method disclosed in the present invention. Fig.14 As shown in the above Figure 12-13 On the basis of any of the embodiments shown, after outputting the first search result or the second search result, taking the first search result as an example, the embodiment may further include:
[0306] Step 2345: In response to receiving first feedback information sent by the first user, the first feedback information is used to indicate that the first search result is correct, determine whether the registered item information table of the first user includes the item registration information of the first item.
[0307] In response to the first user's registered item information table including the item registration information of the first item, step 2346 is executed; in response to the first user's registered item information table not including the item registration information of the first item, step 2347 is executed.
[0308] Step 2346: store the first characteristic information as characteristic information of the first item in the item registration information of the first item, and store the first search result as a positive sample in the item registration information of the first item.
[0309] Storing the first characteristic information as the characteristic information of the first item in the item registration information of the first item can enrich the characteristic information of the first item, which helps to improve the probability of successful subsequent item search and the accuracy of the item search result.
[0310] Step 2347, using the first characteristic information as the characteristic information of the first item, using the first search result as the positive sample, and adding the item registration information of the first item to the registered item information table of the first user, the item registration information of the first item includes the first item information, the characteristic information of the first item and the positive sample.
[0311] Step 2348, in response to receiving second feedback information sent by the first user, the second feedback information is used to indicate that the first search result is wrong, determine whether the registered item information table of the first user includes the item registration information of the first item.
[0312] In response to the first user's registered item information table including the item registration information of the first item, step 2349 is executed; in response to the first user's registered item information table not including the item registration information of the first item, step 2350 is executed.
[0313] Step 2349: store the first characteristic information as characteristic information of the first item in the item registration information of the first item, and store the first search result as a negative sample in the item registration information of the first item.
[0314] Storing the first characteristic information as the characteristic information of the first item in the item registration information of the first item can enrich the characteristic information of the first item, which helps to improve the probability of successful subsequent item search and the accuracy of the item search result.
[0315] Step 2350, using the first characteristic information as the characteristic information of the first item, using the first search result as a negative sample, and adding the item registration information of the first item to the registered item information table of the first user, wherein the item registration information of the first item includes the first item information, the characteristic information of the first item, and the negative sample.
[0316] In this embodiment, after the second search result is output, the second search result may also be executed. Fig.14 For specific implementation of steps 2345-2350 in the illustrated embodiment, please refer to Fig.14 The illustrated embodiment is directed to the implementation method of the first search result, which will not be described in detail here.
[0317] Based on this embodiment, after outputting the search result about the first item to the user, feedback information from the user on whether the search result is correct can also be received. If the first item is not registered, the first item can be automatically registered, and the search result can be recorded as positive and negative samples based on the user's feedback information on whether the search result is correct, so as to be used for subsequent searches of the first item, thereby improving the accuracy of subsequent search results for the same item and the probability of successful searches.
[0318] Optionally, in Fig.14 Based on the embodiment shown, the above Figure 11-13 In any embodiment, when searching for the first item based on the first item information and the first image, the item registration information of the first item can be further combined, that is, the first item search is performed based on the first item information, the first image and the item registration information of the first item to obtain a first search result.
[0319] Fig.15 FIG. 2 is another exemplary flow chart of searching for a first item in another exemplary embodiment of the item search method disclosed in the present invention. Fig.15 As shown, in some possible implementations, performing a first item search based on the first item information, the first image, and the item registration information of the first item to obtain a first search result may include:
[0320] Step 2360: Search for the first item based on the first item information, the first image, and the feature information of the first item in the item registration information of the first item to obtain a third search result.
[0321] In step 2360, the feature information of the first object in the object registration information of the first object is used as the feature information corresponding to the first object information, and the first object is searched based on the first object information and the first image.
[0322] If the first item information includes the first item image and the first item description information, in some optional examples, Fig.12 Based on the implementation of the embodiment shown, the feature information of the first item is fused with the first feature information obtained by extracting features from the first item image, and the fused feature is used to replace the first feature information to perform step 2322 to determine whether the first image includes the first item image corresponding to the first item information. Alternatively, in some other optional examples, Fig.12 On the basis of the implementation manner of the illustrated embodiment, in step 2322, based on the similarity between the fourth candidate feature information and the first feature information (hereinafter referred to as the first similarity), and the similarity between the fourth candidate feature information and the feature information of the first item (hereinafter referred to as the second similarity), according to a preset standard, for example, whether the first similarity and the second similarity are both greater than the second preset threshold, whether the average value of the first similarity and the second similarity is greater than the second preset threshold, etc., it is determined whether the first image includes the first item image corresponding to the first item information. When the first similarity and the second similarity are both greater than the second preset threshold, or when the average value of the first similarity and the second similarity is greater than the second preset threshold, it can be determined that the first image includes the first item image; otherwise, it is determined that the first image does not include the first item image corresponding to the first item information.
[0323] If the first item information includes the first item description information but does not include the first item image, the feature information of the first item can be used as the second feature information corresponding to the first item description information in the first user's registered item information table. Fig.12 In the illustrated embodiment, steps 2323 - 2326 determine whether the first image includes the first object image corresponding to the first object information.
[0324] If the first item information includes the first item image but does not include the first item description information, in some optional examples, Fig.12 Based on the implementation method shown in steps 2328-2331 in the illustrated embodiment, the feature information of the first item is fused with the first feature information obtained by extracting features from the first item image, and the fused feature is used to replace the first feature information to perform step 2331 to determine whether the first image includes the first item image corresponding to the first item information. Alternatively, in some other optional examples, Fig.12 On the basis of the implementation manner shown in steps 2328-2331 in the illustrated embodiment, in step 2331, based on the similarity between the seventh candidate feature information and the first feature information (hereinafter referred to as the third similarity), and the similarity between the seventh candidate feature information and the feature information of the first item (hereinafter referred to as the fourth similarity), according to a preset standard, for example, whether the third similarity and the fourth similarity are both greater than the second preset threshold, whether the average of the third similarity and the fourth similarity is greater than the second preset threshold, etc., it is determined whether the first image includes the first item image corresponding to the first item information. When the third similarity and the fourth similarity are both greater than the second preset threshold, or when the average of the third similarity and the fourth similarity is greater than the second preset threshold, it may be determined that the first image includes the first item image; otherwise, it is determined that the first image does not include the first item image corresponding to the first item information.
[0325] Step 2361: confirm the third search result based on the sample information in the item registration information of the first item to obtain the first search result.
[0326] The sample information includes at least one of a positive sample and a negative sample.
[0327] In step 2361, if the third search result is that the first object is found from the first image, and the object registration information of the first object includes a positive sample, whether the third search result is correct can be determined based on whether the similarity between the third search result and the positive sample is greater than a third preset threshold. The third search result is determined to be correct only when the similarity between the third search result and the positive sample is greater than the third preset threshold; if correct, the third search result is output as the first search result; if incorrect, the first search result in which the first object is not found is output.
[0328] If the third search result is that the first object is found from the first image, and the item registration information of the first object includes a negative sample, whether the third search result is correct can be determined based on whether the similarity between the third search result and the negative sample is less than a fourth preset threshold. The third search result is determined to be correct only when the similarity between the third search result and the negative sample is less than the fourth preset threshold; if correct, the third search result is output as the first search result; if incorrect, the first search result in which the first object is not found is output.
[0329] If the third search result is that the first object is found from the first image, and the object registration information of the first object includes both positive samples and negative samples, it can be determined whether the third search result is correct based on whether the similarity between the third search result and the positive sample is greater than the third preset threshold, and whether the similarity between the third search result and the negative sample is less than the fourth preset threshold; if it is correct, the third search result is output as the first search result; if it is incorrect, the first search result in which the first object is not found is output. In some optional examples, the third search result can be determined to be correct when the similarity between the third search result and the positive sample is greater than the third preset threshold, and the similarity between the third search result and the negative sample is less than the fourth preset threshold, otherwise, the third search result is determined to be incorrect; or, in other optional examples, the third search result can also be determined to be correct when the similarity between the third search result and the positive sample is greater than the third preset threshold, or the similarity between the third search result and the negative sample is less than the fourth preset threshold; the embodiments of the present disclosure do not limit this specific standard.
[0330] In this embodiment, the specific implementation method of determining the similarity between the third search result and the positive sample, and the similarity between the third search result and the negative sample can be implemented by referring to the method shown in any of the above embodiments, which will not be repeated here.
[0331] Based on this embodiment, item search can be performed in combination with feature information in item registration information, and the search result can be confirmed based on sample information in the item registration information, thereby improving the accuracy of item search results and the probability of successful search.
[0332] Fig.16 FIG. 1 is a schematic diagram of an application flow of another exemplary embodiment of the method for searching for items in the embodiment of the present disclosure. This application embodiment can realize searching for items. Fig.16 As shown above Figure 11-Figure 15 An application of another exemplary embodiment of the object search method shown, taking the target object (corresponding to the first object and the second object in the above embodiment) as lipstick and the sensor as a camera as an example, the application embodiment includes:
[0333] A31, user sends an item search request: the user takes an image of the target item (lipstick) through a camera on a terminal device (such as a mobile terminal or IVI) in the item search system, uploads the image of the lipstick through the human-computer interaction interface of the terminal device, and initiates an item search request in the form of a picture combined with voice or text, such as "please help me find the lipstick in the picture", and the item search request includes the image of the lipstick and the item description information "lipstick". If the user initiates the item search request in the form of voice, the terminal device can convert the voice into text by voice recognition, which will not be described in detail here; the terminal device uploads the item search request to the computing device in the item search system;
[0334] A32, image acquisition: the computing device acquires images from cameras in all the places registered by the user in the item query system (assuming that the places include the living room, balcony and car cabin) to obtain at least one acquired image; if the user carries the location information of the designated place in the item search request, for example, the item search request is "seek if my lipstick has fallen into the car)", the computing device acquires images from cameras in the car cabin of the place registered by the user in the item query system to obtain at least one acquired image;
[0335] A33, target object search: the computing device sequentially traverses each captured image to search for the target object. Specifically, the computing device calls VGM to perform feature extraction on the object search request. For example, the text features of the object description information are extracted through the text encoder in the VGM, and the image features of the target object are extracted through the image encoder in the VGM. The text features and image features are input into the decoder in the VGM as prompt information (prompt); the visual features of each captured image are extracted through the visual encoder and input into the decoder; the decoder performs target detection based on the prompt information and visual features to obtain an object detection frame, and compares the text features and image features with the features of the object detection frame to determine whether each captured image includes the target object image. In this step, if the target object is a registered object, the target object search can be performed in combination with the object registration information of the target object, which will not be repeated here;
[0336] If the target object image is not found in all the collected images, execute step A34; if the target object image is found in any collected image, execute step A39;
[0337] A34, historical image search: If the target object image is not found in all the collected images, the computing device calls the historical cache images corresponding to the corresponding place (all the registered places or designated places in step A32) (for example, a total of 6 images collected by the camera corresponding to the corresponding place within the previous 3 hours), and searches for the target object in the manner of step A33;
[0338] If the target object image is still not found from all historical cache images, execute step A35; if the target object image is still not found from any historical image, execute step A39;
[0339] A35, log file query: if the target object image is still not found from all historical cache images, the computing device queries whether the log file includes the event analysis result of the target object; if the log file includes the event analysis result of the first object, the destination of the target object is determined based on the event analysis result, and the destination determination result is obtained, and then step A36 is executed; if the log file does not include the event analysis result of the first object, step A37 is executed;
[0340] A36, outputting the destination determination result: the computing device returns the destination determination result of the target item to the human-computer interaction interface of the terminal device;
[0341] After that, the subsequent steps of this embodiment are no longer executed.
[0342] A37, target item not found: if the log file does not include the event analysis result of the first item, the computing device returns a null value or a search result message indicating that the target item was not found to the human-computer interaction interface of the terminal device;
[0343] A38, user prompt search: in response to receiving the query auxiliary information sent by the user human-computer interaction interface, such as "I used it in the living room at 2 pm", the computing device obtains the historical images collected by the camera corresponding to the place information "living room" in the query auxiliary information, obtains the two frames of images corresponding to the time information "2 pm" in the query auxiliary information, and searches for the target object in the manner of step A33;
[0344] If the target object image is still not found from the historical images based on the query auxiliary information, execute step A37; if the target object image is found from any historical image based on the query auxiliary information, execute step A39;
[0345] A39, output search results: if the target object image is found from any of the collected images, the computing device returns the search results of the target object to the human-computer interaction interface of the terminal device, including the location description information of the target object (for example, the lipstick is on the sofa in the living room) and the image with the object detection frame marked with the target object, and the terminal device displays the search results to the user in at least one form such as text, voice, and image;
[0346] A40, user feedback on whether the search result is correct: the user sends feedback information on whether the search result is correct (ie, the first feedback information or the second feedback information) through the human-computer interaction interface on the terminal device;
[0347] A41, item registration information enhancement: If the target item is not registered, the computing device automatically registers the target item and generates the item registration information of the target item. If the user feedback indicates that the search result is correct, the query result is stored as a positive sample in the item registration information of the target item, and the image feature of the target item is added to the item registration information of the target item for subsequent target item search; if the user feedback indicates that the search result is incorrect, the computing device stores the query result as a negative sample in the item registration information of the target item, and the image feature of the target item is added to the item registration information of the target item for subsequent target item search.
[0348] In some other possible implementations, the first event is detecting that new item registration information is added to the registered item information table of the first user. Fig.17 1 is an exemplary flow chart of searching for a first item in another exemplary embodiment of the item search method disclosed in the present invention. Another exemplary embodiment of the item search method is an embodiment corresponding to the event of detecting that new item registration information is added to the first user's registered item information table. Fig.17 As shown, in this embodiment, Figure 3 Based on the illustrated embodiment, step 21 may include:
[0349] Step 2130, in response to detecting that new item registration information is added to the registered item information table of the first user, the new item registration information is obtained as the first item information, and sensor information corresponding to at least one registration location registered by the first user in the item query system is obtained as the first sensor information.
[0350] For details, please refer to the exemplary implementation methods corresponding to the above step 21, which will not be described again here.
[0351] and Fig.17 Correspondingly to the illustrated embodiment, in some optional examples, in step 22, a first image captured by the first sensor can be obtained, and the first image can include one or more images most recently captured by each sensor in the first sensor; or, each sensor in the first sensor can be called to perform image capture, and each sensor can capture one or more images respectively to obtain a first image.
[0352] Fig.18 FIG. 1 is an exemplary flow chart of determining first item information and first sensor information in another exemplary embodiment of the object search method disclosed in the present invention. Fig.18 As shown, in Figure 3 Based on the illustrated embodiment, in another exemplary embodiment of the object search method, step 23 may include:
[0353] Step 2370: Search for the first object based on the newly added object registration information and the first image to determine the sixth location where the first object is located, and obtain a first search result.
[0354] Among them, the newly added item registration information includes item information of the first item. In step 2370, the first item can be searched based on the item information in the newly added item registration information and the first image. The specific implementation method can refer to the implementation method of searching the first item based on the first item information and the first image in the above-mentioned embodiments, which will not be repeated here.
[0355] In step 2370, after determining the first image including the image corresponding to the first object, the first image including the image corresponding to the first object is used as the target image, and the place corresponding to the sensor that captured the target image can be determined as the sixth place where the first object is located.
[0356] Optionally, see Fig.18 In some possible implementations, after obtaining the first search result in step 2370, the following steps may also be included:
[0357] Step 2371, according to the fourth cycle, call the sixth sensor corresponding to the sixth field to perform image acquisition, and based on the acquired fourth image sequence and the newly added object registration information, perform target detection and position tracking on the first object to obtain a second position tracking result.
[0358] The fourth image sequence may include a plurality of frames of images acquired by the sixth sensor according to the second cycle and arranged based on the image acquisition time sequence.
[0359] The fourth period can be set according to actual needs and can be updated as needed. In actual applications, for items with frequent location changes (such as mobile phone terminals), a smaller fourth period (such as 2 minutes) can be set to achieve timely tracking of the items; and for items with less frequent location changes (such as wallets), a larger fourth period (such as 2 minutes) can be set, and so on. The embodiments of the present disclosure do not limit this.
[0360] After determining the sixth place where the first object is located, through step 2371, the sixth sensor corresponding to the sixth place can be specifically called to collect images according to the fourth cycle, and based on the object information in the object registration information of the first object, an arbitrary object detection algorithm is used to perform target detection on the image collected by the third sensor to track the position of the first object according to the second cycle. The first position tracking result can be a first image sequence marked with an object detection frame of the first object.
[0361] Among them, the implementation method of performing target detection on the image collected by the third sensor based on the item information of the first item can refer to the implementation method of searching for the first item based on the first item information and the first image in the above embodiments, which will not be repeated here.
[0362] Step 2372, based on the second position tracking result, in response to a change in the position of the first object, according to the fifth cycle, call the sixth sensor to perform image acquisition, and based on the acquired fifth image sequence and the newly added object registration information, perform target detection and position tracking on the first object.
[0363] The fifth period is shorter than the fourth period. The fifth period can be set according to actual needs, for example, can be set to 10 seconds, and can be updated as needed.
[0364] The fifth image sequence may include multiple frames of images collected by the sixth sensor according to the fifth period and arranged based on the image collection time sequence. Through this step 2372, when the position of the first object changes, the position tracking period of the first object can be reduced and the position tracking frequency can be increased to achieve effective tracking of the position of the first object. For details, please refer to the implementation method of step 2371, which will not be repeated here.
[0365] Step 2373, in response to not detecting the first object in the most recent frame of image acquired from the sixth visual sensor, acquiring a sixth image sequence.
[0366] The sixth image sequence includes: at least one frame of image acquired by the sixth visual sensor and acquired earlier than the most recent frame of image.
[0367] Step 2374: Use the analysis model to perform event analysis on the sixth image sequence to obtain an event analysis result of the first object, and store the event analysis result of the first object in a log file of the object query system.
[0368] The event analysis results may include the possible destination of the first object (for example, put into a bag) or the possible location (for example, on the sofa), and may also include a sixth image sequence marked with an object detection frame of the first object, which is not limited in this embodiment of the present disclosure.
[0369] The analysis model may be a pre-trained deep learning model. The specific implementation of the deep learning model and the method of performing event analysis on the sixth image sequence may refer to the above step 56 and will not be described in detail here.
[0370] The specific implementation of each step in this embodiment can refer to the above Fig. 9 The relevant introduction of the illustrated embodiment will not be repeated here.
[0371] Based on this embodiment, the positioning and position tracking of the registered item is realized. When the registered item is not visible, event analysis can be performed on the image sequence before the registered item becomes visible, and the obtained event analysis results can be stored in a log file. When the registered item is not found subsequently, the possible whereabouts or possible location of the registered item can be determined based on the event analysis results, thereby increasing the possibility of finding the item, thereby improving the efficiency and effect of item search.
[0372] In some possible implementations, the item search method of any embodiment of the present disclosure can be implemented by an Internet of Things system configured as an item query system in any of the above embodiments of the present disclosure, and can be specifically implemented by a computing device in the item query system. The item query system includes at least one sensor 11 and at least one end-side device 12, wherein at least one sensor 11 is deployed in a location corresponding to the item query system, and at least one end-side device 12 includes one or more terminal devices that a first user or family has permission to use, and the first end-side device 120 in the at least one end-side device 12 is configured as a computing device for implementing the item search method embodiment of the present disclosure, and the computing device may also be referred to as an item search device.
[0373] For the specific implementation of the places, sensors and terminal devices corresponding to the item query system, please refer to the relevant introduction of the above-mentioned Internet of Things system embodiment, which will not be repeated here.
[0374] In each embodiment of the method for searching for items implemented by the item query system of this embodiment, the user can send relevant information to the computing device through the human-computer interaction interface (as an information input unit) of each terminal device in the item query system, and receive feedback information sent by the computing device through the human-computer interaction interface (as an information output unit) of the terminal device, or can also receive relevant information sent by the computing device through other terminal devices in the item query system, and the embodiment of the disclosure does not limit this. As a result, the data involved in the item query process is only circulated between private devices within the scope of the family, without having to be uploaded to the cloud, which can effectively ensure the privacy and security of users.
[0375] Exemplary Devices
[0376] Fig.19 is a schematic diagram of the structure of an exemplary embodiment of the computing device of the present disclosure. The computing device of the present embodiment can be used to implement the above-mentioned various object search method embodiments of the present disclosure, and therefore can also be referred to as an object search device. Fig.19 As shown, the computing device of this embodiment includes:
[0377] The first determination module 61 is used to determine the first item information and the first sensor information corresponding to the first event. The first sensor information includes sensor information of the first sensor, and the first sensor includes at least one sensor corresponding to a registered place, wherein the registered place is a place registered in the item query system.
[0378] The first acquisition module 62 is used to acquire a first image captured by a first sensor.
[0379] The object search module 63 is used to perform a first object search based on the first object information and the first image to obtain a first search result.
[0380] In some possible implementations, the first event is the first user leaving the first place. In this implementation, the first determination module 61 is specifically used to: in response to detecting that the first user has left the first place, obtain the alarm item information corresponding to the first place as the first item information, and obtain the sensor information corresponding to the first place as the first sensor information. The first place is one of the registered places registered by the first user in the item query system, and the alarm item information includes information about at least one alarm item, and the alarm item is an item of the first user that is not allowed to be left in the corresponding registered place. Accordingly, in this embodiment, the first acquisition module 62 is specifically used to: obtain the first image captured by the first sensor; or, call the first sensor to perform image capture to obtain the first image.
[0381] Fig. 20 FIG. 1 is a schematic diagram of another exemplary embodiment of the computing device disclosed in the present invention. Fig. 20 As shown, corresponding to the above implementation, in this embodiment, the computing device may further include: a second acquisition module 64, used to acquire a second image collected by the first sensor according to a first period; a detection module 65, used to perform a first detection on the first user based on the second image, and determine whether the first user leaves the first place based on the first detection result obtained.
[0382] Optionally, see Fig. 20 In some possible implementations, the object search module 63 may include: a first target detection unit 6310, a first feature extraction unit 6311 and a first determination unit 6312.
[0383] In some optional examples, the first target detection unit 6310 is used to perform target detection on the first image based on the item description information of the first item in response to the first item information including item description information and feature information of the first item, and obtain a first candidate area; the first feature extraction unit 6311 is used to perform image feature extraction on the first candidate area to obtain first candidate feature information; the first determination unit 6312 is used to determine whether the first image includes an image corresponding to the first item based on the similarity between the first candidate feature information and the feature information of the first item, and obtain a first search result.
[0384] In some other optional examples, the first target detection unit 6310 is used to perform target detection on the first image based on the item description information of the first item in response to the first item information including the item description information of the first item but not including the feature information of the first item, so as to obtain a second candidate area; the first determination unit 6312 is used to obtain a first search result based on the second candidate area.
[0385] In some other optional examples, the first target detection unit 6310 is used to perform target detection on the first image in response to the first item information including feature information of the first item but not including item description information of the first item, so as to obtain a third candidate area; the first feature extraction unit 6311 is used to perform image feature extraction on the third candidate area to obtain third candidate feature information; and the first determination unit 6312 is used to determine whether the first image includes an image corresponding to the first item based on the similarity between the third candidate feature information and the feature information of the first item, so as to obtain a first search result.
[0386] Optionally, see Fig. 20 In some other possible implementations, the computing device may further include: an alarm module 66, configured to send an alarm message to the first user in response to the first search result being that the first item is found, to prompt that the first item is left in the first place.
[0387] Optionally, in some possible implementations, the computing device is applied to an item query system, and the item query system includes at least one of a registration location information table and a registered item information table of a first user. The registration location information table of the first user includes the registration location information of at least one registration location registered by the first user in the item query system, and the registration location information of each registration location includes the location information of each registration location and the sensor information corresponding to each registration location; the registered item information table of the first user includes the item registration information of at least one registered item registered by the first user in the item query system, and the item registration information of each registered item includes the item information of each registered item and a first location list; the item information of each registered item includes at least one of an image and item description information of each registered item, and when the item information of each registered item includes an image, the item registration information of each registered item also includes feature information obtained by extracting features from the image; the first location list of each registered item includes the location information of a second location where each registered item is allowed to be left behind; the alarm item information corresponding to the first location is determined based on the registration location information table of the first user and each first location list in the registered item information table.
[0388] Fig.21 FIG. 1 is a schematic diagram of a structure of another exemplary embodiment of a computing device disclosed in the present invention. Fig.21As shown, the computing device of this embodiment may further include: a tracking module 67 and an analysis model 68. The first acquisition module 62 is further used to acquire the third image collected by the second sensor, and the second sensor includes a sensor corresponding to the sensor information in the registration location information table of the first user. The item search module 63 is further used to search for the first item based on the item registration information of the first item and the third image to determine the third location where the first item is located. The first acquisition module 62 is further used to call the third sensor corresponding to the third field to perform image acquisition according to the second cycle. The tracking module 67 is used to perform target detection and position tracking on the first item based on the first image sequence collected by the third sensor and the item registration information of the first item to obtain a first position tracking result. Based on the first position tracking result, in response to a change in the position of the first item, the first acquisition module 62 is further used to call the third sensor to perform image acquisition according to the third cycle. The tracking module 68 is further used to perform target detection and position tracking on the first item based on the second image sequence collected by the third sensor and the item registration information of the first item, wherein the third cycle is smaller than the second cycle. In response to the first object not being detected in the most recent frame of image acquired by the third sensor, the first acquisition module 62 is further configured to acquire a third image sequence, the third image sequence comprising: at least one frame of image acquired by the third sensor and acquired earlier than the most recent frame of image. The analysis model 68 is configured to perform event analysis on the third image sequence, obtain an event analysis result of the first object, and store the event analysis result of the first object in a log file of the object query system.
[0389] In some other possible implementations, the first event is receiving an item search request sent by a first user, and the item search request includes the first item information. In this implementation, the first determination module 61 is specifically used to: in response to receiving the item search request sent by the first user, obtain the first item information from the item search request, and obtain the sensor information corresponding to at least one registered place registered by the first user in the item query system as the first sensor information. For example, in response to the item search request including the first place information, obtain the sensor information corresponding to the first place information as the first sensor information; or, in response to the item search request not including the place information, obtain the sensor information corresponding to at least one registered place registered by the first user in the item query system as the first sensor information.
[0390] The first acquisition module 62 is specifically used to: in response to the item search request including the first time information, acquire at least one frame of image corresponding to the first time information from the historical images captured by the first sensor as the first image; in response to the item search request not including the time information, acquire the first image captured by the first sensor, or call the first sensor to perform image capture to obtain the first image.
[0391] Fig. 22 FIG. 1 is a schematic diagram of a structure of another exemplary embodiment of a computing device disclosed in the present invention. Fig. 22 As shown, corresponding to the above implementation, in this embodiment, the object search module 63 may include: a second target detection unit 6320, a second feature extraction unit 6321 and a second determination unit 6322.
[0392] In some optional examples, the second target detection unit 6320 is used to perform target detection on the first image based on the first item description information in response to the first item information including the first item image and the first item description information, to obtain a fourth candidate area; the second feature extraction unit 6321 is used to perform feature extraction on the first item image to obtain first feature information, and to perform image feature extraction on the fourth candidate area to obtain fourth candidate feature information; the second determination unit 6322 is used to determine whether the first image includes the first item image corresponding to the first item information based on the similarity between the fourth candidate feature information and the first feature information, to obtain a first search result, and to output the first search result.
[0393] In some other optional examples, the item search module 63 may also include a third determination unit 6323, which is used to determine whether the first user's registered item information table includes feature information corresponding to the first item description information in response to the first item information including the first item description information but not including the first item image; a second target detection unit 6320, which is used to perform target detection on the first image based on the first item description information in response to the first user's registered item information table including second feature information corresponding to the first item description information, to obtain a fifth candidate area; a second feature extraction unit 6321, which is used to perform image feature extraction on the fifth candidate area to obtain fifth candidate feature information; and a second determination unit 6322, which is used to determine whether the first image includes the first item image corresponding to the first item information based on the similarity between the fifth candidate feature information and the second feature information, to obtain a first search result, and output the first search result. The second target detection unit 6320 is further used to perform target detection on the first image based on the first item description information in response to the fact that the registered item information table of the first user does not include feature information corresponding to the first item description information; the second determination unit 6322 is used for the second target detection unit 6320 to determine whether the first image includes the first item image corresponding to the first item information based on the obtained sixth candidate area, obtain a first search result, and output the first search result.
[0394] In some further optional examples, the second feature extraction unit 6321 is used to perform feature extraction on the first object image to obtain first feature information in response to the first object information including the first object image but not including the first object description information; the second target detection unit 6320 is used to perform target detection on the first image to obtain a seventh candidate area; the second feature extraction unit 6321 is also used to perform image feature extraction on the seventh candidate area to obtain seventh candidate feature information; the second determination unit 6322 is used to determine whether the first image includes the first object image corresponding to the first object information based on the similarity between the seventh candidate feature information and the first feature information, to obtain a first search result, and output the first search result.
[0395] Optionally, in Fig. 22 On the basis of any of the embodiments shown, the first acquisition module 62 can also be used to obtain at least one frame of image from the historical image collected by the first sensor as the fourth image in response to the first image not including the first object image, based on the determination result of the second determination unit 6322 on whether the first image includes the first object image corresponding to the first object information. The object search module 63 can also be used to search for the first object based on the first object information and the fourth image to obtain a first search result.
[0396] Fig.23 FIG. 1 is a schematic diagram of a structure of another exemplary embodiment of the computing device disclosed in the present invention. Fig.23 As shown, in Fig. 22 On the basis of any of the embodiments shown, in this embodiment, the computing device may further include: a receiving module 69, a third acquisition module 70 and a fourth acquisition module 71. After the first search result is that the first item is not found, and the item search module 63 outputs the first search result, in some optional examples, the receiving module 69 is used to receive the query auxiliary information sent by the first user; the third acquisition module 70 is used to obtain the fourth sensor corresponding to the fourth location information in response to the query auxiliary information including the second time information and the fourth location information, and at least one registered location registered by the first user in the item query system includes the fourth location corresponding to the fourth location information; the fourth acquisition module 71 is used to obtain at least one frame of image corresponding to the second time information from the image captured by the fourth sensor as the fifth image; the item search module 63 is also used to search for the first item based on the first item information and the fifth image to obtain the second search result.
[0397] In other optional examples, the third acquisition module 70 is used to obtain a first sensor corresponding to at least one registered place registered by the first user in the item query system in response to the query auxiliary information including the second time information but not including the place information; the fourth acquisition module 71 is used to obtain at least one frame of image corresponding to the second time information from the image captured by the first sensor as the sixth image; the item search module 63 is also used to perform a first item search based on the first item information and the sixth image to obtain a second search result.
[0398] In some other optional examples, the third acquisition module 70 is used to obtain the fourth sensor corresponding to the fourth place information in response to the query auxiliary information including the fourth place information but not the time information; the fourth acquisition module 71 is used to obtain a plurality of frames of images captured within the first historical time period as the seventh image from the images captured by the fourth sensor; the object search module 63 is also used to perform a first object search based on the first object information and the seventh image to obtain a second search result.
[0399] Optionally, see Fig.23 In a further embodiment, the computing device may further include: a query module 72 and a second determination module 73. If the second search result is that the first item is not found, after the item search module 63 obtains the second search result, the query module 72 is used to query whether the log file of the item query system includes the event analysis result of the first item. The second determination module 73 is used to determine the whereabouts of the first item based on the event analysis result of the first item according to the query result of the query module 72, in response to the event analysis result of the first item included in the log file, obtain the whereabouts determination result, and output the whereabouts determination result; in response to the event analysis result of the first item not included in the log file, output the second search result.
[0400] In some possible implementations, when the first search result is that the first object is found from the first image, the first search result may include location information of the first object, and the location information of the first object may include but is not limited to at least one of the following: a first image marked with an object detection frame and / or location coordinates of the first object, and location description information of the first object.
[0401] Optionally, see Fig.23In a further embodiment, the computing device may further include: a third determination module 74 and a storage module 75. After the item search module 63 outputs the first search result, in some optional examples, the third determination module 74 is used to determine whether the first user's registered item information table includes the item registration information of the first item in response to receiving the first feedback information sent by the first user, the first feedback information being used to indicate that the first search result is correct. The storage module 75 is used to store the first characteristic information as the characteristic information of the first item in the item registration information of the first item in response to the first user's registered item information table including the item registration information of the first item, and store the first search result as a positive sample in the item registration information of the first item; in response to the first user's registered item information table not including the item registration information of the first item, use the first characteristic information as the characteristic information of the first item, use the first search result as a positive sample, and add the item registration information of the first item to the first user's registered item information table, the item registration information of the first item including the first item information, the characteristic information of the first item, and the positive sample.
[0402] In some other optional examples, the third determination module 74 is used to determine whether the first user's registered item information table includes the item registration information of the first item in response to receiving the second feedback information sent by the first user, the second feedback information is used to indicate that the first search result is wrong. The storage module 75 is used to store the first characteristic information as the characteristic information of the first item in the item registration information of the first item in response to the first user's registered item information table including the item registration information of the first item, and store the first search result as a negative sample in the item registration information of the first item; in response to the first user's registered item information table not including the item registration information of the first item, use the first characteristic information as the characteristic information of the first item, use the first search result as a negative sample, add the item registration information of the first item to the first user's registered item information table, and the item registration information of the first item includes the first item information, the characteristic information of the first item, and the negative sample.
[0403] In some possible implementations, the object search module 63 is specifically configured to perform a first object search based on the first object information, the first image, and the object registration information of the first object to obtain a first search result.
[0404] For example, in some optional examples, the object search module 63 is specifically used to search for the first object based on the first object information, the first image, and the characteristic information of the first object in the object registration information of the first object to obtain a third search result; based on the sample information in the object registration information of the first object, the third search result is confirmed to obtain the first search result; wherein the sample information includes at least one of a positive sample and a negative sample.
[0405] In some other possible implementations, the first event is detecting that new item registration information is added to the registered item information table of the first user. In this implementation, the first determination module 61 is specifically configured to: in response to detecting that new item registration information is added to the registered item information table of the first user, obtain the new item registration information as the first item information, and obtain sensor information corresponding to at least one registered place registered by the first user in the item query system as the first sensor information.
[0406] Correspondingly, the item search module 63 is specifically configured to search for the first item based on the newly added item registration information and the first image, so as to determine the sixth location where the first item is located, and obtain a first search result.
[0407] The exemplary embodiments of the item search method, computing device, Internet of Things system, and item query system disclosed in the present invention correspond to each other, and can refer to and quote each other in the corresponding implementation methods, and will not be repeated here. When the Internet of Things system provided by the exemplary embodiment of the present invention is configured as an item query system, one of the terminal devices in the first end-side device 120 may include the computing device provided by the exemplary embodiment of the present invention; or, one of the terminal devices in the first end-side device 120 is configured as the computing device provided by the exemplary embodiment of the present invention, for implementing the item search method provided by the exemplary embodiment of the present invention.
[0408] The beneficial technical effects corresponding to the exemplary embodiments of the disclosed object search method, computing device, Internet of Things system and object query system can also be referenced to each other and will not be repeated here.
[0409] Exemplary Electronic Devices
[0410] An embodiment of the present disclosure further provides an electronic device, which includes: a processor; a memory for storing processor executable instructions; wherein the processor is used to read the executable instructions from the memory and execute the instructions to implement the object search method of any of the above embodiments of the present disclosure.
[0411] The electronic device can serve as the first terminal device in any of the above exemplary embodiments or be configured as a computing device in any of the above exemplary embodiments to execute the object search method in any of the above embodiments of the present disclosure.
[0412] Fig.24 A structural diagram of an electronic device provided in an embodiment of the present disclosure includes at least one processor 81 and a memory 82.
[0413] The processor 81 may be a central processing unit (CPU) or other forms of processing units with data processing capabilities and / or instruction execution capabilities, and may control other components in the electronic device to perform desired functions. The processor 81 may be used as a computing resource to provide computing capabilities for the computing device to implement the object search method of any of the above embodiments.
[0414] The memory 82 may include one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. Volatile memory may include, for example, random access memory (RAM) and / or cache memory (cache), etc. Non-volatile memory may include, for example, read-only memory (ROM), hard disk, flash memory, etc. The memory 82 may also be used to store at least part of the information related to sensor acquisition data (such as images acquired by at least one sensor 11), registration location information table, registration item information table, alarm item information, log file, registered user information table, item query system usage authority information, etc. involved in the item query system. One or more computer program instructions may be stored on the computer-readable storage medium, and the processor 81 may execute one or more computer program instructions to implement the item search method and / or other desired functions of the various embodiments of the present disclosure described above.
[0415] In one example, the electronic device may further include: an input device 83 and an output device 84, and these components are interconnected via a bus system and / or other forms of connection mechanisms (not shown).
[0416] The input device 83 can receive various forms of information input by the user, which may include, for example, a keyboard, a mouse, a touch screen, a microphone, a camera or other human-computer interaction interfaces, etc., and is used to collect various information input by the user in the form of text, voice, pictures, etc., such as but not limited to: item search requests, query auxiliary information, feedback information about the search results, configuration information of the item query system (such as sensor information, terminal device information, and usage permission information of the item query system included in the item query system), various registration information (such as registration location information, item registration information, registered user information, etc.), and various setting information (such as location whitelist, non-location whitelist, etc.).
[0417] The output device 84 can output various information or signals to the outside, which may include, for example, a display, a speaker, a seat, a window, a steering wheel, a vibration sensor, or other human-machine interaction interfaces, as well as a communication network and its connected remote output device, etc.
[0418] Of course, to simplify, Fig.24Only some of the components related to the present disclosure in the electronic device are shown, and components such as a bus, an input / output interface, etc. are omitted. In addition, according to specific application situations, the electronic device may further include any other appropriate components.
[0419] Exemplary computer program products and computer-readable storage media
[0420] In addition to the above-mentioned methods and devices, embodiments of the present disclosure may also provide a computer program product, including computer program instructions, which, when executed by a processor, enable the processor to execute the steps of the item search method of various embodiments of the present disclosure described in the above-mentioned "Exemplary Method" section.
[0421] The computer program product may be written in any combination of one or more programming languages to write program codes for executing the steps of the disclosed embodiments, including object-oriented programming languages such as Java, C++, etc., and conventional procedural programming languages such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server.
[0422] In addition, an embodiment of the present disclosure may also be a computer-readable storage medium having computer program instructions stored thereon. When the computer program instructions are executed by a processor, the processor executes the steps of the object search method of various embodiments of the present disclosure described in the above “Exemplary Method” section.
[0423] Computer readable storage media can adopt any combination of one or more readable media. The readable medium can be a readable signal medium or a readable storage medium. The readable storage medium is, for example, but not limited to, a system, device or device including electricity, magnetism, light, electromagnetic, infrared, or semiconductor, or any combination of the above. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.
[0424] The basic principles of the present disclosure are described above in conjunction with specific embodiments. However, the advantages, strengths, effects, etc. mentioned in the present disclosure are only examples and not limitations, and cannot be considered as necessary for each embodiment of the present disclosure. In addition, the specific details disclosed above are only for the purpose of illustration and ease of understanding, rather than limitation, and the above details do not limit the present disclosure to being implemented by adopting the above specific details.
[0425] Those skilled in the art may make various changes and modifications to the present disclosure without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the present disclosure claims and their equivalents, the present disclosure is also intended to include these modifications and variations.
Claims
1. An item search method, applied to an item query system, comprising: Determine first item information and first sensor information corresponding to the first event; wherein the first sensor information includes sensor information of a first sensor, and the first sensor includes a sensor corresponding to at least one registered place, and the registered place is a place registered in the item query system; Acquire a first image captured by the first sensor; A first item search is performed based on the first item information and the first image to obtain a first search result.
2. The method according to claim 1, wherein: The first event includes a first user leaving a first location; Determining first object information and first sensor information corresponding to a first event includes: In response to detecting that the first user leaves the first place, alarm item information corresponding to the first place is obtained as the first item information, and sensor information corresponding to the first place is obtained as the first sensor information; wherein, the first place is one of the at least one registered places registered by the first user in the item query system, the alarm item information includes information of at least one alarm item, and the alarm item is an item registered by the first user and is not allowed to be left at the corresponding registered place.
3. The method according to claim 2, wherein: In response to detecting that the first user leaves the first place, the method further includes: According to a first cycle, acquiring a second image captured by the first sensor; A first detection is performed on the first user based on the second image, and it is determined whether the first user leaves the first place based on the obtained first detection result.
4. The method according to any one of claims 2-3, wherein: Performing a first item search based on the first item information and the first image to obtain a first search result includes: In response to the first item information including item description information and feature information of the first item, performing object detection on the first image based on the item description information of the first item to obtain a first candidate region; Performing image feature extraction on the first candidate area to obtain first candidate feature information; Based on the similarity between the first candidate feature information and the feature information of the first object, determining whether the first image includes an image corresponding to the first object, and obtaining the first search result; or, In response to the first item information including item description information of the first item but not including feature information of the first item, performing object detection on the first image based on the item description information of the first item to obtain a second candidate region; Based on the second candidate area, obtaining the first search result; or, In response to the first item information including feature information of the first item but excluding item description information of the first item, performing object detection on the first image to obtain a third candidate region; Performing image feature extraction on the third candidate region to obtain third candidate feature information; Based on the similarity between the third candidate feature information and the feature information of the first object, it is determined whether the first image includes an image corresponding to the first object, and the first search result is obtained.
5. The method according to any one of claims 2 to 4, wherein: After obtaining the first search result, the following steps are also included: In response to the first search result being that the first item is found, an alarm message is sent to the first user to prompt that the first item is left in the first place.
6. The method according to any one of claims 2 to 5, wherein: The item query system includes at least one of a registered place information table and a registered item information table of the first user; The registration location information table of the first user includes the registration location information of at least one registration location registered by the first user in the item query system, and the registration location information of each registration location includes location information of each registration location and sensor information corresponding to each registration location; The registered item information table of the first user includes item registration information of at least one registered item registered by the first user in the item query system, and the item registration information of each registered item includes item information of each registered item and a first place list; wherein the item information of each registered item includes at least one of an image and item description information of each registered item, and when the item information of each registered item includes an image, the item registration information of each registered item also includes feature information obtained by extracting features from the image; the first place list of each registered item includes place information of a second place where each registered item is allowed to be left; The alarm item information corresponding to the first place is determined based on the registered place information table of the first user and the list of each first place in the registered item information table.
7. The method according to claim 6, further comprising: Acquire a third image captured by a second sensor, where the second sensor includes a sensor corresponding to the sensor information in the registration location information table of the first user; Searching for the first object based on the object registration information of the first object and the third image to determine a third place where the first object is located; According to the second cycle, calling the third sensor corresponding to the third field to perform image acquisition, and performing target detection and position tracking on the first object based on the acquired first image sequence and the object registration information of the first object, to obtain a first position tracking result; Based on the first position tracking result, in response to a change in the position of the first object, calling the third sensor to perform image acquisition according to a third period, and performing target detection and position tracking on the first object based on the acquired second image sequence and the object registration information of the first object; wherein the third period is smaller than the second period; In response to not detecting the first object in the most recent frame of image acquired by the third sensor, acquiring a third image sequence; wherein the third image sequence includes: at least one frame of image acquired by the third sensor and acquired earlier than the most recent frame of image; Using the analysis model, event analysis is performed on the third image sequence to obtain an event analysis result of the first object, and the event analysis result of the first object is stored in a log file of the object query system.
8. The method according to claim 1, wherein: The first event includes receiving an item search request sent by a first user, wherein the item search request includes the first item information; Determining first object information and first sensor information corresponding to a first event includes: In response to receiving the item search request sent by the first user, the first item information is obtained from the item search request, and sensor information corresponding to at least one registration location registered by the first user in the item query system is obtained as the first sensor information.
9. The method according to claim 8, wherein: Acquiring sensor information corresponding to at least one registration location registered by the first user in the item query system as the first sensor information includes: In response to the item search request including first location information, acquiring sensor information corresponding to the first location information as the first sensor information; or, In response to the item search request not including location information, sensor information corresponding to at least one registered location registered by the first user in the item query system is obtained as the first sensor information.
10. The method according to any one of claims 8-9, wherein: Performing a first item search based on the first item information and the first image to obtain a first search result includes: In response to the first item information including a first item image and first item description information, feature extraction is performed on the first item image to obtain first feature information; and object detection is performed on the first image based on the first item description information to obtain a fourth candidate region; Performing image feature extraction on the fourth candidate region to obtain fourth candidate feature information; Based on the similarity between the fourth candidate feature information and the first feature information, determining whether the first image includes a first object image corresponding to the first object information, obtaining a first search result, and outputting the first search result; or, In response to the first item information including first item description information but not including first item image, determining whether the first user's registered item information table includes feature information corresponding to the first item description information; In response to the first user's registered item information table including second feature information corresponding to the first item description information, target detection is performed on the first image based on the first item description information to obtain a fifth candidate region; image feature extraction is performed on the fifth candidate region to obtain fifth candidate feature information; based on the similarity between the fifth candidate feature information and the second feature information, it is determined whether the first image includes a first item image corresponding to the first item information, to obtain a first search result, and the first search result is output; In response to the fact that the registered item information table of the first user does not include feature information corresponding to the first item description information, target detection is performed on the first image based on the first item description information, and based on the obtained sixth candidate area, whether the first image includes a first item image corresponding to the first item information is determined, the first search result is obtained, and the first search result is output; or, In response to the first item information including the first item image but not including the first item description information, extracting features from the first item image to obtain first feature information; Performing object detection on the first image to obtain a seventh candidate region; Extracting image features of the seventh candidate region to obtain seventh candidate feature information; Based on the similarity between the seventh candidate feature information and the first feature information, it is determined whether the first image includes a first object image corresponding to the first object information, a first search result is obtained, and the first search result is output.
11. The method according to claim 10, wherein: After determining whether the first image includes the first object image corresponding to the first object information, the method further includes: In response to the first image not including the first object image, acquiring at least one frame of image from historical images acquired by the first sensor as a fourth image; The first item search is performed based on the first item information and the fourth image to obtain the first search result.
12. The method according to any one of claims 10-11, wherein: The first search result is that the first item is not found. After outputting the first search result, the method further includes: receiving query auxiliary information sent by the first user; In response to the query auxiliary information including the second time information and the fourth place information, acquiring a fourth sensor corresponding to the fourth place information, wherein at least one registered place registered by the first user in the item query system includes a fourth place corresponding to the fourth place information; acquiring, from the images captured by the fourth sensor, at least one frame of image corresponding to the second time information as a fifth image; Searching for the first item based on the first item information and the fifth image to obtain a second search result; or, In response to the query auxiliary information including the second time information but not including the location information, acquiring a first sensor corresponding to at least one registered location registered by the first user in the item query system; acquiring, from the images captured by the first sensor, at least one frame of image corresponding to the second time information as a sixth image; Searching for the first item based on the first item information and the sixth image to obtain a second search result; or, In response to the query auxiliary information including fourth location information but not including time information, acquiring a fourth sensor corresponding to the fourth location information; Acquire, from the images acquired by the fourth sensor, a plurality of frames of images acquired within a first historical time period as a seventh image; The first item search is performed based on the first item information and the seventh image to obtain a second search result.
13. The method according to claim 12, wherein: The second search result is that the first item is not found. After obtaining the second search result, the method further includes: querying whether a log file of the item query system includes an event analysis result of the first item; In response to the event analysis result of the first item being included in the log file, determining the whereabouts of the first item based on the event analysis result of the first item, obtaining a whereabouts determination result, and outputting the whereabouts determination result; In response to the event analysis result that the log file does not include the first item, the second search result is output.
14. The method according to any one of claims 10-11, wherein: The first search result is that the first object is found in the first image, and the first search result includes location information of the first object, and the location information of the first object includes at least one of the following: a first image marked with an object detection frame and / or location coordinates of the first object, and location description information of the first object; After outputting the first search result, the method further includes: In response to receiving first feedback information sent by the first user, the first feedback information is used to indicate that the first search result is correct, determining whether the registered item information table of the first user includes item registration information of the first item; In response to the first user's registered item information table including the item registration information of the first item, storing the first feature information as the feature information of the first item in the item registration information of the first item, and storing the first search result as a positive sample in the item registration information of the first item; In response to the first user's registered item information table not including the item registration information of the first item, taking the first characteristic information as the characteristic information of the first item, taking the first search result as a positive sample, adding the item registration information of the first item to the first user's registered item information table, the item registration information of the first item including the first item information, the characteristic information of the first item, and the positive sample; or, In response to receiving second feedback information sent by the first user, the second feedback information is used to indicate that the first search result is wrong, determining whether the first user's registered item information table includes item registration information of the first item; In response to the first user's registered item information table including the item registration information of the first item, storing the first feature information as the feature information of the first item in the item registration information of the first item, and storing the first search result as a negative sample in the item registration information of the first item; In response to the fact that the registered item information table of the first user does not include the item registration information of the first item, the first characteristic information is used as the characteristic information of the first item, and the first search result is used as a negative sample, and the item registration information of the first item is added to the registered item information table of the first user, and the item registration information of the first item includes the first item information, the characteristic information of the first item and the negative sample.
15. The method according to claim 14, wherein: Performing a first item search based on the first item information and the first image to obtain a first search result includes: A first item search is performed based on the first item information, the first image, and item registration information of the first item to obtain the first search result.
16. The method according to any one of claims 1 to 15, wherein: The object search method is specifically applied to a computing device in the object query system; The item query system includes at least one sensor and at least one end-side device; wherein the at least one sensor is deployed in a location corresponding to the item query system, the at least one end-side device includes one or more terminal devices that a first user or a family has permission to use, and a first end-side device among the at least one end-side device is configured as the computing device; The place corresponding to the item query system includes at least one of at least one mobile device and at least one fixed place that the first user or family has permission to use; The at least one terminal side device includes at least one of the following types of terminal devices: a personal computer, a handheld or laptop device, a mobile terminal, a wearable device, a smart home appliance, and an in-vehicle infotainment device.
17. A computing device, applied to an item query system, comprising: A first determination module, configured to determine first item information and first sensor information corresponding to a first event; wherein the first sensor information includes sensor information of a first sensor, and the first sensor includes at least one sensor corresponding to a registered place, and the registered place is a place registered in the item query system; A first acquisition module, used to acquire a first image captured by the first sensor; The item search module is used to perform a first item search based on the first item information and the first image to obtain a first search result.
18. An Internet of Things system, comprising at least one sensor and at least one end-side device; wherein: The at least one sensor is deployed in at least one place corresponding to the Internet of Things system, the first end-side device among the at least one end-side device includes a computing unit, the second end-side device among the at least one end-side device includes an information input unit, the third end-side device among the at least one end-side device includes an information output unit, the fourth end-side device among the at least one end-side device includes an information storage unit, the first end-side device, the second end-side device, the third end-side device, and the fourth end-side device respectively include one or more terminal devices; each of the sensors and each of the terminal devices respectively includes a communication unit, the at least one sensor is communicatively connected to the first end-side device through the communication unit, and the other terminal devices among the at least one end-side device except the first end-side device are communicatively connected to the first end-side device through the communication unit.
19. The system of claim 18, wherein: The at least one terminal side device includes a terminal device that a user or a family has permission to use; the at least one terminal side device includes at least one of the following types of terminal devices: a personal computer, a mobile terminal, a handheld or laptop device, a wearable device, a smart home appliance, and an in-vehicle infotainment device; The at least one place corresponding to the Internet of Things system includes at least one of at least one mobile device and at least one fixed place that the user or family has permission to use.
20. The system according to any one of claims 18-19, wherein: The Internet of Things system is configured as an item query system; One of the terminal devices in the first end-side device includes the computing device described in claim 17; or, one of the terminal devices in the first end-side device is configured as a computing device, which is used to implement the item search method described in any one of claims 1-16.
21. A computer-readable storage medium, wherein the storage medium stores a computer program, and when the computer program is executed, it is used to implement the object search method described in any one of claims 1 to 16.
22. An electronic device, comprising: processor; a memory for storing instructions executable by the processor; The processor is used to read the executable instructions from the memory and execute the instructions to implement the item search method described in any one of claims 1-16.
Citation Information
Patent Citations
Image-matching-based automatic article management method and system
CN103150631A
Target tracking method and device, electronic equipment and storage medium
CN108734726A
Data processing method, system and device and computer readable storage medium
CN113822133A
Article searching method and device, AR equipment and readable storage medium
CN115639908A
Internet of Things data processing method and device, electronic equipment and storage medium
CN116912760A
