How to manage your household item inventory

Abstract

According to one embodiment, a method, computer system, and computer program product for item management are provided that may include identifying all items in sensor feeds of a set of sensors in a site; creating a list containing entries for the identified items, the entries including the name, location, and presence of the items; dynamically updating the list to track the location and / or presence of the items in real time based on continuous monitoring of the sensor feeds; in response to receiving a query from a user requesting a lost item, selecting an item in the list that matches the name of the lost item; and navigating the user to the location of the selected item.

Description

[Background technology]

[0001] The present invention relates generally to the field of computing, and more particularly to mixed reality.

[0002] Mixed reality is a field related to merging real and virtual worlds so that physical and digital objects coexist and interact in real time. Mixed reality does not occur exclusively in either the physical or the virtual world, but is a hybrid of real and virtual reality; as such, mixed reality represents everything in the real-virtual continuum except for the two extremes, i.e., purely physical and purely virtual environments. Mixed reality therefore includes augmented virtuality (AV), augmented reality (AR), and virtual reality (VR). Mixed reality has found practical applications in areas such as remote work, construction, gaming, and military, academic, and commercial training. Summary of the Invention

[0003] According to one embodiment, a method, computer system, and computer program product for item management are provided that may include identifying all items in sensor feeds of a set of sensors in a site; creating a list containing entries for the identified items, the entries including the name, location, and presence of the items; dynamically updating the list to track the location and / or presence of the items in real time based on continuous monitoring of the sensor feeds; in response to receiving a query from a user requesting a lost item, selecting an item in the list that matches the name of the lost item; and navigating the user to the location of the selected item. [Brief explanation of the drawings]

[0004] These and other objects, features, and advantages of the present invention will become apparent from the following detailed description of illustrative embodiments thereof, which is to be read in connection with the accompanying drawings. Since the illustrations are for clarity purposes only to facilitate understanding of the invention by those skilled in the art, together with the detailed description, the various features of the drawings are not to scale. The drawings are as follows:

[0005] [Figure 1] 1 illustrates an exemplary networked computer environment in accordance with at least one embodiment.

[0006] [Figure 2] 1 is an operational flowchart illustrating an initial list creation process of a household goods inventory management process in accordance with at least one embodiment.

[0007] [Figure 3] 1 is an operational flowchart illustrating a household goods inventory management process according to at least one embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0008] Detailed embodiments of the claimed structures and methods are disclosed herein; however, it should be understood that the disclosed embodiments are merely exemplary of the claimed structures and methods, which may be embodied in various forms. However, the present invention may be embodied in many different forms and should not be construed as being limited to the exemplary embodiments set forth herein. In the description, details of well-known features and techniques may be omitted to avoid unnecessarily obscuring the presented embodiments.

[0009] FIELD OF THE INVENTION Embodiments of the present invention relate to the field of computing, and more particularly to mixed reality. The exemplary embodiments described below provide, among other things, systems, methods, and program products for identifying items in a location via sensors, creating a list of items, dynamically updating the list with item locations, and highlighting the items in mixed reality based on location information in the list in response to a user query regarding the items.

[0010] As mentioned above, mixed reality is a field related to merging real and virtual worlds so that physical and digital objects coexist and interact in real time. Mixed reality systems use software to generate images, sounds, haptic feedback, and other sensations to augment the real-world environment. Creating this augmented environment can be accomplished using mobile devices such as cell phones or tablets, but more specialized equipment, typically in the form of glasses or headsets, is also used, in which computer-generated elements are overlaid on a view of the real world by being projected or mapped onto lenses in front of the user's eyes. With the aid of computer augmentation, information about the world around the user, and other digital elements overlaid on this world, become interactive and digitally manipulable.

[0011] People lose things all the time. The average person may lose thousands of items in their lifetime and may spend days or even weeks of their life searching for those lost items in total. There are several solutions that have been attempted to help forgetful individuals locate lost items. One such approach involves the use of tracking devices such as Bluetooth-based key ring locators, devices such as RFID tags, and even larger GPS tracking devices that are attached to frequently lost items. However, these solutions require that a tracking device be attached to any given object that may be lost. Placing trackers only on objects that a person expects to lose would be expensive, bulky, and inconvenient; placing trackers on every object that one might need to find would be nearly impossible.

[0012] Other attempted solutions may use cameras or other devices to identify the location of an item once it is identified as missing. However, this method is flawed in that locating an object once it is determined to be lost can be a time-consuming and unreliable process without knowledge of where the item was before it was identified as missing. Additionally, attempted solutions may attempt to communicate the location of the lost item using images, audible or textual descriptions, etc.; these can be problematic communication methods and may not accurately convey the location of the lost item to the user. Mixed reality devices may be better suited to communicating the location of a lost object by highlighting objects in the scene to the user so that they are clearly visible to the user even through walls and other objects. Therefore, it may be advantageous to implement a system that, among other things, automatically creates a list of all items in a scene based on sensor data collected in the scene, dynamically updates the list with the locations and presence of those items in the scene based on the sensor data, and, when a user asks for the location of the item, utilizes the list to identify the presence and / or location of an item in the scene and uses mixed reality to highlight the item in the scene. Such a method would not require bulky and expensive tracking hardware, and would therefore be cheaper and more convenient, would have a superior ability to communicate the location of a lost item to a user, would have data on that item, and would therefore be faster than methods attempted in the art.

[0013] According to at least one embodiment, the present invention may be a method for creating and maintaining an inventory of items in a site using a set of sensors.

[0014] In embodiments of the present invention, a site may be any discretely bounded geographic location, such as a home, factory, store, etc., that includes multiple items and a set of sensors. The area that includes the site may be defined by a geofence, which may be a virtual perimeter that represents a real-world geographic area that defines the site's geographic location. The geofence may be pre-provisioned to the system by a user or a human administrator or programmer. A user may be any individual who desires to locate items within and / or associated with a site. In embodiments of the present invention, a user may own the site, own the items, own the sensor set, and / or have exclusive or joint control.

[0015] An item may be any physical object associated with the site or the user that a user may desire to find and that is currently within the site or has been within the site at some point in the past, such as, for example, a cell phone, a breakfast sandwich, keys, tools, an instruction manual, etc. An item may be owned by the user and / or associated with the user, associated with an individual in the user's group, and / or owned by and / or associated with an individual who is not the user. In some embodiments, an item is small enough to be portable; for example, it may include only objects that are smaller than a threshold size and / or lighter than a threshold weight, the threshold size and weight representing size and weight values, respectively, above which an item is no longer considered portable. Larger, non-portable items often cannot be carried or moved and, therefore, are much less likely to be misplaced as a result.

[0016] The set of sensors may be one or more Internet of Things (IoT) devices that may be located within the site and may be capable of collecting data about the environment within the site, including about objects located within the site, and transmitting that data to the system. The sensors may be installed at static locations within the site and / or may be mobile devices such as cell phones or tablets, and / or may be attached to devices within the site that may move around the site and exit or enter the site. The set of sensors may include infrared and / or optical cameras, and may additionally include accelerometers, GPS receivers, motion sensors, SONAR / LIDAR emitters, etc., and may be integrated into mobile devices such as laptops, tablets, and cell phones, wearable devices such as smart watches and mixed reality headsets, and / or stationary devices such as wall-mounted security cameras, webcams, desktop computers, etc. A set of sensors may be owned and / or exclusively controlled and / or accessed by a user, owned and / or controlled by multiple individuals including the user, publicly controlled and / or accessed, owned and / or exclusively controlled by multiple users, etc.

[0017] In embodiments of the invention, a system may create a list of items by monitoring data from a set of sensors. The system may use optical recognition to identify all items or all portable items in camera footage captured within the scene. The system may obtain or be provided with a reference list of common objects, which may include the name and physical description of each of a number of common objects, including, for example, the size, weight, color, etc. of the common objects. The system may search for candidate items in frames comprising the camera feed of sensor data and may identify the object as a candidate item if it visually matches a common object on the initial list. In some embodiments of the invention, the system may continuously monitor sensor data for objects entering the scene so that objects can be identified and added to the list in real time or near real time as they are brought into the scene.

[0018] The terms “real-time,” “near-real-time,” and “dynamic” herein refer to computer processes that execute on a timescale on the order of milliseconds in response to the receipt of information, such that there is some delay between the receipt of information and the actual output. While the term “real-time” colloquially describes a human (as opposed to machine) sense of time that appears instantaneous, it should be understood herein to refer to a guaranteed level of computer responsiveness within a specified time constraint, on the order of milliseconds or microseconds, between an event and a response deadline that marks the boundary of the specified time constraint. Thus, real-time updates of a list may appear to a human to smoothly track the movement of an item, but in reality, several milliseconds may have elapsed between updates. In contrast to mainframe batch computers, where processes may be grouped and scheduled for processing at a later time, real-time computer processes may occur without delays exceeding transmission and processing times. In some embodiments of the present invention, for example, where it may be desirable to reduce resource usage, the system may introduce delays into the execution of computer processes so that they are no longer said to be occurring in real time, but rather at continuous, regular intervals spaced on the order of milliseconds or microseconds apart; such embodiments may be referred to herein as occurring in near real time. However, any embodiment involving real-time runs and spectra between runs spaced as little as five minutes apart may still be referred to herein as "dynamic."

[0019] In some embodiments of the present invention, if an identified object does not match any generic object in the reference list, the system may prompt the user to identify the identified object, for example, by highlighting the object in mixed reality, making an audible request to the user using synthesized and / or pre-recorded speech, sending a text and / or graphical prompt to a display-equipped computing device associated with the user, etc. The prompt may request a name and / or a physical description of the object. The system may then receive the user's answer, for example, through a textual or audible response to the prompt transmitted to the system and / or received through one or more microphones comprising the sensor set. The system may place the object as an item in a list under the provided name and with the provided description. In some embodiments of the present invention, the system may prompt the user to identify whether the object should be counted as an item and whether the object should be included in the list.

[0020] If a detected object matches a generic object in the reference list and / or is identified to the system by the user in response to a prompt, the object may be designated as a candidate item. The system may add the candidate item as an item to the list, along with a description provided by the reference list and / or by the user and the location and position of the object as identified by the sensor set. In embodiments of the invention, the system may determine that a candidate item is too large or heavy to be portable based on the size and / or weight of the corresponding generic object in the description from the reference list, and may remove the candidate item from consideration for inclusion in the list.

[0021] In some embodiments of the present invention, the system may assign priorities to items in the list. The priority associated with an item may reflect the importance of the item to the user and, therefore, the importance of quickly finding the item once it is lost. For example, a user's cell phone may be an item that the user relies heavily on in their daily life. As such, the cell phone item may be rated as "high priority." Similarly, an appliance such as a fire extinguisher may be used infrequently, but it may be absolutely important to find it quickly if needed, so it may also be "high priority." The system may base its assessment of an item's priority on usage patterns identified by a machine learning model based on sensor data from a sensor set.

[0022] In some embodiments of the present invention, the system may assign a loss rate to an item. The loss rate may indicate the likelihood that an item may be lost or that a user may need to find the item. For example, an instrument such as a socket wrench set may be used infrequently and therefore unlikely to be misplaced or moved; therefore, the system may assign a low loss rate to the socket wrench set item. On the other hand, an item such as a favorite hair tie may be an item that a user uses daily and therefore very likely to be misplaced; therefore, the system may assign a high loss rate to the hair tie item. The system may base its assessment of an item's priority on an item's usage pattern identified by a machine learning model based on sensor data from a sensor set.

[0023] In embodiments of the present invention, the system may maintain a list of items by continuously monitoring sensor data to detect item movement and dynamically updating the locations and positions of listed items in real time or near real time as items are moved within the site. The system may analyze live camera feeds from a set of sensors to identify the location of items and revise the list with the items' updated locations and positions as items are moved. In some embodiments, the system may analyze an area of ​​recorded sensor feed from a stationary camera for movement and update only the locations and positions of items visible within the area of ​​the sensor feed where movement is detected. In some embodiments of the present invention, the system may prioritize monitoring and updating the location of items in order from highest priority to lowest priority and / or highest loss rate to lowest loss rate, such that changes in the location and / or position of high-priority and / or high-loss-rate items are detected and / or updated in the list first. In some embodiments of the present invention, the system may detect when an item has entered or left the site by, for example, determining that the item has crossed a geofence that marks the perimeter of the site based on the item's location. The system may update the list to reflect whether the item is on or off site. In some embodiments of the present invention, the system may feed the location and position of items into a machine learning algorithm to derive usage patterns for different items, which may include the time of day that items are moved, the location that items are moved to, how often users interact with the items, the time of day that items leave and / or enter a site, etc.

[0024] In embodiments of the invention, if the system has previously identified an item, but the item is no longer in the previously recorded location and the system cannot identify the item's current location, the system may prompt the user to locate the item and provide the location to the system, for example, by describing the item's location, taking video of the item, showing the item in the view of a camera in the sensor set, etc. In some embodiments of the invention, for example, if the list includes priority and / or loss rate, the system may prompt the user to locate only high priority and / or high loss rate items, or may prompt the user with a frequency and / or urgency commensurate with the priority and / or loss rate of the item.

[0025] According to at least one embodiment, the present invention may be a method for identifying a lost item in a list in response to a user query and navigating the user to the lost item using mixed reality.

[0026] In embodiments of the present invention, the user query may be a text request transmitted to the system from the user's computing device, e.g., laptop, mobile phone, tablet, etc., and / or may be an audible voice request made by the user and received at a microphone integrated into, e.g., mobile phone, voice assistant device, laptop, etc. The query may include a request to find a lost item, which may be an item that the user has placed in an improper location and wishes to find.

[0027] In embodiments of the present invention, a mixed reality session may be a discrete period of time during which a user interacts with a mixed reality experience through a mixed reality device. A mixed reality program may be a software program such as a training simulator, a game, a narrative experience such as a video, a social platform, etc. that creates or provides one or more mixed reality experiences for a user to interact with through the mixed reality device. A mixed reality device or mixed reality-enabled device may be any device or combination of devices capable of recording real-world information that a mixed reality program may overlay with computer-generated perceptual elements to create a mixed reality environment experienced by a user; the mixed reality device may further record a user's actions, position, movement, etc. to track the user's movement within the mixed reality environment. The mixed reality device may display the mixed reality environment to a user. A mixed reality device may be equipped with or include multiple sensors, which may include a sensor set, such as a camera, a microphone, an accelerometer, etc., and / or multiple user interface devices, such as a display, a touchscreen, a speaker, etc. In some embodiments, the mixed reality device may be a headset worn by the user.

[0028] In embodiments of the present invention, the system may identify lost items by obtaining location and position data in the list of list items that correspond to items identified in a user query. If the system is unable to find a lost item identified in the query, the system may request that the user describe the object and may compare the user's description against the descriptions in the list.

[0029] The system may guide the user to an item by highlighting or otherwise indicating the item by overlaying a graphical element on the item in the mixed reality session the user is currently experiencing. The system may prompt the user to start a mixed reality session on the user's mixed reality-enabled device, such as a tablet or mobile phone, and may indicate the missing object in the mixed reality session with a graphical element overlaid on the object's location. In some embodiments of the present invention, the system may guide the user to the item by audibly or textually describing the location of the lost item to the user. If the lost item is not present within the scene, the system may notify the user that the item is not within the scene, for example, through a text box, an audible spoken message, and / or a graphical element. The system may further notify the user of the last time the item was within the scene and / or its last known location and / or the date, time, and / or location when the item left the scene.

[0030] In some embodiments of the present invention, the system may store video segments recorded by a mixed reality device associated with a user; in response to receiving a query from the user, the system may search for objects in the recorded video segments, from most recently recorded to least recently recorded, until the system identifies the lost item. The system may then play the recorded video segments to the user on a display and / or within the mixed reality environment to guide the user to the lost item or communicate the last known location of the lost item to the user. As an example, Grandma Sadie uses her trowel in the yard of a house equipped with a security camera on Monday. The system monitors activity within the security camera's view and stores the footage. The system analyzes the stored footage and recognizes the trowel in a segment of recorded footage captured by this exterior security camera covering the yard. The system adds the location of the recognized trowel to a virtual household item inventory list and updates the recorded footage with metadata, including the time and location in the footage where the trowel appears, indicating the trowel's presence in the footage. When Saturday comes, Grandma Sadie wants to garden again but has forgotten where she left her trowel. Grandma Sadie can ask the system, "Where's my trowel?" The system searches its household goods inventory list for the entry "trowel" and retrieves the most recent footage associated with the trowel entry. The system then activates the display on Grandma Sadie's tablet and plays video of the camera footage from the garden on Monday when Grandma Sadie was using the trowel.

[0031] Various aspects of the present disclosure are described through text, flowcharts, block diagrams of computer systems, and / or block diagrams of machine logic included in embodiments of a computer program product (CPP). For any flowchart, depending on the technology involved, operations may be performed in an order different from that shown in a given flowchart. For example, again depending on the technology involved, two operations shown in successive flowchart blocks may be performed in the reverse order, as a single integrated step, simultaneously, or in an at least partially overlapping manner.

[0032] A computer program product embodiment ("CPP embodiment" or "CPP") is a term used in this disclosure to describe any set of one or more storage media (also referred to as "media"), collectively contained in one or more storage devices, that collectively contain machine-readable code corresponding to instructions and / or data for performing the computer operations specified in a given CPP claim. A "storage device" is any tangible device that can hold and store instructions for use by a computer processor. The computer-readable storage medium may be, but is not limited to, an electronic storage medium, a magnetic storage medium, an optical storage medium, an electromagnetic storage medium, a semiconductor storage medium, a mechanical storage medium, or any suitable combination of the foregoing. Some known types of storage devices that include these media include diskettes, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), compact disc read-only memory (CD-ROM), digital versatile disc (DVD), memory stick, floppy disk, mechanically encoded device (such as pits / lands formed on a major surface of a punch card or disk), or any suitable combination of the foregoing. Computer-readable storage media, as the term is used in this disclosure, is not to be construed as storage in the form of transitory signals per se, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through a waveguide, light pulses passing through fiber optic cables, electrical signals communicated through wires, and / or other transmission media. As will be appreciated by those skilled in the art, data is typically moved at some infrequent time during the normal operation of a storage device, such as during access, defragmentation, or garbage collection, but this does not make the storage device transient, as the data is not transient while it is stored.

[0033] The exemplary embodiments described below provide systems, methods, and program products for identifying items in a location via sensors, creating a list of items, dynamically updating the list with item locations, and highlighting the items in mixed reality based on location information in the list in response to a user query regarding the items.

[0034] Referring now to FIG. 1 , computing environment 100 includes an example of an environment for the execution of at least some of the computer code involved in performing the methods of the present invention, such as code blocks 145, which may include a virtual assistant program 107 and a household goods inventory management program 108. In addition to code blocks 145, computing environment 100 includes, for example, a computer 101, a wide area network (WAN) 102, an end user device (EUD) 103, a remote server 104, a public cloud 105, and a private cloud 106. In this embodiment, computer 101 includes a processor set 110 (including processing circuitry 120 and cache 121), a communications fabric 111, volatile memory 112, persistent storage 113 (including operating system 122 and code blocks 145, as identified above), a peripheral device set 114 (including a user interface (UI), a device set 123, storage 124, and an Internet of Things (IoT) sensor set 125), and a network module 115. Remote server 104 includes a remote database 130. The public cloud 105 includes a gateway 140, a cloud orchestration module 141, a set of host physical machines 142, a set of virtual machines 143, and a set of containers 144.

[0035] Computer 101 may take the form of a desktop computer, a laptop computer, a tablet computer, a smartphone, a smartwatch or other wearable computer, a mainframe computer, a quantum computer, or any other form of computer or mobile device now known or later developed that is capable of executing programs, accessing a network, or querying a database, such as remote database 130. As is well understood and consistent with the art of computer technology, execution of a computer-implemented method may be distributed among multiple computers and / or among multiple locations. While in this presentation of computing environment 100, to keep the presentation as simple as possible, the detailed discussion focuses on a single computer, specifically computer 101. Computer 101 may be located in a cloud, even if not depicted within the cloud in FIG. 1 . However, computer 101 is not required to be in a cloud except to any extent that may be categorically indicated.

[0036] Processor set 110 includes one or more computer processors of any type now known or later developed. Processing circuitry 120 may be distributed across multiple packages, e.g., multiple tailored integrated circuit chips. Processing circuitry 120 may implement multiple processor threads and / or multiple processor cores. Cache 121 is memory located within the processor chip package and is typically used for data or code that should be available for quick access by threads or cores running on processor set 110. Cache memory is typically organized into multiple levels depending on relative proximity to the processing circuitry. Alternatively, some or all of the cache for a processor set may be located “off-chip.” In some computing environments, processor set 110 may be designed for operation with qubits and for performing quantum computing.

[0037] Computer-readable program instructions are typically loaded onto computer 101 and cause processor set 110 of computer 101 to perform a series of operational steps, thereby performing a computer-implemented method, such that the instructions so executed instantiate the method specified in the flowcharts and / or descriptions of the computer-implemented method contained herein (collectively referred to as the "methods of the present invention"). These computer-readable program instructions are stored in various types of computer-readable storage media, such as cache 121 and other storage media described below. The program instructions and associated data are accessed by processor set 110 to control and direct the execution of the methods of the present invention. In computing environment 100, at least some of the instructions for performing the methods of the present invention may be stored in code blocks 145 in persistent storage 113.

[0038] Communications fabric 111 is the signal-conducting pathway that allows various components of computer 101 to communicate with one another. Typically, this fabric is made up of switches and conductive pathways, such as those that make up buses, bridges, and physical input / output ports. Other types of signal communication pathways may be used, such as fiber optic and / or wireless communication pathways.

[0039] Volatile memory 112 may be any type of volatile memory now known or later developed. Examples include dynamic random access memory (RAM) or static RAM. Typically, volatile memory is characterized by random access, although this is not required unless expressly stated. In computer 101, volatile memory 112 is located in a single package and internal to computer 101, although alternatively or additionally, volatile memory may be distributed across multiple packages and / or located external to computer 101.

[0040] Persistent storage 113 is any form of non-volatile storage for a computer, now known or later developed. The non-volatility of this storage means that stored data is maintained regardless of whether power is supplied to computer 101 and / or to persistent storage 113 directly. Persistent storage 113 may be read-only memory (ROM), but typically at least a portion of the persistent storage allows data to be written, data to be erased, and data to be rewritten. Some well-known forms of persistent storage include magnetic disks and solid-state storage devices. Operating system 122 can take several forms, such as various known proprietary operating systems or open-source Portable Operating System Interface-type operating systems that utilize a kernel. The code contained within code block 145 typically includes at least some of the computer code involved in performing the methods of the present invention.

[0041] Peripheral device set 114 includes a set of peripheral devices of computer 101. Data communication connections between peripheral devices and other components of computer 101 may be implemented in various ways, such as Bluetooth connections, near field communication (NFC) connections, connections made by cable (such as a universal serial bus (USB)-type cable), insertable connections (e.g., a Secure Digital (SD) card), connections made through a local area communication network, and even connections made through a wide area network, such as the Internet. In various embodiments, UI device set 123 may include components such as a display screen, speakers, microphones, wearable devices (such as goggles and smartwatches), keyboards, mice, printers, touchpads, game controllers, and haptic devices. Storage 124 may be external storage, such as an external hard drive, or insertable storage, such as an SD card. Storage 124 may be persistent and / or volatile. In some embodiments, storage 124 may take the form of a quantum computing storage device for storing data in the form of qubits. In embodiments where computer 101 is required to have a large amount of storage (e.g., computer 101 stores and manages a large database locally), this storage may be provided by a peripheral storage device designed to store very large amounts of data, such as a storage area network (SAN) shared by multiple geographically distributed computers. IoT sensor set 125 consists of sensors that may be used in Internet of Things applications. For example, one sensor may be a thermometer and another sensor may be a motion detector.

[0042] Network module 115 is a collection of computer software, hardware, and firmware that enables computer 101 to communicate with other computers over WAN 102. Network module 115 may include hardware such as a modem or Wi-Fi® signal transceiver, software for packetizing and / or depacketizing data for communication network transmission, and / or web browser software for communicating data over the Internet. In some embodiments, the network control and network forwarding functions of network module 115 are performed on the same physical hardware device. In other embodiments (e.g., embodiments utilizing software-defined networking (SDN)), the control and forwarding functions of network module 115 are performed on physically separate devices, such that the control function manages several different network hardware devices. Computer-readable program instructions for implementing the methods of the present invention may be downloaded to computer 101, typically from an external computer or external storage device, through a network adapter card or network interface included in network module 115.

[0043] WAN 102 is any wide area network (e.g., the Internet) capable of communicating computer data over non-local distances by any technology for communicating computer data now known or later developed. In some embodiments, a WAN may be replaced and / or supplemented by a local area network (LAN) designed to communicate data between devices located in a local area, such as a Wi-Fi network. WANs and / or LANs typically include computer hardware such as copper transmission cables, optical fiber transmissions, wireless transmissions, routers, firewalls, switches, gateway computers, and edge servers.

[0044] End-user device (EUD) 103 is any computer system used and controlled by an end user (e.g., a customer of the enterprise operating computer 101) and may take any of the forms described above in connection with computer 101. EUD 103 typically receives useful and actionable data from the operation of computer 101. For example, in a hypothetical case in which computer 101 is designed to provide recommendations to the end user, the recommendations would typically be communicated from computer 101's network module 115 over WAN 102 to EUD 103. In this manner, EUD 103 can display or otherwise present the recommendations to the end user. In some embodiments, EUD 103 may be a client device, such as, for example, a thin client, a heavy client, a mainframe computer, a desktop computer, or the like.

[0045] Remote server 104 is any computer system that provides at least some data and / or functionality to computer 101. Remote server 104 may be controlled and used by the same entity that operates computer 101. Remote server 104 represents a machine that collects and stores useful and useful data for use by other computers, such as computer 101. For example, in the hypothetical case where computer 101 is designed and programmed to provide recommendations based on historical data, this historical data may be provided to computer 101 from remote database 130 of remote server 104.

[0046] Public cloud 105 is any computer system available for use by multiple entities that provides on-demand availability of computer system resources and / or other computer capabilities, particularly data storage (cloud storage) and computing capacity, without direct, active management by users. Cloud computing typically leverages resource sharing to achieve coherence and economies of scale. Direct, active management of public cloud 105's computing resources is performed by computer hardware and / or software in cloud orchestration module 141. The computing resources provided by public cloud 105 are typically implemented by virtual computing environments running on various computers comprising host physical machine set 142, the universe of physical computers within and / or available in public cloud 105. Virtual computing environments (VCEs) typically take the form of virtual machines from virtual machine set 143 and / or containers from container set 144. It is understood that these VCEs may be stored as images and may be transferred among and between various physical machine hosts either as images or after instantiation of the VCE. The cloud orchestration module 141 manages the transfer and storage of images, deploys new instantiations of the VCE, and manages active instantiations of VCE deployments. The gateway 140 is a collection of computer software, hardware, and firmware that enables the public cloud 105 to communicate over the WAN 102.

[0047] Some further description of virtualized computing environments (VCEs) is now provided. A VCE can be stored as an "image." A new active instance of a VCE can be instantiated from the image. Two well-known types of VCEs are virtual machines and containers. A container is a VCE that uses operating system-level virtualization. This refers to a feature of an operating system in which the kernel allows the existence of multiple isolated user space instances, called containers. These isolated user space instances typically behave as actual computers from the perspective of programs running within them. A computer program running on a typical operating system can utilize all of the computer's resources, such as connected devices, files and folders, network shares, CPU power, and quantifiable hardware capabilities. However, a program running inside a container can only use the contents of the container and the devices assigned to the container; this feature is known as containerization.

[0048] A private cloud 106 is similar to a public cloud 105, except that its computing resources are available only for use by a single enterprise. While the private cloud 106 is shown in communication with the WAN 102, in other embodiments, the private cloud may be completely disconnected from the Internet and accessible only through a local / private network. A hybrid cloud is a composite of multiple clouds of different types (e.g., private, community, or public cloud types), often implemented by different vendors. While each of the multiple clouds remains a separate, discrete entity, the larger hybrid cloud architecture is bound together by standardized or proprietary technologies that enable orchestration, management, and / or data / application portability between the constituent clouds. In this embodiment, both the public cloud 105 and the private cloud 106 are part of a larger hybrid cloud.

[0049] According to this embodiment, the virtual assistant program 107 may be one of any number of software agents capable of interacting with a user through text or audible speech, providing information, or performing tasks based on the user's voice or text input. Examples include voice-based virtual assistants such as Google Home (registered trademark) (Google Home (registered trademark) and all Google Home (registered trademark)-based trademarks and logos are trademarks or registered trademarks of Google Inc. and / or its affiliates), Amazon Echo (registered trademark) (Amazon Echo (registered trademark) and all Amazon Echo (registered trademark)-based trademarks and logos are trademarks or registered trademarks of Amazon Technologies, Inc. and / or its affiliates), and Siri (registered trademark) (Siri (registered trademark) and all Siri (registered trademark)-based trademarks and logos are trademarks or registered trademarks of Apple Inc. and / or its affiliates). The virtual assistant program 107 does not necessarily have to be located on the client computing device 102; the virtual assistant program 107 can be stored and / or executed in or by any number or combination of devices, including the computer 101, the end user device 103, the remote server 104, the private cloud 106 and / or the public cloud 105, the peripheral device set 114, and the server 112, and / or on any other device connected to the WAN 102. Furthermore, the virtual assistant program 107 can be distributed across any number or combination of the above-mentioned devices in its operation. In some embodiments, the virtual assistant program 107 can be stored and / or executed on its own dedicated virtual assistant device, which can be a hardware device equipped with a speaker and a microphone capable of receiving voice-based user queries that can be synthesized and / or composed from pre-recorded segments and responding with audible artificial human speech.

[0050] According to this embodiment, the household goods inventory control program 108 may be a program capable of identifying items in a location via sensors, creating a list of items, dynamically updating the list with item locations, and highlighting the items in mixed reality based on location information in the list in response to a user query regarding the items. When executed, the household goods inventory control program 108 may cause the computing environment 100 to perform a household goods inventory control process 300, which may be described in further detail below with respect to FIG. 3 . In embodiments of the present invention, the household goods inventory control program 108 may be stored and / or executed within or by any number or combination of devices, including the computer 101, the end-user device 103, the remote server 104, the private cloud 106 and / or the public cloud 105, the peripheral device set 114, and the server 112, and / or on any other device connected to the WAN 102. Furthermore, the household goods inventory control program 108 may be distributed in its operation across any number or combination of the aforementioned devices.

[0051] 2, an operational flowchart illustrating an initial list creation process 200 of the household goods inventory management process 300 is shown in accordance with at least one embodiment. At 202, the household goods inventory management program 108 may recognize all objects in the sensor feeds of a sensor set within the site. The objects may be physical objects, such as furniture, coffee mugs, paintings, etc. The sensor set may be one or more Internet of Things (IoT) devices that may be located within the site and may be capable of collecting data about the environment within the site, including about objects located within the site, and transmitting that data to the household goods inventory management program 108. The set of collected data communicated from the sensor set to the household goods inventory management program 108 may be referred to herein as a sensor feed. The sensor set may include any number and / or combination of infrared and / or optical cameras, and may additionally include accelerometers, GPS receivers, motion sensors, SONAR / LIDAR emitters, etc., and may be integrated into mobile devices such as laptops, tablets and mobile phones, wearable devices such as smart watches and mixed reality headsets, and / or stationary devices such as, for example, wall-mounted security cameras, webcams, desktop computers, etc.

[0052] The household goods inventory control program 108 may use image processing, SONAR / LIDAR processing, and other data processing techniques to extract information about the environment within the site from the raw sensor data collected by the sensor set. Specifically, the household goods inventory control process 300 may process the sensor data from the sensor feeds to recognize objects within the site. The household goods inventory control process 300 may comb through all available sensor feeds from the site and recognize objects within the site using any object recognition technology. Recognizing an object may involve, for example, flagging an area in an image or model of the area within the site created from the sensor data as containing the object. In some embodiments, for example, if the list includes only portable items, the household goods inventory control program 108 may only recognize objects below a certain size.

[0053] At 204, the household goods inventory management program 108 may identify the recognized object as an item based on a list of reference objects. An item may be any physical object associated with the site or the user that a user may desire to find and that is currently on site or was on site at some point in the past, such as a cell phone, a breakfast sandwich, keys, tools, an instruction manual, etc. The household goods inventory management program 108 may obtain or be provided with a reference list of generic objects, which may include the name and physical description of each of a plurality of generic objects, which may be a list, database, or other logical structure, for example, including the size, weight, color, etc. of generic objects. The household goods inventory management program 108 may compare the recognized object in the sensor feed against the list of reference objects and may identify the object as an item if the recognized object visually matches a generic object on the initial list. In some embodiments, the items may include only objects that are small enough to be portable; for example, the threshold size and / or weight may represent a size and / or weight value, respectively, above which an item is no longer considered portable. The household goods inventory management program 108 may compare the size and / or weight of an item identified in the description as matching a common object from the reference list against a threshold size and / or weight; if the item meets or exceeds the threshold, the household goods inventory management program 108 may not add the item to the list.

[0054] At 206, the home goods inventory management program 108 may determine the location of the identified item within the scene based on the sensor feed. The home goods inventory management program 108 may analyze the area of ​​the sensor feed containing the identified item to determine the location and presence of the item. The home goods inventory management program 108 may identify the location of the item within the scene using image analysis techniques, such as monoplotting, laser ranging, SONAR, and LIDAR ranging techniques. In some embodiments of the present invention, for example, the home goods inventory management program 108 may identify the location of the item through image analysis using indoor localization and object measurement methods. If the home goods inventory management program 108 knows the location and angle of the sensor, the home goods inventory management program 108 can determine the location of the object relative to the sensor because each frame / image in CCTV or other IoT device footage may include metadata. The metadata may include camera ID, location, time, etc.

[0055] At 208, the household item inventory management program 108 may create a list including descriptions from the list of identified items, locations, and reference objects. The list may be a list, database, or other logical structure that includes an entry for every item identified in the sensor feeds of the sensor set within the scene, along with the item's name, an indicator of whether the item is present within the scene, and, if present, the item's location within the scene. Each entry for an item may additionally include a physical description provided by the reference list and / or a user, a web service, a human administrator, etc., alternative names for the item in cases where the item can be referred to in multiple different ways, etc. In some embodiments, an entry for an item in the list may further include a video recording or a link to a video recording recorded while the user was experiencing a mixed reality session within the scene in which the item is visible.

[0056] At 210, the home goods inventory control program 108 may update the list based on continuous monitoring of the sensor feeds as the location and presence of identified items change in the sensor feeds. The home goods inventory control program 108 may analyze live sensor feeds from the sensor set to identify the location of items and revise the list with the updated location of items when items are moved. In some embodiments, the home goods inventory control program 108 may analyze only areas of the recorded sensor feed from stationary sensors where motion is detected and may update only the locations of items in the areas of the sensor feed where motion is detected. In some embodiments of the present invention, the home goods inventory control program 108 may continuously monitor sensor data to recognize new objects entering the site so that as new objects are brought to the site, they can be identified as items and added to the list in real time or near real time. Similarly, the household goods inventory control program 108 may continuously monitor the sensor data to recognize previously identified items that have previously left the site and then entered the site, so that the presence data of a previously identified item may be updated in the list to reflect in real time or near real time that the item is currently present within the site when the item is brought into the site. The household goods inventory control program 108 may continuously monitor the sensor data to identify items that have left the site and update the missing item's presence data in the list in real time or near real time to reflect that the item has left the site when the item leaves the site.

[0057] The household goods inventory control program 108 may generate a 3D model of the site and / or area monitored by the set of sensors based on the sensor feeds, and may track and represent items from the list within the 3D model. The household goods inventory control program 108 may update areas of the 3D model not covered by static sensors by operating mobile sensors and / or recording sensor data from mobile devices that move through the site, such as those used or held by a user or other expressly consenting participant.

[0058] In some embodiments of the present invention, the household goods inventory management program 108 may assign a priority to an item in the list. The priority associated with an item may reflect the importance of the item to the user and, therefore, the importance of quickly finding the item once it is lost. The household goods inventory management program 108 may base an assessment of the item's priority on a usage pattern for the item identified by a machine learning model based on sensor data from the sensor set. In some embodiments of the present invention, the household goods inventory management program 108 may assign a loss rate to the item. The loss rate may indicate the likelihood that the item may be lost or that a user may need to find the item. The household goods inventory management program 108 may base an assessment of the item's priority on a usage pattern for the item identified by a machine learning model based on sensor data from the sensor set. In some embodiments of the present invention, the household goods inventory management program 108 may prioritize monitoring and updating of item locations in order from highest priority to lowest priority and / or highest loss rate to lowest loss rate, such that changes in the location and / or position of high priority and / or high loss rate items are detected and / or updated first in the list.

[0059] In an embodiment of the present invention, if the household goods inventory management program 108 has previously identified an item, but the item is no longer in the previously recorded location, and the household goods inventory management program 108 is unable to identify the item's current location for a predetermined period of time, the household goods inventory management program 108 may prompt the user to locate the item and provide the location to the household goods inventory management program 108.

[0060] Referring now to FIG. 3, an operational flowchart illustrating a household goods inventory management process 300 is shown in accordance with at least one embodiment. At 302, the household goods inventory management program 108 may receive a query from a user in the field to identify a lost item. The user query may be a text request transmitted from the user's computing device, such as a laptop, cell phone, tablet, etc., to the household goods inventory management program 108, and / or an audible voice request made by the user and received at a microphone integrated into, for example, a cell phone, voice assistant device, laptop, etc. The query may include a request to find a lost item, which may be an item that the user misplaced and wishes to find. For example, if the user has forgotten both the object and the name of the object but remembers what it looks like, the query may alternatively or additionally include a description of the object.

[0061] At 304, the household goods inventory management program 108 may search the list for items that match the name or description of the lost item. The household goods inventory management program 108 may use database search techniques to find an exact match of the name and / or description of an item in the list. If an exact match does not exist, the household goods inventory management program 108 may use text processing techniques to identify names and / or descriptions of items in the list that fall within a threshold level of similarity to the name / description of the lost item identified in the query.

[0062] At 306, the household goods inventory management program 108 may retrieve the location and presence of items in the list that match the name or description of the lost item. Upon identifying a match, the household goods inventory management program 108 may retrieve presence and / or location data from entries corresponding to items with names and / or descriptions that match the lost item. If the items are not present on-site, there may be no location data to retrieve.

[0063] At 308, in response to determining that the lost item is no longer present within the site based on the presence status, the household goods inventory management program 108 may notify the user that the item is no longer present within the site. If the presence data indicates that the lost item is no longer within the site, the household goods inventory management program 108 may notify the user that the item is no longer within the site, for example, through a graphical, textual, or audible prompt. In some embodiments, the household goods inventory management program 108 may notify the user of the item's last location within the sensor feed, the route the item took to leave the site, the item's movement pattern, etc., by displaying a 3D model to the user. The household goods inventory management program 108 may notify the user of the last time the item was within the site and / or the date and / or time the item left the site. In some embodiments, the household goods inventory management program 108 may offer to display the item's last known location to the user in mixed reality and may prompt the user to start a mixed reality session. If the user is in a mixed reality session, the household goods inventory control program 108 may display the most recent record taken while the user was in the previous mixed reality session in which the item was visible.

[0064] At 310, the household goods inventory management program 108 may, in response to determining that a lost item is present within the site, direct the user to the lost item. The household goods inventory management program 108 may direct the user to the item by highlighting or otherwise indicating the item by overlaying a graphical element on the item in a mixed reality session currently being experienced by the user. The household goods inventory management program 108 may prompt the user to start a mixed reality session on the user's mixed reality-enabled device, such as a tablet or mobile phone, and may indicate the missing object in the mixed reality session with a graphical element overlaid on the object's location. In some embodiments of the present invention, the household goods inventory management program 108 may direct the user to the item by auditorily or textually describing the location of the lost item to the user. In some embodiments, for example, if there is a lost item or the household goods inventory control program 108 has not determined that an item has left the scene but does not currently know the location of the item within the scene, the household goods inventory control program 108 may offer to display the last known location of the item to the user within the mixed reality environment and may prompt the user to begin a mixed reality session. If the user is within the mixed reality session, the household goods inventory control program 108 may display the most recent video recording taken while the user was within the mixed reality session prior to the item being visible. In some embodiments of the present invention, the household goods inventory control program 108 may guide the user to the item by remotely operating an autonomous device, such as a small remote-controlled car, a lightweight drone, or a robotic dog, to guide the user along a path to the item and present the item to the user.

[0065] It can be appreciated that Figures 2-3 are intended to be illustrative of particular implementations only and are not intended to imply any limitations on how different embodiments may be implemented. Many modifications to the depicted environments may be made based on design and implementation requirements.

[0066] The description of various embodiments of the present invention has been presented for illustrative purposes and is not intended to be exhaustive or limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope of the described embodiments. The terms used herein have been selected to best explain the principles, practical applications, or technical improvements of the embodiments over the art found in the market, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. 1. A processor-implemented method for item management, comprising: identifying all items in the sensor feeds of the sensor sets in the scene; creating a list containing entries for the identified items, the entries including the name, location and existence of the items; dynamically updating the list to track the location and / or the presence of the items in real time based on continuous monitoring of the sensor feed; In response to receiving a query from a user requesting a lost item, selecting an item in the list that matches the name of the lost item; and directing the user to the location of the selected item; A method comprising:

2. The step of updating comprises: updating the presence of one of the items in response to detecting that the item has crossed a geofence defining a perimeter of the site. further comprising The method of claim 1.

3. The method of claim 1 , wherein the item is less than a threshold size and a threshold weight.

4. directing the user to the location of the selected item, highlighting the location of the selected item with a graphical element in a mixed reality environment. further comprising The method of claim 1.

5. displaying a recorded mixed reality session in response to the lost item being lost, wherein the lost item is located in a location last seen by the user. The method of claim 1 further comprising:

6. assigning a priority or loss probability to said items; and wherein the updating step is based on the priority or the loss rate of the item. The method of claim 1.

7. prompting the user to identify the location of one of the items in response to the location of the item being lost for a threshold period of time. The method of claim 1 further comprising:

8. A computer system for item management, comprising: one or more processors, one or more computer-readable memories, one or more computer-readable tangible storage media, and program instructions stored on at least one of the one or more tangible storage media for execution by at least one of the one or more processors via at least one of the one or more memories. Equipped with wherein the computer system is identifying all items in the sensor feeds of the sensor sets in the scene; creating a list containing entries for the identified items, the entries including the name, location and existence of the items; dynamically updating the list to track the location and / or the presence of the items in real time based on continuous monitoring of the sensor feed; In response to receiving a query from a user requesting a lost item, selecting an item in the list that matches the name of the lost item; and directing the user to the location of the selected item; A computer system capable of performing a method including:

9. The step of updating comprises: updating the presence of one of the items in response to detecting that the item has crossed a geofence defining a perimeter of the site. further comprising:

9. The computer system of claim 8.

10. The computer system of claim 8 , wherein the item is less than a threshold size and a threshold weight.

11. directing the user to the location of the selected item, highlighting the location of the selected item with a graphical element in a mixed reality environment. further comprising:

9. The computer system of claim 8.

12. displaying a recorded mixed reality session in response to the lost item being lost, wherein the lost item is located in a location last seen by the user. The computer system of claim 8 further comprising:

13. assigning a priority or loss probability to said items; and wherein the updating step is based on the priority or the loss rate of the item.

9. The computer system of claim 8.

14. prompting the user to identify the location of one of the items in response to the location of the item being lost for a threshold period of time. The computer system of claim 8 further comprising:

15. 1. A computer program product for item management, comprising: one or more computer-readable tangible storage media and program instructions stored on at least one of the one or more tangible storage media; Equipped with The program instructions include: identifying all items in the sensor feeds of the sensor sets in the scene; creating a list containing entries for the identified items, the entries including the name, location and existence of the items; dynamically updating the list to track the location and / or the presence of the items in real time based on continuous monitoring of the sensor feed; In response to receiving a query from a user requesting a lost item, selecting an item in the list that matches the name of the lost item; and directing the user to the location of the selected item; Executable by a processor to cause the processor to perform a method including: Computer program products.

16. The step of updating comprises: updating the presence of one of the items in response to detecting that the item has crossed a geofence defining a perimeter of the site. further comprising:

16. A computer program product according to claim 15.

17. 16. The computer program product of claim 15, wherein the item is less than a threshold size and a threshold weight.

18. directing the user to the location of the selected item, highlighting the location of the selected item with a graphical element in a mixed reality environment. further comprising:

16. A computer program product according to claim 15.

19. displaying a recorded mixed reality session in response to the lost item being lost, wherein the lost item is located in a location last seen by the user.

16. The computer program product of claim 15, further comprising:

20. assigning a priority or loss probability to said items; and wherein the updating step is based on the priority or the loss rate of the item.

16. A computer program product according to claim 15.

Images