system

Abstract

We provide the system. [Solution] An information acquisition device for obtaining the position of an object, A data storage device for saving acquired location information to a database in the cloud, A search device for retrieving object location information from a cloud-based database and reporting it to the user, A voice recognition device for analyzing user voice commands, A system including monitoring devices for monitoring the movement of objects and detecting anomalies.

Description

Technical Field

[0001] The technology of the present disclosure relates to a system.

Background Art

[0002] Patent Document 1 discloses a persona chatbot control method performed by at least one processor, including steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to an explanation of a chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In modern times, as many people lead busy daily lives, it has become difficult to remember the positions of household items or manage them efficiently. In particular, it is even more difficult for the elderly and disabled to manage the positions of things, and they often spend a lot of time and effort finding the necessary items. Furthermore, when household items are inappropriately moved, it is impossible to immediately grasp this, so it is also insufficient as a security measure. It is an object to solve such problems.

Means for Solving the Problems

[0005] According to the present invention, a system is provided that allows users to constantly know the location of items in their home by equipping an information acquisition device for obtaining the location of an object and storing this information in a cloud-based database. Furthermore, by equipping a search device that analyzes user queries using a voice recognition device and instantly searches for and reports the current location of an item, users can check the location of an item simply by asking, "Where is XX?" In addition, a means of enhancing security measures is provided by using a monitoring device to detect abnormal movement of an object and issuing a warning to the user. This improves the management and security of items.

[0006] An "information acquisition device" is a device that accurately captures the current position of an object and provides that position information to other devices or systems.

[0007] A "data storage device" is a device that retains acquired information over a long period of time and stores it in a format that can be accessed as needed.

[0008] A "database" is a collection of digital information designed to store information efficiently and systematically, and to allow for searching and updating.

[0009] A "search device" is a device or function that quickly retrieves specific data from a database and provides it to the user.

[0010] A "voice recognition device" is a device that converts the voice spoken by a user into digital data, analyzes its content, and processes it as information.

[0011] A "monitoring device" is a device that observes changes in objects and the environment in real time and uses that information to detect anomalies or events. [Brief explanation of the drawing]

[0012] [Figure 1] This is a conceptual diagram showing an example of the configuration of a data processing system according to the first embodiment. [Figure 2] This is a conceptual diagram showing an example of the essential functions of a data processing device and a smart device according to the first embodiment. [Figure 3] This is a conceptual diagram showing an example of the configuration of a data processing system according to the second embodiment. [Figure 4] This is a conceptual diagram showing an example of the main functions of a data processing device and smart glasses according to the second embodiment. [Figure 5] This is a conceptual diagram showing an example of the configuration of a data processing system according to the third embodiment. [Figure 6] This is a conceptual diagram showing an example of the main functions of a data processing device and a headset-type terminal according to the third embodiment. [Figure 7] This is a conceptual diagram showing an example of the configuration of a data processing system according to the fourth embodiment. [Figure 8] This is a conceptual diagram showing an example of the main functions of a data processing device and a robot according to the fourth embodiment. [Figure 9] This shows an emotion map where multiple emotions are mapped. [Figure 10] This shows an emotion map where multiple emotions are mapped. [Figure 11] This is a sequence diagram showing the processing flow of the data processing system in Example 1. [Figure 12] This is a sequence diagram showing the processing flow of the data processing system in Application Example 1. [Figure 13] This is a sequence diagram showing the processing flow of the data processing system in Example 2, which incorporates an emotion engine. [Figure 14] This is a sequence diagram showing the processing flow of the data processing system in Application Example 2, which combines an emotion engine. [Modes for carrying out the invention]

[0013] Hereinafter, an example of an embodiment of the system relating to the technology of this disclosure will be described with reference to the attached drawings.

[0014] First, the terms used in the following description will be explained.

[0015] In the following embodiments, the numbered processor (hereinafter simply referred to as "processor") may be a single arithmetic unit or a combination of multiple arithmetic units. Also, the processor may be a single type of arithmetic unit or a combination of multiple types of arithmetic units. Examples of arithmetic units include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), an APU (Accelerated Processing Unit), and the like.

[0016] In the following embodiments, the numbered RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a work memory by the processor.

[0017] In the following embodiments, the numbered storage is one or more non-volatile storage devices that store various programs and various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), or magnetic tapes, and the like.

[0018] In the following embodiments, the numbered communication I / F (Interface) is an interface including a communication processor and an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), or Bluetooth (registered trademark), and the like.

[0019] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." That is, "A and / or B" means that it may be A alone, or B alone, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" applies when expressing three or more things linked by "and / or."

[0020] [First Embodiment]

[0021] Figure 1 shows an example of the configuration of the data processing system 10 according to the first embodiment.

[0022] As shown in Figure 1, the data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0023] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0024] The smart device 14 comprises a computer 36, a reception device 38, an output device 40, a camera 42, and a communication interface 44. The computer 36 comprises a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The reception device 38, output device 40, and camera 42 are also connected to the bus 52.

[0025] The reception device 38 is equipped with a touch panel 38A and a microphone 38B, etc., and receives user input. The touch panel 38A receives user input by detecting contact with an object (e.g., a pen or finger). The microphone 38B receives user input by detecting the user's voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0026] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form perceptible to the user 20 (e.g., audio and / or text). The display 40A displays visible information such as text and images according to instructions from the processor 46. The speaker 40B outputs audio according to instructions from the processor 46. The camera 42 is a small digital camera equipped with an optical system such as a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0027] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various types of information between processor 46 and processor 28 via network 54.

[0028] Figure 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0029] As shown in Figure 2, in the data processing device 12, a specific processing is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" related to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0030] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0031] In the smart device 14, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The reception output program 60 is used in conjunction with a specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0032] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0033] This invention provides a system and program for streamlining the management of items placed in a household. The embodiments thereof are described below.

[0034] This system determines the current location of objects through an information acquisition device that obtains the object's position in real time. The information acquisition device works in conjunction with sensors and cameras placed in each room to monitor the movement of objects. The acquired location information is stored in a cloud-based database via a data storage device and can be accessed by the user when needed.

[0035] The terminal receives voice commands from the user and converts them into text via a voice recognition device. If the user inquires about the location of a specific item, the terminal sends the command to the server. The server searches the database and retrieves the location information of the item. It then returns the location information to the terminal, which reports the item's location to the user via a visual map.

[0036] Furthermore, the server has the ability to analyze the item placement information in the database and suggest the optimal storage method. This allows users to organize and store their belongings efficiently. For example, the server may suggest rearranging kitchen items that tend to get cluttered.

[0037] Furthermore, the system includes a security feature that constantly monitors the movement of items using surveillance devices. In particular, if valuables or important items are moved inappropriately, the server immediately detects the anomaly and sends a warning to the terminal. Users can respond quickly upon receiving this warning.

[0038] Furthermore, the device learns the user's daily behavior patterns and generates reminders based on that information. It can notify the user at the appropriate time to prevent them from forgetting essential items. For example, it might prompt the user to check if they have their keys before leaving the house.

[0039] Thus, this invention significantly improves the efficiency of users' lives and provides a sense of security in the management and security of goods.

[0040] The following describes the processing flow.

[0041] Step 1:

[0042] The user gives a voice command to the device, such as "Where is XX?". The device then uses a speech recognition device to convert this voice into text data.

[0043] Step 2:

[0044] The terminal sends the converted text data to the server. There, the voice instructions are analyzed and the names of specific items are extracted.

[0045] Step 3:

[0046] The server searches for the location information of the received item in a cloud-based database based on the item's name. This search process is performed using the most up-to-date information in the database.

[0047] Step 4:

[0048] The server retrieves the location information of an item and sends it back to the terminal. This location information includes the specific location and coordinates of the item.

[0049] Step 5:

[0050] The device displays the received location information on a visual map. The map is shown on the user's screen, and highlights and icons are used to indicate the precise location of items.

[0051] Step 6:

[0052] The user checks the display on the device, recognizes the location of the selected item, and takes action to retrieve it.

[0053] This processing flow allows users to quickly and efficiently locate items and find what they need without stress.

[0054] (Example 1)

[0055] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0056] In modern living environments, numerous items exist within the home, making it difficult to accurately and efficiently manage their locations. In particular, items frequently go missing, and storage space is often used inefficiently. Furthermore, there are security concerns regarding misuse and theft due to the improper movement of items. Additionally, users often fail to keep track of and forget the items they need in their daily lives.

[0057] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0058] In this invention, the server includes detection means for acquiring the location of an item, recording means for storing the acquired location information on a storage medium on the network, and query means for retrieving the location information of an item from the storage medium on the network and reporting it to the user. This enables accurate tracking of the current location of an item, allowing for suggestions for efficient storage methods, security measures, and reminder functions based on the user's behavior.

[0059] A "detection means" is a device or mechanism that senses the location of an object in real time and acquires that information.

[0060] "Recording means" refers to a device or mechanism for storing the location information of acquired items in a storage device on a network.

[0061] "Inquiry means" refers to a device or mechanism that retrieves location information of an item from a storage device on a network and provides that information to the user.

[0062] "Voice processing means" refers to a device or mechanism that analyzes the user's voice instructions, converts them into text data, and transmits them to another device.

[0063] "Monitoring means" refers to a device or mechanism that constantly monitors the movement of an object and detects inappropriate movement or abnormalities.

[0064] A "learning tool" is a device or system that analyzes the user's behavior patterns, predicts how an item will be used, and generates reminders.

[0065] A "proposal means" is a device or mechanism that calculates the optimal arrangement and storage method for items and proposes it to the user.

[0066] This invention is a system for streamlining the management and security of household items. Specific embodiments of this system are described below.

[0067] The server functions as a "detection mechanism" to acquire the location of items in real time by working in conjunction with sensors and cameras placed in each room. The hardware used includes common indoor surveillance cameras and location sensors. The information acquired by the sensors and cameras is stored in a cloud-based database and functions as a "recording mechanism".

[0068] The terminal serves as a "voice processing device," receiving voice commands from the user and converting those commands into text using voice recognition software. For example, a voice recognition API is used to analyze voice commands such as "Where are the keys?"

[0069] The server receives these textual instructions and acts as a "query tool" to search a database in the cloud. By obtaining the location information of the item and sending it to the terminal, the user can find out the item's current location.

[0070] Furthermore, the server is equipped with a "suggestion mechanism" that proposes the optimal arrangement and storage method for items based on information in the database. For example, it can suggest the arrangement of kitchen utensils while considering storage efficiency.

[0071] In terms of security, the server acts as a "monitoring tool," monitoring the inappropriate movement of items and immediately sending a warning to the terminal if an anomaly is detected. This function allows users to quickly address problems.

[0072] Furthermore, the device learns the user's behavioral patterns and serves as a "learning tool." This allows it to provide reminders about items that are often forgotten in daily life, supporting the user's lifestyle. For example, it has a function that notifies the user, "Did you bring your keys?" when they leave the house.

[0073] Examples of prompt statements include the following:

[0074] "Please describe your household item management system. Please provide detailed information on how it handles situations where a user asks for the location of an item using voice commands."

[0075] This invention significantly improves the management and security of household items, providing users with peace of mind and convenience.

[0076] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0077] Step 1:

[0078] The user gives voice commands to the device. For example, a command such as "Tell me where the remote control is" might be input. The device uses voice recognition software to convert the input voice signal into text data. In this conversion process, the voice data is converted into a series of digital signals, which are then output as text.

[0079] Step 2:

[0080] The terminal sends the converted text data to the server. This allows the server to receive basic information to understand the user's request. The server analyzes the received text data and identifies the names of the items derived from it. Natural language processing techniques may be used for this analysis.

[0081] Step 3:

[0082] The server searches a database in the cloud to find the location information of the identified item. In this process, the server executes database queries and retrieves the latest location information as output. The output from the database is digital map information about the item's location.

[0083] Step 4:

[0084] The server transmits the acquired location information to the device. In doing so, the server converts the location information into an easily understandable format before sending it. For example, it might use JSON format data to transmit location information to the device.

[0085] Step 5:

[0086] The device uses the received location information to provide the user with visual information. The device uses a graphical user interface to display the current location of an item on a map. The device also outputs a notification, directly displaying the location information to the user.

[0087] Step 6:

[0088] The server analyzes the location data of items stored in the database and proposes the optimal storage method for those items. In this process, the server analyzes the data using statistics and machine learning algorithms. The output generates suggestions for optimizing item placement.

[0089] Step 7:

[0090] The server constantly monitors the movement of items and immediately sends a warning to the terminal if any inappropriate movement is detected. The server's input is real-time data from sensors, and its output is a warning message. This allows the user to respond to problems quickly.

[0091] Step 8:

[0092] The device learns the user's past behavior and generates reminders. In this process, the device takes the user's daily behavior data as input and creates notifications about items that are easily forgotten based on that data. The device provides these reminders to the user in a timely manner.

[0093] (Application Example 1)

[0094] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0095] There is a need to improve the efficiency of managing household items, reduce the risk of losing particularly important items, and promote tidiness and organization in living spaces. Furthermore, providing features such as item reminders based on users' daily behavior and anomaly detection functions for security purposes are also challenges.

[0096] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0097] In this invention, the server includes location detection means for acquiring the location of an object, information storage means for storing the acquired location information in a remote data area, and behavioral analysis and notification means for analyzing the user's daily behavior and reminding them of necessary items. This enables tracking the location of items, optimal item organization, and behavioral reminder notifications.

[0098] "Position detection means" refers to devices or methods for determining the position of an object, and is a technology that acquires information using sensors or cameras.

[0099] "Information retention means" refers to technology that stores acquired data in remote data areas such as storage devices or the cloud, making it accessible when needed.

[0100] "Information retrieval means" refers to a device or procedure that retrieves location information stored in a remote data area at the request of a user and reports the results.

[0101] "Voice interpretation means" refers to technology that analyzes voice instructions from users, interprets them as sentences, and performs appropriate information processing.

[0102] "Dynamic monitoring means" refers to devices and technologies for continuously monitoring the movement of an object and detecting abnormal behavior.

[0103] "Behavioral analysis and notification means" refers to technology that learns the user's daily behavioral patterns and generates appropriate item notifications and reminders based on those patterns.

[0104] The system that realizes this application example streamlines the management of items within the home and improves security. Specifically, a robot placed in the user's home uses location detection means to identify the location of items and stores that information in a cloud-based information storage system. Sensors and cameras are incorporated into the robot to detect surrounding items.

[0105] The server manages information in the cloud, uses information retrieval tools to quickly locate the relevant items, and reports the location to the user via a terminal. The user can give instructions to the robot via voice interpretation tools to find specified items. This function utilizes the Google® Speech-to-Text API to convert speech to text.

[0106] Furthermore, the server uses dynamic monitoring to detect suspicious movement of items within the home and immediately issues a warning to the user's terminal in case of an anomaly. In addition, behavioral analysis and notification means can learn the user's daily behavior and notify them of necessary items as a reminder. In this process, software such as TENSORFLOW® and OpenCV are used to acquire and analyze the user's behavioral patterns.

[0107] For example, to prevent misplacing everyday keys, the robot will ask "Do you have your keys?" when you leave the house. Also, if you ask "Where is the salt?" while cooking, the robot will check its location and quickly report the result.

[0108] An example of a prompt message is as follows:

[0109] "Learn the user's daily behavioral patterns and generate appropriate reminders. Provide specific behavioral scenarios, such as 'Check the items the user needs before leaving the house,' and design reminders based on those behaviors."

[0110] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0111] Step 1:

[0112] The user gives voice commands to the robot to locate an item. The user's voice is used as the voice input, and the device uses the Google Speech-to-Text API to convert the voice commands into text data, which is then sent to the server.

[0113] Step 2:

[0114] The server uses an information retrieval mechanism to search a database in the cloud based on the received text data. The input information for the database is the location information of an item, and the search result retrieves the latest location data for a specific item.

[0115] Step 3:

[0116] The server transmits the acquired location data to the terminal. The terminal displays the location of the item on a visually easy-to-understand map for the user and reports it to the user. The input is the acquired location data, and the output is a visual information display.

[0117] Step 4:

[0118] The server constantly monitors acquired data using dynamic monitoring devices and performs a process to detect anomalies. Based on user-defined criteria, it analyzes location data and, if suspicious movement is detected, generates an anomaly warning and sends it to the terminal. The input is location data, and the output is a warning notification.

[0119] Step 5:

[0120] The user's daily behavior patterns are analyzed on the server side using behavioral analysis and notification methods. Sensor data acquired using TensorFlow, etc., is analyzed to create reminders about items the user needs. For example, when the user leaves the house, a confirmation reminder such as "Did you bring your keys?" is sent. The input is behavioral pattern data, and the output is a reminder notification.

[0121] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0122] This invention provides an information provision system that takes into account the management of items within the home and the emotional state of the user. The embodiments thereof are described below.

[0123] This system includes an information acquisition device for obtaining the location of objects in real time. This device works in conjunction with cameras and sensors within the home to monitor the location and movement of items. The acquired location information is stored in a cloud-based database via a data storage device and can be retrieved as needed.

[0124] When a user gives a voice command to the device, the voice recognition device converts the command into text data and sends it to the server. The server searches a database based on this text data to locate the specified item. Furthermore, the device displays a visual map based on this information and reports the item's location to the user.

[0125] A distinctive feature of this invention is the addition of a function that analyzes the user's emotional state using an emotion engine. The emotion engine recognizes emotions from the user's voice and facial expressions, and adjusts the content and method of reporting based on that state. For example, if the user is feeling stressed, the device will provide information in a gentler voice and with simpler instructions. Specifically, if the device detects that the user is in a hurry, it will immediately highlight important location information to support efficient action.

[0126] Furthermore, the emotion engine adjusts the operation of the suggestion device based on the user's emotions. For example, when the user is feeling down, it offers ideas to refresh their life, such as suggesting the rearrangement of items. In this way, it provides information in a form that takes the user's psychological stability into consideration, realizing a system that is useful for managing items and improving quality of life.

[0127] The following describes the processing flow.

[0128] Step 1:

[0129] The user asks the device by voice, "Where is XX?" The device receives this voice and converts it into text data using a speech recognition device.

[0130] Step 2:

[0131] The terminal sends the converted text data to the server. The server analyzes this data and extracts the names of specific items.

[0132] Step 3:

[0133] The server searches a database in the cloud to retrieve the current location information of the requested item. This information includes location details.

[0134] Step 4:

[0135] The server sends the acquired location information back to the device. The device receives this information and displays it on a map in an easy-to-understand format for the user.

[0136] Step 5:

[0137] The device uses its built-in emotion engine to analyze the user's voice and facial expressions and evaluate their emotional state. This emotional state influences how notifications are delivered in the next step.

[0138] Step 6:

[0139] When the device recognizes the user's emotional state, it adjusts the information it presents accordingly. For example, if the user is stressed, the device will deliver information in a calm voice and in a soothing manner.

[0140] Step 7:

[0141] If the emotion engine detects a stressed state, the server will choose a storage suggestion method that is less burdensome for the user and propose it through the terminal.

[0142] Step 8:

[0143] The user checks information from the device and, based on that, retrieves the desired item or organizes items according to storage suggestions. Through this process, the user can maintain an efficient and emotionally comfortable environment.

[0144] (Example 2)

[0145] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0146] Managing household items is often cumbersome, and users frequently lose track of their belongings. Furthermore, the quality of information provided to users needs to change depending on their emotional state. These challenges necessitate a system that efficiently manages items while maintaining the user's psychological well-being.

[0147] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0148] In this invention, the server includes detection means for acquiring the location of an object, storage means for storing the acquired location information on a storage medium on a remote server, and emotion analysis means for analyzing the user's emotional state and adjusting the content and method of reporting. This enables efficient management of household items and the provision of information tailored to the user's emotional state.

[0149] "Detection means" refers to a device or method arranged to acquire the position of an object, and which has the function of collecting accurate positional information of the object.

[0150] "Storage means" refers to a device or method for temporarily or permanently storing acquired information or data on a storage medium on a remote server.

[0151] "Search means" refers to a device or method for analyzing data stored on a storage medium, extracting necessary information, and providing it to the user.

[0152] "Analysis means" refers to a device or method for analyzing user instructions and data and converting them into necessary information.

[0153] "Emotional analysis means" refers to a device or method for analyzing a user's voice, facial expressions, etc., to understand their emotional state and adjust the content and method of reporting accordingly.

[0154] "Monitoring means" refers to a device or method for continuously observing the movement or state of an object and detecting any abnormalities.

[0155] A "proposal means" is a device or method that presents the user with the most appropriate action or option depending on the situation.

[0156] "Learning means" refers to a device or method for continuously observing a user's behavioral patterns and predicting future behavior based on that data.

[0157] The embodiments for carrying out the present invention are described below.

[0158] This system utilizes a combination of devices and technologies to manage household items and provide information based on the user's emotional state. Specifically, it employs the following hardware and software:

[0159] The server is the central hub for integrated management of information within the home. Located on a remote server, it functions as a data storage medium. When a voice command is received from a user via a terminal, the server processes it using analysis tools and retrieves the location information of necessary items from its database.

[0160] The server also analyzes the user's voice and facial expressions using emotion analysis tools. This is done through speech recognition and facial expression analysis software on the user's device. For example, if the server identifies that the user is stressed, it adjusts the information provided accordingly.

[0161] The terminal functions as an interface with the user. It is equipped with a voice recognition device that converts voice commands uttered by the user into text data. This information is sent to a server, and the information returned from the server is displayed visually. A visual map is displayed on the terminal, reporting the location of specific items to the user.

[0162] Users operate the system by giving voice commands to the device. For example, by asking "Where is the remote control?", the system can instantly obtain its location information. Furthermore, the system analyzes the user's emotional state and provides more appropriate information based on that analysis.

[0163] As a concrete example of this operation, if a user is in a hurry and gives a voice command such as "Where is it?", the server will quickly obtain the latest location information, and the device will highlight it and report it to the user.

[0164] An example of a prompt is, "Consider an approach to quickly and gently provide location information for everyday items when a user is experiencing emotional stress." Based on this prompt, the generative AI model generates an appropriate method of providing information.

[0165] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0166] Step 1:

[0167] The user gives voice commands about an item to the terminal. This voice is converted into text data by the terminal's voice recognition device. The input is the user's voice data, and the output is generated as text data. For example, if the user says "Find the remote control," the terminal converts this into text data.

[0168] Step 2:

[0169] The terminal sends the converted text data to the server. The server receives this text data as input and searches a database in the cloud. To obtain the location information of a specific item, the server analyzes the text data and extracts relevant information using search tools. The output is the location information of the specified item. Specifically, at this stage, the server queries the database based on the instruction "find the remote control" and locates the remote control.

[0170] Step 3:

[0171] The server transmits the acquired location information to the terminal. The terminal receives this location information as input and creates a visual map. The visual output clearly displays the location of the items. Specifically, the terminal displays a map with the remote control's location highlighted.

[0172] Step 4:

[0173] To provide information based on the user's emotional state, the device utilizes emotion analysis tools along with location information from the server. The server takes the user's emotion data as input and determines the appropriate reporting method. For example, if the device determines that the user is in a hurry, it will highlight important location information and provide guidance in a gentle voice. The output is a personalized report.

[0174] Step 5:

[0175] If necessary, the server generates suggestions for managing and rearranging items using suggestion mechanisms. These suggestions are based on the user's current state and past behavioral patterns. Input is the user's emotional state and behavioral pattern data, and output is specific suggestions. A specific action would be to display rearrangement suggestions when the user is feeling stressed.

[0176] (Application Example 2)

[0177] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as a "server" and the smart device 14 as a "terminal".

[0178] When managing the location of items within the home, current systems provide information without considering the user's emotional state, resulting in a failure to alleviate the user's psychological burden. In particular, there is a need for information that not only identifies the location of items but also takes the user's feelings into consideration. Furthermore, improving information provision based on the user's emotions is a crucial element in improving the quality of life within the home.

[0179] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0180] In this invention, the server includes data acquisition means for obtaining the location of an object, data storage means for storing the acquired location data in a cloud storage device, search means for searching for the location data of an object from the cloud storage device and reporting it to the user, voice recognition means for analyzing the user's voice instructions, monitoring means for monitoring the movement of an object and detecting anomalies, and emotion analysis means for analyzing the user's emotional state and adjusting the information provision method. This enables more efficient management of items within the home and comfortable information provision tailored to the user's emotions.

[0181] A "data acquisition means" is a device that detects the position of an object and collects the necessary data.

[0182] A "data storage means" is a device that securely stores and manages collected data in a cloud-based storage device.

[0183] A "search method" is a device that searches for specific location data from storage devices in the cloud and reports it to the user through visual or audio means.

[0184] A "voice recognition means" is a device that analyzes the user's voice and converts the instructions into text data.

[0185] A "monitoring device" is a device that continuously tracks the movement of an object and detects any anomalies.

[0186] An "emotion analysis device" is a device that determines emotions from the user's voice and facial expressions, and improves the method of providing information based on the results.

[0187] To implement this invention, a system is utilized that manages household items and provides information while taking into account the user's emotional state. Specifically, it is configured as follows:

[0188] The server is equipped with data acquisition means and uses cameras and sensors installed in the home to acquire object location data in real time. This information is securely stored in cloud storage via data storage means. The data in the cloud can be quickly retrieved using search means as needed and the information is provided to the terminal.

[0189] The terminal uses voice recognition to accurately analyze the user's voice instructions. The analyzed voice instructions are converted into text data and used for location retrieval. After this information is obtained, the terminal uses search tools to visually display the object's location and provides an audio report. Furthermore, it uses emotion analysis tools to analyze the user's voice and facial expressions, providing flexible information tailored to the user's emotional state.

[0190] When a user searches for their keys at home, if they say something like, "I'm looking for my keys," the system will display the key's location in real time. If the user is feeling stressed, it will also gently inform them, "They're on the table by the front door."

[0191] An example of a prompt message is, "Real-time location of household items and gentle voice guidance tailored to the user's feelings." Through this system, the aim is to significantly improve the convenience of managing household items and make users' lives richer and more comfortable.

[0192] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0193] Step 1:

[0194] The server acquires data from cameras and sensors placed within the home. Its inputs are camera video data and sensor information, which it analyzes and outputs as object location data. Specifically, it uses image processing technology to analyze the shape and color of objects and determines the location of specific items.

[0195] Step 2:

[0196] The server saves the acquired object location data to a cloud storage device using a data storage mechanism. The input here is the location data obtained in step 1, and the output is the updated location information in the cloud database. During saving, the data is organized and compressed to enable efficient searching.

[0197] Step 3:

[0198] The user gives voice commands to the device. The input is the user's voice commands, which are converted into text data by a speech recognition system. Specifically, the voice waveform is converted into digital data, its acoustic characteristics are analyzed, and the content of the command is converted into a string of characters.

[0199] Step 4:

[0200] The terminal sends text data to the server and uses a search mechanism to retrieve the location information of the target object from a cloud-based recording device. The input here is a text-based instruction, and the output is the object's location information retrieved from the cloud. The server automatically queries the database and performs calculations to extract relevant information.

[0201] Step 5:

[0202] The device evaluates the user's emotional state using emotion analysis tools. Input is additional voice or facial expression data from the user, and based on this, it outputs an emotional state (relaxed, stressed, etc.). It performs processing to calculate multiple emotional indicators using voice tone analysis and facial expression recognition technology.

[0203] Step 6:

[0204] The terminal provides appropriate voice guidance, taking into account the location information of the acquired objects and the user's emotional state. The input is the information obtained in steps 4 and 5, and the output is the voice and visual guidance provided to the user. Specifically, voice guidance is generated using speech synthesis technology, and a visual map is displayed on the screen as needed.

[0205] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0206] Data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of data generation model 58 is ChatGPT (registered trademark) (Internet search).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0207] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart device 14.

[0208] [Second Embodiment]

[0209] Figure 3 shows an example of the configuration of the data processing system 210 according to the second embodiment.

[0210] As shown in Figure 3, the data processing system 210 includes a data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0211] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0212] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication interface 44. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, and camera 42 are also connected to the bus 52.

[0213] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0214] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0215] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0216] Figure 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Figure 4, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0217] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0218] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0219] In the smart glasses 214, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0220] Next, the identification processing performed by the identification processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0221] This invention provides a system and program for streamlining the management of items placed in a household. The embodiments thereof are described below.

[0222] This system determines the current location of objects through an information acquisition device that obtains the object's position in real time. The information acquisition device works in conjunction with sensors and cameras placed in each room to monitor the movement of objects. The acquired location information is stored in a cloud-based database via a data storage device and can be accessed by the user when needed.

[0223] The terminal receives voice commands from the user and converts them into text via a voice recognition device. If the user inquires about the location of a specific item, the terminal sends the command to the server. The server searches the database and retrieves the location information of the item. It then returns the location information to the terminal, which reports the item's location to the user via a visual map.

[0224] Furthermore, the server has the ability to analyze the item placement information in the database and suggest the optimal storage method. This allows users to organize and store their belongings efficiently. For example, the server may suggest rearranging kitchen items that tend to get cluttered.

[0225] Furthermore, the system includes a security feature that constantly monitors the movement of items using surveillance devices. In particular, if valuables or important items are moved inappropriately, the server immediately detects the anomaly and sends a warning to the terminal. Users can respond quickly upon receiving this warning.

[0226] Furthermore, the device learns the user's daily behavior patterns and generates reminders based on that information. It can notify the user at the appropriate time to prevent them from forgetting essential items. For example, it might prompt the user to check if they have their keys before leaving the house.

[0227] Thus, this invention significantly improves the efficiency of users' lives and provides a sense of security in the management and security of goods.

[0228] The following describes the processing flow.

[0229] Step 1:

[0230] The user gives a voice command to the device, such as "Where is XX?". The device then uses a speech recognition device to convert this voice into text data.

[0231] Step 2:

[0232] The terminal sends the converted text data to the server. There, the voice instructions are analyzed and the names of specific items are extracted.

[0233] Step 3:

[0234] The server searches for the location information of the received item in a cloud-based database based on the item's name. This search process is performed using the most up-to-date information in the database.

[0235] Step 4:

[0236] The server retrieves the location information of an item and sends it back to the terminal. This location information includes the specific location and coordinates of the item.

[0237] Step 5:

[0238] The device displays the received location information on a visual map. The map is shown on the user's screen, and highlights and icons are used to indicate the precise location of items.

[0239] Step 6:

[0240] The user checks the display on the device, recognizes the location of the selected item, and takes action to retrieve it.

[0241] This processing flow allows users to quickly and efficiently locate items and find what they need without stress.

[0242] (Example 1)

[0243] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0244] In modern living environments, numerous items exist within the home, making it difficult to accurately and efficiently manage their locations. In particular, items frequently go missing, and storage space is often used inefficiently. Furthermore, there are security concerns regarding misuse and theft due to the improper movement of items. Additionally, users often fail to keep track of and forget the items they need in their daily lives.

[0245] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0246] In this invention, the server includes detection means for acquiring the location of an item, recording means for storing the acquired location information on a storage medium on the network, and query means for retrieving the location information of an item from the storage medium on the network and reporting it to the user. This enables accurate tracking of the current location of an item, allowing for suggestions for efficient storage methods, security measures, and reminder functions based on the user's behavior.

[0247] A "detection means" is a device or mechanism that senses the location of an object in real time and acquires that information.

[0248] "Recording means" refers to a device or mechanism for storing the location information of acquired items in a storage device on a network.

[0249] "Inquiry means" refers to a device or mechanism that retrieves location information of an item from a storage device on a network and provides that information to the user.

[0250] "Voice processing means" refers to a device or mechanism that analyzes the user's voice instructions, converts them into text data, and transmits them to another device.

[0251] "Monitoring means" refers to a device or mechanism that constantly monitors the movement of an object and detects inappropriate movement or abnormalities.

[0252] A "learning tool" is a device or system that analyzes the user's behavior patterns, predicts how an item will be used, and generates reminders.

[0253] A "proposal means" is a device or mechanism that calculates the optimal arrangement and storage method for items and proposes it to the user.

[0254] This invention is a system for streamlining the management and security of household items. Specific embodiments of this system are described below.

[0255] The server functions as a "detection mechanism" to acquire the location of items in real time by working in conjunction with sensors and cameras placed in each room. The hardware used includes common indoor surveillance cameras and location sensors. The information acquired by the sensors and cameras is stored in a cloud-based database and functions as a "recording mechanism".

[0256] The terminal serves as a "voice processing device," receiving voice commands from the user and converting those commands into text using voice recognition software. For example, a voice recognition API is used to analyze voice commands such as "Where are the keys?"

[0257] The server receives these textual instructions and acts as a "query tool" to search a database in the cloud. By obtaining the location information of the item and sending it to the terminal, the user can find out the item's current location.

[0258] Furthermore, the server is equipped with a "suggestion mechanism" that proposes the optimal arrangement and storage method for items based on information in the database. For example, it can suggest the arrangement of kitchen utensils while considering storage efficiency.

[0259] In terms of security, the server acts as a "monitoring tool," monitoring the inappropriate movement of items and immediately sending a warning to the terminal if an anomaly is detected. This function allows users to quickly address problems.

[0260] Furthermore, the device learns the user's behavioral patterns and serves as a "learning tool." This allows it to provide reminders about items that are often forgotten in daily life, supporting the user's lifestyle. For example, it has a function that notifies the user, "Did you bring your keys?" when they leave the house.

[0261] Examples of prompt statements include the following:

[0262] "Please describe your household item management system. Please provide detailed information on how it handles situations where a user asks for the location of an item using voice commands."

[0263] This invention significantly improves the management and security of household items, providing users with peace of mind and convenience.

[0264] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0265] Step 1:

[0266] The user gives voice commands to the device. For example, a command such as "Tell me where the remote control is" might be input. The device uses voice recognition software to convert the input voice signal into text data. In this conversion process, the voice data is converted into a series of digital signals, which are then output as text.

[0267] Step 2:

[0268] The terminal sends the converted text data to the server. This allows the server to receive basic information to understand the user's request. The server analyzes the received text data and identifies the names of the items derived from it. Natural language processing techniques may be used for this analysis.

[0269] Step 3:

[0270] The server searches a database in the cloud to find the location information of the identified item. In this process, the server executes database queries and retrieves the latest location information as output. The output from the database is digital map information about the item's location.

[0271] Step 4:

[0272] The server transmits the acquired location information to the device. In doing so, the server converts the location information into an easily understandable format before sending it. For example, it might use JSON format data to transmit location information to the device.

[0273] Step 5:

[0274] The device uses the received location information to provide the user with visual information. The device uses a graphical user interface to display the current location of an item on a map. The device also outputs a notification, directly displaying the location information to the user.

[0275] Step 6:

[0276] The server analyzes the location data of items stored in the database and proposes the optimal storage method for those items. In this process, the server analyzes the data using statistics and machine learning algorithms. The output generates suggestions for optimizing item placement.

[0277] Step 7:

[0278] The server constantly monitors the movement of items and immediately sends a warning to the terminal if any inappropriate movement is detected. The server's input is real-time data from sensors, and its output is a warning message. This allows the user to respond to problems quickly.

[0279] Step 8:

[0280] The terminal learns the user's past actions and generates reminders. In this process, the terminal takes the user's daily behavior data as input and creates notifications regarding easily forgotten items based on this data. The terminal provides this reminder to the user in a timely manner.

[0281] (Application Example 1)

[0282] Next, Application Example 1 will be described. In the following description, the data processing device 12 is referred to as the "server", and the smart glasses 214 are referred to as the "terminal".

[0283] There is a demand to improve the management efficiency of items within a home, particularly reduce the risk of loss of important items, and promote the tidying up of the living space. Also, providing a reminder function for items based on the user's daily actions and an anomaly detection function as a security measure are also issues.

[0284] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following respective means.

[0285] In this invention, the server includes a position detection means for acquiring the position of an object, an information holding means for storing the acquired position information in a remote data area, and a behavior analysis and notification means for analyzing the user's daily behavior and reminding the user of necessary items. Thereby, it becomes possible to grasp the location of objects, perform optimal item organization, and provide reminder notifications according to behavior.

[0286] The "position detection means" refers to a device or method for specifying the position of an object, and is a technology for acquiring information using sensors or cameras.

[0287] The "information holding means" is a technology for storing the acquired data in a remote data area such as a storage device or the cloud and making it retrievable when necessary.

[0288] "Information retrieval means" refers to a device or procedure that retrieves location information stored in a remote data area at the request of a user and reports the results.

[0289] "Voice interpretation means" refers to technology that analyzes voice instructions from users, interprets them as sentences, and performs appropriate information processing.

[0290] "Dynamic monitoring means" refers to devices and technologies for continuously monitoring the movement of an object and detecting abnormal behavior.

[0291] "Behavioral analysis and notification means" refers to technology that learns the user's daily behavioral patterns and generates appropriate item notifications and reminders based on those patterns.

[0292] The system that realizes this application example streamlines the management of items within the home and improves security. Specifically, a robot placed in the user's home uses location detection means to identify the location of items and stores that information in a cloud-based information storage system. Sensors and cameras are incorporated into the robot to detect surrounding items.

[0293] The server manages information in the cloud, uses information retrieval tools to quickly locate the relevant items, and reports the location to the user via a terminal. The user can give instructions to the robot via voice interpretation tools to find specified items. This function utilizes the Google Speech-to-Text API to convert speech to text.

[0294] Furthermore, the server uses dynamic monitoring to detect suspicious movement of items within the home and immediately issues a warning to the user's terminal in case of an anomaly. In addition, behavioral analysis and notification methods allow the system to learn the user's daily behavior and notify them of necessary items as a reminder. Software such as TensorFlow and OpenCV are used to acquire and analyze the user's behavioral patterns.

[0295] For example, to prevent misplacing everyday keys, the robot will ask "Do you have your keys?" when you leave the house. Also, if you ask "Where is the salt?" while cooking, the robot will check its location and quickly report the result.

[0296] An example of a prompt message is as follows:

[0297] "Learn the user's daily behavioral patterns and generate appropriate reminders. Provide specific behavioral scenarios, such as 'Check the items the user needs before leaving the house,' and design reminders based on those behaviors."

[0298] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0299] Step 1:

[0300] The user gives voice commands to the robot to locate an item. The user's voice is used as the voice input, and the device uses the Google Speech-to-Text API to convert the voice commands into text data, which is then sent to the server.

[0301] Step 2:

[0302] The server uses an information retrieval mechanism to search a database in the cloud based on the received text data. The input information for the database is the location information of an item, and the search result retrieves the latest location data for a specific item.

[0303] Step 3:

[0304] The server transmits the acquired location data to the terminal. The terminal displays the location of the item on a visually easy-to-understand map for the user and reports it to the user. The input is the acquired location data, and the output is a visual information display.

[0305] Step 4:

[0306] The server constantly monitors the acquired data using dynamic monitoring means and performs a process of detecting abnormalities. Based on the criteria set by the user, it analyzes the position data and generates an anomaly warning and sends it to the terminal when detecting suspicious movements. The input is position information data, and the output is a warning notification.

[0307] Step 5:

[0308] The user's daily behavior patterns are analyzed on the server side by the behavior analysis and notification means. It analyzes the sensor data acquired using TensorFlow or the like and creates a reminder regarding the items required by the user. In a specific example, a confirmation reminder "Did you bring the keys?" is sent when the user leaves home. The input is behavior pattern data, and the output is a reminder notification.

[0309] Furthermore, an emotion engine for estimating the user's emotions may be combined. That is, the specific processing unit 290 may estimate the user's emotions using the emotion identification model 59 and perform specific processing using the user's emotions.

[0310] The present invention provides an information providing system that takes into account the management of items within a home and the emotional state of the user. The embodiments thereof will be described below.

[0311] This system includes an information acquisition device for acquiring the position of an object in real time. The information acquisition device cooperates with cameras and sensors within the home to monitor the position and movement of items. Also, the acquired position information is stored in a database on the cloud via a data storage device and retrieved as needed.

[0312] When the user gives a voice instruction to the terminal, the voice recognition device converts the instruction into text data and sends it to the server. The server searches the database based on this text data to identify the position of the specified item. Furthermore, based on this information, the terminal displays a visual map and reports the position of the item to the user.

[0313] A distinctive feature of this invention is the addition of a function that analyzes the user's emotional state using an emotion engine. The emotion engine recognizes emotions from the user's voice and facial expressions, and adjusts the content and method of reporting based on that state. For example, if the user is feeling stressed, the device will provide information in a gentler voice and with simpler instructions. Specifically, if the device detects that the user is in a hurry, it will immediately highlight important location information to support efficient action.

[0314] Furthermore, the emotion engine adjusts the operation of the suggestion device based on the user's emotions. For example, when the user is feeling down, it offers ideas to refresh their life, such as suggesting the rearrangement of items. In this way, it provides information in a form that takes the user's psychological stability into consideration, realizing a system that is useful for managing items and improving quality of life.

[0315] The following describes the processing flow.

[0316] Step 1:

[0317] The user asks the device by voice, "Where is XX?" The device receives this voice and converts it into text data using a speech recognition device.

[0318] Step 2:

[0319] The terminal sends the converted text data to the server. The server analyzes this data and extracts the names of specific items.

[0320] Step 3:

[0321] The server searches a database in the cloud to retrieve the current location information of the requested item. This information includes location details.

[0322] Step 4:

[0323] The server sends the acquired location information back to the device. The device receives this information and displays it on a map in an easy-to-understand format for the user.

[0324] Step 5:

[0325] The device uses its built-in emotion engine to analyze the user's voice and facial expressions and evaluate their emotional state. This emotional state influences how notifications are delivered in the next step.

[0326] Step 6:

[0327] When the device recognizes the user's emotional state, it adjusts the information it presents accordingly. For example, if the user is stressed, the device will deliver information in a calm voice and in a soothing manner.

[0328] Step 7:

[0329] If the emotion engine detects a stressed state, the server will choose a storage suggestion method that is less burdensome for the user and propose it through the terminal.

[0330] Step 8:

[0331] The user checks information from the device and, based on that, retrieves the desired item or organizes items according to storage suggestions. Through this process, the user can maintain an efficient and emotionally comfortable environment.

[0332] (Example 2)

[0333] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0334] Managing household items is often cumbersome, and users frequently lose track of their belongings. Furthermore, the quality of information provided to users needs to change depending on their emotional state. These challenges necessitate a system that efficiently manages items while maintaining the user's psychological well-being.

[0335] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0336] In this invention, the server includes detection means for acquiring the location of an object, storage means for storing the acquired location information on a storage medium on a remote server, and emotion analysis means for analyzing the user's emotional state and adjusting the content and method of reporting. This enables efficient management of household items and the provision of information tailored to the user's emotional state.

[0337] "Detection means" refers to a device or method arranged to acquire the position of an object, and which has the function of collecting accurate positional information of the object.

[0338] "Storage means" refers to a device or method for temporarily or permanently storing acquired information or data on a storage medium on a remote server.

[0339] "Search means" refers to a device or method for analyzing data stored on a storage medium, extracting necessary information, and providing it to the user.

[0340] "Analysis means" refers to a device or method for analyzing user instructions and data and converting them into necessary information.

[0341] "Emotional analysis means" refers to a device or method for analyzing a user's voice, facial expressions, etc., to understand their emotional state and adjust the content and method of reporting accordingly.

[0342] "Monitoring means" refers to a device or method for continuously observing the movement or state of an object and detecting any abnormalities.

[0343] A "proposal means" is a device or method that presents the user with the most appropriate action or option depending on the situation.

[0344] "Learning means" refers to a device or method for continuously observing a user's behavioral patterns and predicting future behavior based on that data.

[0345] The embodiments for carrying out the present invention are described below.

[0346] This system utilizes a combination of devices and technologies to manage household items and provide information based on the user's emotional state. Specifically, it employs the following hardware and software:

[0347] The server is the central hub for integrated management of information within the home. Located on a remote server, it functions as a data storage medium. When a voice command is received from a user via a terminal, the server processes it using analysis tools and retrieves the location information of necessary items from its database.

[0348] The server also analyzes the user's voice and facial expressions using emotion analysis tools. This is done through speech recognition and facial expression analysis software on the user's device. For example, if the server identifies that the user is stressed, it adjusts the information provided accordingly.

[0349] The terminal functions as an interface with the user. It is equipped with a voice recognition device that converts voice commands uttered by the user into text data. This information is sent to a server, and the information returned from the server is displayed visually. A visual map is displayed on the terminal, reporting the location of specific items to the user.

[0350] Users operate the system by giving voice commands to the device. For example, by asking "Where is the remote control?", the system can instantly obtain its location information. Furthermore, the system analyzes the user's emotional state and provides more appropriate information based on that analysis.

[0351] As a concrete example of this operation, if a user is in a hurry and gives a voice command such as "Where is it?", the server will quickly obtain the latest location information, and the device will highlight it and report it to the user.

[0352] An example of a prompt is, "Consider an approach to quickly and gently provide location information for everyday items when a user is experiencing emotional stress." Based on this prompt, the generative AI model generates an appropriate method of providing information.

[0353] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0354] Step 1:

[0355] The user gives voice commands about an item to the terminal. This voice is converted into text data by the terminal's voice recognition device. The input is the user's voice data, and the output is generated as text data. For example, if the user says "Find the remote control," the terminal converts this into text data.

[0356] Step 2:

[0357] The terminal sends the converted text data to the server. The server receives this text data as input and searches a database in the cloud. To obtain the location information of a specific item, the server analyzes the text data and extracts relevant information using search tools. The output is the location information of the specified item. Specifically, at this stage, the server queries the database based on the instruction "find the remote control" and locates the remote control.

[0358] Step 3:

[0359] The server transmits the acquired location information to the terminal. The terminal receives this location information as input and creates a visual map. The visual output clearly displays the location of the items. Specifically, the terminal displays a map with the remote control's location highlighted.

[0360] Step 4:

[0361] To provide information based on the user's emotional state, the device utilizes emotion analysis tools along with location information from the server. The server takes the user's emotion data as input and determines the appropriate reporting method. For example, if the device determines that the user is in a hurry, it will highlight important location information and provide guidance in a gentle voice. The output is a personalized report.

[0362] Step 5:

[0363] If necessary, the server generates suggestions for managing and rearranging items using suggestion mechanisms. These suggestions are based on the user's current state and past behavioral patterns. Input is the user's emotional state and behavioral pattern data, and output is specific suggestions. A specific action would be to display rearrangement suggestions when the user is feeling stressed.

[0364] (Application Example 2)

[0365] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0366] When managing the location of items within the home, current systems provide information without considering the user's emotional state, resulting in a failure to alleviate the user's psychological burden. In particular, there is a need for information that not only identifies the location of items but also takes the user's feelings into consideration. Furthermore, improving information provision based on the user's emotions is a crucial element in improving the quality of life within the home.

[0367] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0368] In this invention, the server includes data acquisition means for obtaining the location of an object, data storage means for storing the acquired location data in a cloud storage device, search means for searching for the location data of an object from the cloud storage device and reporting it to the user, voice recognition means for analyzing the user's voice instructions, monitoring means for monitoring the movement of an object and detecting anomalies, and emotion analysis means for analyzing the user's emotional state and adjusting the information provision method. This enables more efficient management of items within the home and comfortable information provision tailored to the user's emotions.

[0369] A "data acquisition means" is a device that detects the position of an object and collects the necessary data.

[0370] A "data storage means" is a device that securely stores and manages collected data in a cloud-based storage device.

[0371] A "search method" is a device that searches for specific location data from storage devices in the cloud and reports it to the user through visual or audio means.

[0372] A "voice recognition means" is a device that analyzes the user's voice and converts the instructions into text data.

[0373] A "monitoring device" is a device that continuously tracks the movement of an object and detects any anomalies.

[0374] An "emotion analysis device" is a device that determines emotions from the user's voice and facial expressions, and improves the method of providing information based on the results.

[0375] To implement this invention, a system is utilized that manages household items and provides information while taking into account the user's emotional state. Specifically, it is configured as follows:

[0376] The server is equipped with data acquisition means and uses cameras and sensors installed in the home to acquire object location data in real time. This information is securely stored in cloud storage via data storage means. The data in the cloud can be quickly retrieved using search means as needed and the information is provided to the terminal.

[0377] The terminal uses voice recognition to accurately analyze the user's voice instructions. The analyzed voice instructions are converted into text data and used for location retrieval. After this information is obtained, the terminal uses search tools to visually display the object's location and provides an audio report. Furthermore, it uses emotion analysis tools to analyze the user's voice and facial expressions, providing flexible information tailored to the user's emotional state.

[0378] When a user searches for their keys at home, if they say something like, "I'm looking for my keys," the system will display the key's location in real time. If the user is feeling stressed, it will also gently inform them, "They're on the table by the front door."

[0379] An example of a prompt message is, "Real-time location of household items and gentle voice guidance tailored to the user's feelings." Through this system, the aim is to significantly improve the convenience of managing household items and make users' lives richer and more comfortable.

[0380] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0381] Step 1:

[0382] The server acquires data from cameras and sensors placed within the home. Its inputs are camera video data and sensor information, which it analyzes and outputs as object location data. Specifically, it uses image processing technology to analyze the shape and color of objects and determines the location of specific items.

[0383] Step 2:

[0384] The server saves the acquired object location data to a cloud storage device using a data storage mechanism. The input here is the location data obtained in step 1, and the output is the updated location information in the cloud database. During saving, the data is organized and compressed to enable efficient searching.

[0385] Step 3:

[0386] The user gives voice commands to the device. The input is the user's voice commands, which are converted into text data by a speech recognition system. Specifically, the voice waveform is converted into digital data, its acoustic characteristics are analyzed, and the content of the command is converted into a string of characters.

[0387] Step 4:

[0388] The terminal sends text data to the server and uses a search mechanism to retrieve the location information of the target object from a cloud-based recording device. The input here is a text-based instruction, and the output is the object's location information retrieved from the cloud. The server automatically queries the database and performs calculations to extract relevant information.

[0389] Step 5:

[0390] The device evaluates the user's emotional state using emotion analysis tools. Input is additional voice or facial expression data from the user, and based on this, it outputs an emotional state (relaxed, stressed, etc.). It performs processing to calculate multiple emotional indicators using voice tone analysis and facial expression recognition technology.

[0391] Step 6:

[0392] The terminal provides appropriate voice guidance, taking into account the location information of the acquired objects and the user's emotional state. The input is the information obtained in steps 4 and 5, and the output is the voice and visual guidance provided to the user. Specifically, voice guidance is generated using speech synthesis technology, and a visual map is displayed on the screen as needed.

[0393] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0394] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0395] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart glasses 214.

[0396] [Third Embodiment]

[0397] Figure 5 shows an example of the configuration of the data processing system 310 according to the third embodiment.

[0398] As shown in Figure 5, the data processing system 310 includes a data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0399] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0400] The headset terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a display 343. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and display 343 are also connected to the bus 52.

[0401] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0402] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0403] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0404] Figure 6 shows an example of the main functions of the data processing device 12 and the headset terminal 314. As shown in Figure 6, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0405] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0406] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0407] In the headset terminal 314, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0408] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the headset terminal 314 will be referred to as the "terminal".

[0409] This invention provides a system and program for streamlining the management of items placed in a household. The embodiments thereof are described below.

[0410] This system determines the current location of objects through an information acquisition device that obtains the object's position in real time. The information acquisition device works in conjunction with sensors and cameras placed in each room to monitor the movement of objects. The acquired location information is stored in a cloud-based database via a data storage device and can be accessed by the user when needed.

[0411] The terminal receives voice commands from the user and converts them into text via a voice recognition device. If the user inquires about the location of a specific item, the terminal sends the command to the server. The server searches the database and retrieves the location information of the item. It then returns the location information to the terminal, which reports the item's location to the user via a visual map.

[0412] Furthermore, the server has the ability to analyze the item placement information in the database and suggest the optimal storage method. This allows users to organize and store their belongings efficiently. For example, the server may suggest rearranging kitchen items that tend to get cluttered.

[0413] Furthermore, the system includes a security feature that constantly monitors the movement of items using surveillance devices. In particular, if valuables or important items are moved inappropriately, the server immediately detects the anomaly and sends a warning to the terminal. Users can respond quickly upon receiving this warning.

[0414] Furthermore, the device learns the user's daily behavior patterns and generates reminders based on that information. It can notify the user at the appropriate time to prevent them from forgetting essential items. For example, it might prompt the user to check if they have their keys before leaving the house.

[0415] Thus, this invention significantly improves the efficiency of users' lives and provides a sense of security in the management and security of goods.

[0416] The following describes the processing flow.

[0417] Step 1:

[0418] The user gives a voice command to the device, such as "Where is XX?". The device then uses a speech recognition device to convert this voice into text data.

[0419] Step 2:

[0420] The terminal sends the converted text data to the server. There, the voice instructions are analyzed and the names of specific items are extracted.

[0421] Step 3:

[0422] The server searches for the location information of the received item in a cloud-based database based on the item's name. This search process is performed using the most up-to-date information in the database.

[0423] Step 4:

[0424] The server retrieves the location information of an item and sends it back to the terminal. This location information includes the specific location and coordinates of the item.

[0425] Step 5:

[0426] The device displays the received location information on a visual map. The map is shown on the user's screen, and highlights and icons are used to indicate the precise location of items.

[0427] Step 6:

[0428] The user checks the display on the device, recognizes the location of the selected item, and takes action to retrieve it.

[0429] This processing flow allows users to quickly and efficiently locate items and find what they need without stress.

[0430] (Example 1)

[0431] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0432] In modern living environments, numerous items exist within the home, making it difficult to accurately and efficiently manage their locations. In particular, items frequently go missing, and storage space is often used inefficiently. Furthermore, there are security concerns regarding misuse and theft due to the improper movement of items. Additionally, users often fail to keep track of and forget the items they need in their daily lives.

[0433] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0434] In this invention, the server includes detection means for acquiring the location of an item, recording means for storing the acquired location information on a storage medium on the network, and query means for retrieving the location information of an item from the storage medium on the network and reporting it to the user. This enables accurate tracking of the current location of an item, allowing for suggestions for efficient storage methods, security measures, and reminder functions based on the user's behavior.

[0435] A "detection means" is a device or mechanism that senses the location of an object in real time and acquires that information.

[0436] "Recording means" refers to a device or mechanism for storing the location information of acquired items in a storage device on a network.

[0437] "Inquiry means" refers to a device or mechanism that retrieves location information of an item from a storage device on a network and provides that information to the user.

[0438] "Voice processing means" refers to a device or mechanism that analyzes the user's voice instructions, converts them into text data, and transmits them to another device.

[0439] "Monitoring means" refers to a device or mechanism that constantly monitors the movement of an object and detects inappropriate movement or abnormalities.

[0440] A "learning tool" is a device or system that analyzes the user's behavior patterns, predicts how an item will be used, and generates reminders.

[0441] A "proposal means" is a device or mechanism that calculates the optimal arrangement and storage method for items and proposes it to the user.

[0442] This invention is a system for streamlining the management and security of household items. Specific embodiments of this system are described below.

[0443] The server functions as a "detection mechanism" to acquire the location of items in real time by working in conjunction with sensors and cameras placed in each room. The hardware used includes common indoor surveillance cameras and location sensors. The information acquired by the sensors and cameras is stored in a cloud-based database and functions as a "recording mechanism".

[0444] The terminal serves as a "voice processing device," receiving voice commands from the user and converting those commands into text using voice recognition software. For example, a voice recognition API is used to analyze voice commands such as "Where are the keys?"

[0445] The server receives these textual instructions and acts as a "query tool" to search a database in the cloud. By obtaining the location information of the item and sending it to the terminal, the user can find out the item's current location.

[0446] Furthermore, the server is equipped with a "suggestion mechanism" that proposes the optimal arrangement and storage method for items based on information in the database. For example, it can suggest the arrangement of kitchen utensils while considering storage efficiency.

[0447] In terms of security, the server acts as a "monitoring tool," monitoring the inappropriate movement of items and immediately sending a warning to the terminal if an anomaly is detected. This function allows users to quickly address problems.

[0448] Furthermore, the device learns the user's behavioral patterns and serves as a "learning tool." This allows it to provide reminders about items that are often forgotten in daily life, supporting the user's lifestyle. For example, it has a function that notifies the user, "Did you bring your keys?" when they leave the house.

[0449] Examples of prompt statements include the following:

[0450] "Please describe your household item management system. Please provide detailed information on how it handles situations where a user asks for the location of an item using voice commands."

[0451] This invention significantly improves the management and security of household items, providing users with peace of mind and convenience.

[0452] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0453] Step 1:

[0454] The user gives voice commands to the device. For example, a command such as "Tell me where the remote control is" might be input. The device uses voice recognition software to convert the input voice signal into text data. In this conversion process, the voice data is converted into a series of digital signals, which are then output as text.

[0455] Step 2:

[0456] The terminal sends the converted text data to the server. This allows the server to receive basic information to understand the user's request. The server analyzes the received text data and identifies the names of the items derived from it. Natural language processing techniques may be used for this analysis.

[0457] Step 3:

[0458] The server searches a database in the cloud to find the location information of the identified item. In this process, the server executes database queries and retrieves the latest location information as output. The output from the database is digital map information about the item's location.

[0459] Step 4:

[0460] The server transmits the acquired location information to the device. In doing so, the server converts the location information into an easily understandable format before sending it. For example, it might use JSON format data to transmit location information to the device.

[0461] Step 5:

[0462] The device uses the received location information to provide the user with visual information. The device uses a graphical user interface to display the current location of an item on a map. The device also outputs a notification, directly displaying the location information to the user.

[0463] Step 6:

[0464] The server analyzes the location data of items stored in the database and proposes the optimal storage method for those items. In this process, the server analyzes the data using statistics and machine learning algorithms. The output generates suggestions for optimizing item placement.

[0465] Step 7:

[0466] The server constantly monitors the movement of items and immediately sends a warning to the terminal if any inappropriate movement is detected. The server's input is real-time data from sensors, and its output is a warning message. This allows the user to respond to problems quickly.

[0467] Step 8:

[0468] The device learns the user's past behavior and generates reminders. In this process, the device takes the user's daily behavior data as input and creates notifications about items that are easily forgotten based on that data. The device provides these reminders to the user in a timely manner.

[0469] (Application Example 1)

[0470] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0471] There is a need to improve the efficiency of managing household items, reduce the risk of losing particularly important items, and promote tidiness and organization in living spaces. Furthermore, providing features such as item reminders based on users' daily behavior and anomaly detection functions for security purposes are also challenges.

[0472] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0473] In this invention, the server includes location detection means for acquiring the location of an object, information storage means for storing the acquired location information in a remote data area, and behavioral analysis and notification means for analyzing the user's daily behavior and reminding them of necessary items. This enables tracking the location of items, optimal item organization, and behavioral reminder notifications.

[0474] "Position detection means" refers to devices or methods for determining the position of an object, and is a technology that acquires information using sensors or cameras.

[0475] "Information retention means" refers to technology that stores acquired data in remote data areas such as storage devices or the cloud, making it accessible when needed.

[0476] "Information retrieval means" refers to a device or procedure that retrieves location information stored in a remote data area at the request of a user and reports the results.

[0477] "Voice interpretation means" refers to technology that analyzes voice instructions from users, interprets them as sentences, and performs appropriate information processing.

[0478] "Dynamic monitoring means" refers to devices and technologies for continuously monitoring the movement of an object and detecting abnormal behavior.

[0479] "Behavioral analysis and notification means" refers to technology that learns the user's daily behavioral patterns and generates appropriate item notifications and reminders based on those patterns.

[0480] The system that realizes this application example streamlines the management of items within the home and improves security. Specifically, a robot placed in the user's home uses location detection means to identify the location of items and stores that information in a cloud-based information storage system. Sensors and cameras are incorporated into the robot to detect surrounding items.

[0481] The server manages information in the cloud, uses information retrieval tools to quickly locate the relevant items, and reports the location to the user via a terminal. The user can give instructions to the robot via voice interpretation tools to find specified items. This function utilizes the Google Speech-to-Text API to convert speech to text.

[0482] Furthermore, the server uses dynamic monitoring to detect suspicious movement of items within the home and immediately issues a warning to the user's terminal in case of an anomaly. In addition, behavioral analysis and notification methods allow the system to learn the user's daily behavior and notify them of necessary items as a reminder. Software such as TensorFlow and OpenCV are used to acquire and analyze the user's behavioral patterns.

[0483] For example, to prevent misplacing everyday keys, the robot will ask "Do you have your keys?" when you leave the house. Also, if you ask "Where is the salt?" while cooking, the robot will check its location and quickly report the result.

[0484] An example of a prompt message is as follows:

[0485] "Learn the user's daily behavioral patterns and generate appropriate reminders. Provide specific behavioral scenarios, such as 'Check the items the user needs before leaving the house,' and design reminders based on those behaviors."

[0486] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0487] Step 1:

[0488] The user gives voice commands to the robot to locate an item. The user's voice is used as the voice input, and the device uses the Google Speech-to-Text API to convert the voice commands into text data, which is then sent to the server.

[0489] Step 2:

[0490] The server uses an information retrieval mechanism to search a database in the cloud based on the received text data. The input information for the database is the location information of an item, and the search result retrieves the latest location data for a specific item.

[0491] Step 3:

[0492] The server transmits the acquired location data to the terminal. The terminal displays the location of the item on a visually easy-to-understand map for the user and reports it to the user. The input is the acquired location data, and the output is a visual information display.

[0493] Step 4:

[0494] The server constantly monitors acquired data using dynamic monitoring devices and performs a process to detect anomalies. Based on user-defined criteria, it analyzes location data and, if suspicious movement is detected, generates an anomaly warning and sends it to the terminal. The input is location data, and the output is a warning notification.

[0495] Step 5:

[0496] The user's daily behavior patterns are analyzed on the server side using behavioral analysis and notification methods. Sensor data acquired using TensorFlow, etc., is analyzed to create reminders about items the user needs. For example, when the user leaves the house, a confirmation reminder such as "Did you bring your keys?" is sent. The input is behavioral pattern data, and the output is a reminder notification.

[0497] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0498] This invention provides an information provision system that takes into account the management of items within the home and the emotional state of the user. The embodiments thereof are described below.

[0499] This system includes an information acquisition device for obtaining the location of objects in real time. This device works in conjunction with cameras and sensors within the home to monitor the location and movement of items. The acquired location information is stored in a cloud-based database via a data storage device and can be retrieved as needed.

[0500] When a user gives a voice command to the device, the voice recognition device converts the command into text data and sends it to the server. The server searches a database based on this text data to locate the specified item. Furthermore, the device displays a visual map based on this information and reports the item's location to the user.

[0501] A distinctive feature of this invention is the addition of a function that analyzes the user's emotional state using an emotion engine. The emotion engine recognizes emotions from the user's voice and facial expressions, and adjusts the content and method of reporting based on that state. For example, if the user is feeling stressed, the device will provide information in a gentler voice and with simpler instructions. Specifically, if the device detects that the user is in a hurry, it will immediately highlight important location information to support efficient action.

[0502] Furthermore, the emotion engine adjusts the operation of the suggestion device based on the user's emotions. For example, when the user is feeling down, it offers ideas to refresh their life, such as suggesting the rearrangement of items. In this way, it provides information in a form that takes the user's psychological stability into consideration, realizing a system that is useful for managing items and improving quality of life.

[0503] The following describes the processing flow.

[0504] Step 1:

[0505] The user asks the device by voice, "Where is XX?" The device receives this voice and converts it into text data using a speech recognition device.

[0506] Step 2:

[0507] The terminal sends the converted text data to the server. The server analyzes this data and extracts the names of specific items.

[0508] Step 3:

[0509] The server searches a database in the cloud to retrieve the current location information of the requested item. This information includes location details.

[0510] Step 4:

[0511] The server sends the acquired location information back to the device. The device receives this information and displays it on a map in an easy-to-understand format for the user.

[0512] Step 5:

[0513] The device uses its built-in emotion engine to analyze the user's voice and facial expressions and evaluate their emotional state. This emotional state influences how notifications are delivered in the next step.

[0514] Step 6:

[0515] When the device recognizes the user's emotional state, it adjusts the information it presents accordingly. For example, if the user is stressed, the device will deliver information in a calm voice and in a soothing manner.

[0516] Step 7:

[0517] If the emotion engine detects a stressed state, the server will choose a storage suggestion method that is less burdensome for the user and propose it through the terminal.

[0518] Step 8:

[0519] The user checks information from the device and, based on that, retrieves the desired item or organizes items according to storage suggestions. Through this process, the user can maintain an efficient and emotionally comfortable environment.

[0520] (Example 2)

[0521] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0522] Managing household items is often cumbersome, and users frequently lose track of their belongings. Furthermore, the quality of information provided to users needs to change depending on their emotional state. These challenges necessitate a system that efficiently manages items while maintaining the user's psychological well-being.

[0523] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0524] In this invention, the server includes detection means for acquiring the location of an object, storage means for storing the acquired location information on a storage medium on a remote server, and emotion analysis means for analyzing the user's emotional state and adjusting the content and method of reporting. This enables efficient management of household items and the provision of information tailored to the user's emotional state.

[0525] "Detection means" refers to a device or method arranged to acquire the position of an object, and which has the function of collecting accurate positional information of the object.

[0526] "Storage means" refers to a device or method for temporarily or permanently storing acquired information or data on a storage medium on a remote server.

[0527] "Search means" refers to a device or method for analyzing data stored on a storage medium, extracting necessary information, and providing it to the user.

[0528] "Analysis means" refers to a device or method for analyzing user instructions and data and converting them into necessary information.

[0529] "Emotional analysis means" refers to a device or method for analyzing a user's voice, facial expressions, etc., to understand their emotional state and adjust the content and method of reporting accordingly.

[0530] "Monitoring means" refers to a device or method for continuously observing the movement or state of an object and detecting any abnormalities.

[0531] A "proposal means" is a device or method that presents the user with the most appropriate action or option depending on the situation.

[0532] "Learning means" refers to a device or method for continuously observing a user's behavioral patterns and predicting future behavior based on that data.

[0533] The embodiments for carrying out the present invention are described below.

[0534] This system utilizes a combination of devices and technologies to manage household items and provide information based on the user's emotional state. Specifically, it employs the following hardware and software:

[0535] The server is the central hub for integrated management of information within the home. Located on a remote server, it functions as a data storage medium. When a voice command is received from a user via a terminal, the server processes it using analysis tools and retrieves the location information of necessary items from its database.

[0536] The server also analyzes the user's voice and facial expressions using emotion analysis tools. This is done through speech recognition and facial expression analysis software on the user's device. For example, if the server identifies that the user is stressed, it adjusts the information provided accordingly.

[0537] The terminal functions as an interface with the user. It is equipped with a voice recognition device that converts voice commands uttered by the user into text data. This information is sent to a server, and the information returned from the server is displayed visually. A visual map is displayed on the terminal, reporting the location of specific items to the user.

[0538] Users operate the system by giving voice commands to the device. For example, by asking "Where is the remote control?", the system can instantly obtain its location information. Furthermore, the system analyzes the user's emotional state and provides more appropriate information based on that analysis.

[0539] As a concrete example of this operation, if a user is in a hurry and gives a voice command such as "Where is it?", the server will quickly obtain the latest location information, and the device will highlight it and report it to the user.

[0540] An example of a prompt is, "Consider an approach to quickly and gently provide location information for everyday items when a user is experiencing emotional stress." Based on this prompt, the generative AI model generates an appropriate method of providing information.

[0541] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0542] Step 1:

[0543] The user gives voice commands about an item to the terminal. This voice is converted into text data by the terminal's voice recognition device. The input is the user's voice data, and the output is generated as text data. For example, if the user says "Find the remote control," the terminal converts this into text data.

[0544] Step 2:

[0545] The terminal sends the converted text data to the server. The server receives this text data as input and searches a database in the cloud. To obtain the location information of a specific item, the server analyzes the text data and extracts relevant information using search tools. The output is the location information of the specified item. Specifically, at this stage, the server queries the database based on the instruction "find the remote control" and locates the remote control.

[0546] Step 3:

[0547] The server transmits the acquired location information to the terminal. The terminal receives this location information as input and creates a visual map. The visual output clearly displays the location of the items. Specifically, the terminal displays a map with the remote control's location highlighted.

[0548] Step 4:

[0549] To provide information based on the user's emotional state, the device utilizes emotion analysis tools along with location information from the server. The server takes the user's emotion data as input and determines the appropriate reporting method. For example, if the device determines that the user is in a hurry, it will highlight important location information and provide guidance in a gentle voice. The output is a personalized report.

[0550] Step 5:

[0551] If necessary, the server generates suggestions for managing and rearranging items using suggestion mechanisms. These suggestions are based on the user's current state and past behavioral patterns. Input is the user's emotional state and behavioral pattern data, and output is specific suggestions. A specific action would be to display rearrangement suggestions when the user is feeling stressed.

[0552] (Application Example 2)

[0553] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0554] When managing the location of items within the home, current systems provide information without considering the user's emotional state, resulting in a failure to alleviate the user's psychological burden. In particular, there is a need for information that not only identifies the location of items but also takes the user's feelings into consideration. Furthermore, improving information provision based on the user's emotions is a crucial element in improving the quality of life within the home.

[0555] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0556] In this invention, the server includes data acquisition means for obtaining the location of an object, data storage means for storing the acquired location data in a cloud storage device, search means for searching for the location data of an object from the cloud storage device and reporting it to the user, voice recognition means for analyzing the user's voice instructions, monitoring means for monitoring the movement of an object and detecting anomalies, and emotion analysis means for analyzing the user's emotional state and adjusting the information provision method. This enables more efficient management of items within the home and comfortable information provision tailored to the user's emotions.

[0557] A "data acquisition means" is a device that detects the position of an object and collects the necessary data.

[0558] A "data storage means" is a device that securely stores and manages collected data in a cloud-based storage device.

[0559] A "search method" is a device that searches for specific location data from storage devices in the cloud and reports it to the user through visual or audio means.

[0560] A "voice recognition means" is a device that analyzes the user's voice and converts the instructions into text data.

[0561] A "monitoring device" is a device that continuously tracks the movement of an object and detects any anomalies.

[0562] An "emotion analysis device" is a device that determines emotions from the user's voice and facial expressions, and improves the method of providing information based on the results.

[0563] To implement this invention, a system is utilized that manages household items and provides information while taking into account the user's emotional state. Specifically, it is configured as follows:

[0564] The server is equipped with data acquisition means and uses cameras and sensors installed in the home to acquire object location data in real time. This information is securely stored in cloud storage via data storage means. The data in the cloud can be quickly retrieved using search means as needed and the information is provided to the terminal.

[0565] The terminal uses voice recognition to accurately analyze the user's voice instructions. The analyzed voice instructions are converted into text data and used for location retrieval. After this information is obtained, the terminal uses search tools to visually display the object's location and provides an audio report. Furthermore, it uses emotion analysis tools to analyze the user's voice and facial expressions, providing flexible information tailored to the user's emotional state.

[0566] When a user searches for their keys at home, if they say something like, "I'm looking for my keys," the system will display the key's location in real time. If the user is feeling stressed, it will also gently inform them, "They're on the table by the front door."

[0567] An example of a prompt message is, "Real-time location of household items and gentle voice guidance tailored to the user's feelings." Through this system, the aim is to significantly improve the convenience of managing household items and make users' lives richer and more comfortable.

[0568] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0569] Step 1:

[0570] The server acquires data from cameras and sensors placed within the home. Its inputs are camera video data and sensor information, which it analyzes and outputs as object location data. Specifically, it uses image processing technology to analyze the shape and color of objects and determines the location of specific items.

[0571] Step 2:

[0572] The server saves the acquired object location data to a cloud storage device using a data storage mechanism. The input here is the location data obtained in step 1, and the output is the updated location information in the cloud database. During saving, the data is organized and compressed to enable efficient searching.

[0573] Step 3:

[0574] The user gives voice commands to the device. The input is the user's voice commands, which are converted into text data by a speech recognition system. Specifically, the voice waveform is converted into digital data, its acoustic characteristics are analyzed, and the content of the command is converted into a string of characters.

[0575] Step 4:

[0576] The terminal sends text data to the server and uses a search mechanism to retrieve the location information of the target object from a cloud-based recording device. The input here is a text-based instruction, and the output is the object's location information retrieved from the cloud. The server automatically queries the database and performs calculations to extract relevant information.

[0577] Step 5:

[0578] The device evaluates the user's emotional state using emotion analysis tools. Input is additional voice or facial expression data from the user, and based on this, it outputs an emotional state (relaxed, stressed, etc.). It performs processing to calculate multiple emotional indicators using voice tone analysis and facial expression recognition technology.

[0579] Step 6:

[0580] The terminal provides appropriate voice guidance, taking into account the location information of the acquired objects and the user's emotional state. The input is the information obtained in steps 4 and 5, and the output is the voice and visual guidance provided to the user. Specifically, voice guidance is generated using speech synthesis technology, and a visual map is displayed on the screen as needed.

[0581] The specific processing unit 290 transmits the result of the specific processing to the headset terminal 314. In the headset terminal 314, the control unit 46A causes the speaker 240 and display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0582] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0583] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and specific processing may also be performed by the headset terminal 314.

[0584] [Fourth Embodiment]

[0585] Figure 7 shows an example of the configuration of the data processing system 410 according to the fourth embodiment.

[0586] As shown in Figure 7, the data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[0587] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0588] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a controlled object 443. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and controlled object 443 are also connected to the bus 52.

[0589] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0590] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0591] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0592] The controlled object 443 includes a display device, LEDs in the eyes, and motors that drive the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the robot 414's emotions can be expressed by controlling these motors. Furthermore, the robot 414's facial expressions can also be expressed by controlling the illumination state of the LEDs in its eyes.

[0593] Figure 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Figure 8, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0594] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0595] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0596] In robot 414, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0597] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0598] This invention provides a system and program for streamlining the management of items placed in a household. The embodiments thereof are described below.

[0599] This system determines the current location of objects through an information acquisition device that obtains the object's position in real time. The information acquisition device works in conjunction with sensors and cameras placed in each room to monitor the movement of objects. The acquired location information is stored in a cloud-based database via a data storage device and can be accessed by the user when needed.

[0600] The terminal receives voice commands from the user and converts them into text via a voice recognition device. If the user inquires about the location of a specific item, the terminal sends the command to the server. The server searches the database and retrieves the location information of the item. It then returns the location information to the terminal, which reports the item's location to the user via a visual map.

[0601] Furthermore, the server has the ability to analyze the item placement information in the database and suggest the optimal storage method. This allows users to organize and store their belongings efficiently. For example, the server may suggest rearranging kitchen items that tend to get cluttered.

[0602] Furthermore, the system includes a security feature that constantly monitors the movement of items using surveillance devices. In particular, if valuables or important items are moved inappropriately, the server immediately detects the anomaly and sends a warning to the terminal. Users can respond quickly upon receiving this warning.

[0603] Furthermore, the device learns the user's daily behavior patterns and generates reminders based on that information. It can notify the user at the appropriate time to prevent them from forgetting essential items. For example, it might prompt the user to check if they have their keys before leaving the house.

[0604] Thus, this invention significantly improves the efficiency of users' lives and provides a sense of security in the management and security of goods.

[0605] The following describes the processing flow.

[0606] Step 1:

[0607] The user gives a voice command to the device, such as "Where is XX?". The device then uses a speech recognition device to convert this voice into text data.

[0608] Step 2:

[0609] The terminal sends the converted text data to the server. There, the voice instructions are analyzed and the names of specific items are extracted.

[0610] Step 3:

[0611] The server searches for the location information of the received item in a cloud-based database based on the item's name. This search process is performed using the most up-to-date information in the database.

[0612] Step 4:

[0613] The server retrieves the location information of an item and sends it back to the terminal. This location information includes the specific location and coordinates of the item.

[0614] Step 5:

[0615] The device displays the received location information on a visual map. The map is shown on the user's screen, and highlights and icons are used to indicate the precise location of items.

[0616] Step 6:

[0617] The user checks the display on the device, recognizes the location of the selected item, and takes action to retrieve it.

[0618] This processing flow allows users to quickly and efficiently locate items and find what they need without stress.

[0619] (Example 1)

[0620] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0621] In modern living environments, numerous items exist within the home, making it difficult to accurately and efficiently manage their locations. In particular, items frequently go missing, and storage space is often used inefficiently. Furthermore, there are security concerns regarding misuse and theft due to the improper movement of items. Additionally, users often fail to keep track of and forget the items they need in their daily lives.

[0622] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0623] In this invention, the server includes detection means for acquiring the location of an item, recording means for storing the acquired location information on a storage medium on the network, and query means for retrieving the location information of an item from the storage medium on the network and reporting it to the user. This enables accurate tracking of the current location of an item, allowing for suggestions for efficient storage methods, security measures, and reminder functions based on the user's behavior.

[0624] A "detection means" is a device or mechanism that senses the location of an object in real time and acquires that information.

[0625] "Recording means" refers to a device or mechanism for storing the location information of acquired items in a storage device on a network.

[0626] "Inquiry means" refers to a device or mechanism that retrieves location information of an item from a storage device on a network and provides that information to the user.

[0627] "Voice processing means" refers to a device or mechanism that analyzes the user's voice instructions, converts them into text data, and transmits them to another device.

[0628] "Monitoring means" refers to a device or mechanism that constantly monitors the movement of an object and detects inappropriate movement or abnormalities.

[0629] A "learning tool" is a device or system that analyzes the user's behavior patterns, predicts how an item will be used, and generates reminders.

[0630] A "proposal means" is a device or mechanism that calculates the optimal arrangement and storage method for items and proposes it to the user.

[0631] This invention is a system for streamlining the management and security of household items. Specific embodiments of this system are described below.

[0632] The server functions as a "detection mechanism" to acquire the location of items in real time by working in conjunction with sensors and cameras placed in each room. The hardware used includes common indoor surveillance cameras and location sensors. The information acquired by the sensors and cameras is stored in a cloud-based database and functions as a "recording mechanism".

[0633] The terminal serves as a "voice processing device," receiving voice commands from the user and converting those commands into text using voice recognition software. For example, a voice recognition API is used to analyze voice commands such as "Where are the keys?"

[0634] The server receives these textual instructions and acts as a "query tool" to search a database in the cloud. By obtaining the location information of the item and sending it to the terminal, the user can find out the item's current location.

[0635] Furthermore, the server is equipped with a "suggestion mechanism" that proposes the optimal arrangement and storage method for items based on information in the database. For example, it can suggest the arrangement of kitchen utensils while considering storage efficiency.

[0636] In terms of security, the server acts as a "monitoring tool," monitoring the inappropriate movement of items and immediately sending a warning to the terminal if an anomaly is detected. This function allows users to quickly address problems.

[0637] Furthermore, the device learns the user's behavioral patterns and serves as a "learning tool." This allows it to provide reminders about items that are often forgotten in daily life, supporting the user's lifestyle. For example, it has a function that notifies the user, "Did you bring your keys?" when they leave the house.

[0638] Examples of prompt statements include the following:

[0639] "Please describe your household item management system. Please provide detailed information on how it handles situations where a user asks for the location of an item using voice commands."

[0640] This invention significantly improves the management and security of household items, providing users with peace of mind and convenience.

[0641] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0642] Step 1:

[0643] The user gives voice commands to the device. For example, a command such as "Tell me where the remote control is" might be input. The device uses voice recognition software to convert the input voice signal into text data. In this conversion process, the voice data is converted into a series of digital signals, which are then output as text.

[0644] Step 2:

[0645] The terminal sends the converted text data to the server. This allows the server to receive basic information to understand the user's request. The server analyzes the received text data and identifies the names of the items derived from it. Natural language processing techniques may be used for this analysis.

[0646] Step 3:

[0647] The server searches a database in the cloud to find the location information of the identified item. In this process, the server executes database queries and retrieves the latest location information as output. The output from the database is digital map information about the item's location.

[0648] Step 4:

[0649] The server transmits the acquired location information to the device. In doing so, the server converts the location information into an easily understandable format before sending it. For example, it might use JSON format data to transmit location information to the device.

[0650] Step 5:

[0651] The device uses the received location information to provide the user with visual information. The device uses a graphical user interface to display the current location of an item on a map. The device also outputs a notification, directly displaying the location information to the user.

[0652] Step 6:

[0653] The server analyzes the location data of items stored in the database and proposes the optimal storage method for those items. In this process, the server analyzes the data using statistics and machine learning algorithms. The output generates suggestions for optimizing item placement.

[0654] Step 7:

[0655] The server constantly monitors the movement of items and immediately sends a warning to the terminal if any inappropriate movement is detected. The server's input is real-time data from sensors, and its output is a warning message. This allows the user to respond to problems quickly.

[0656] Step 8:

[0657] The device learns the user's past behavior and generates reminders. In this process, the device takes the user's daily behavior data as input and creates notifications about items that are easily forgotten based on that data. The device provides these reminders to the user in a timely manner.

[0658] (Application Example 1)

[0659] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0660] There is a need to improve the efficiency of managing household items, reduce the risk of losing particularly important items, and promote tidiness and organization in living spaces. Furthermore, providing features such as item reminders based on users' daily behavior and anomaly detection functions for security purposes are also challenges.

[0661] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0662] In this invention, the server includes location detection means for acquiring the location of an object, information storage means for storing the acquired location information in a remote data area, and behavioral analysis and notification means for analyzing the user's daily behavior and reminding them of necessary items. This enables tracking the location of items, optimal item organization, and behavioral reminder notifications.

[0663] "Position detection means" refers to devices or methods for determining the position of an object, and is a technology that acquires information using sensors or cameras.

[0664] "Information retention means" refers to technology that stores acquired data in remote data areas such as storage devices or the cloud, making it accessible when needed.

[0665] "Information retrieval means" refers to a device or procedure that retrieves location information stored in a remote data area at the request of a user and reports the results.

[0666] "Voice interpretation means" refers to technology that analyzes voice instructions from users, interprets them as sentences, and performs appropriate information processing.

[0667] "Dynamic monitoring means" refers to devices and technologies for continuously monitoring the movement of an object and detecting abnormal behavior.

[0668] "Behavioral analysis and notification means" refers to technology that learns the user's daily behavioral patterns and generates appropriate item notifications and reminders based on those patterns.

[0669] The system that realizes this application example streamlines the management of items within the home and improves security. Specifically, a robot placed in the user's home uses location detection means to identify the location of items and stores that information in a cloud-based information storage system. Sensors and cameras are incorporated into the robot to detect surrounding items.

[0670] The server manages information in the cloud, uses information retrieval tools to quickly locate the relevant items, and reports the location to the user via a terminal. The user can give instructions to the robot via voice interpretation tools to find specified items. This function utilizes the Google Speech-to-Text API to convert speech to text.

[0671] Furthermore, the server uses dynamic monitoring to detect suspicious movement of items within the home and immediately issues a warning to the user's terminal in case of an anomaly. In addition, behavioral analysis and notification methods allow the system to learn the user's daily behavior and notify them of necessary items as a reminder. Software such as TensorFlow and OpenCV are used to acquire and analyze the user's behavioral patterns.

[0672] For example, to prevent misplacing everyday keys, the robot will ask "Do you have your keys?" when you leave the house. Also, if you ask "Where is the salt?" while cooking, the robot will check its location and quickly report the result.

[0673] An example of a prompt message is as follows:

[0674] "Learn the user's daily behavioral patterns and generate appropriate reminders. Provide specific behavioral scenarios, such as 'Check the items the user needs before leaving the house,' and design reminders based on those behaviors."

[0675] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0676] Step 1:

[0677] The user gives voice commands to the robot to locate an item. The user's voice is used as the voice input, and the device uses the Google Speech-to-Text API to convert the voice commands into text data, which is then sent to the server.

[0678] Step 2:

[0679] The server uses an information retrieval mechanism to search a database in the cloud based on the received text data. The input information for the database is the location information of an item, and the search result retrieves the latest location data for a specific item.

[0680] Step 3:

[0681] The server transmits the acquired location data to the terminal. The terminal displays the location of the item on a visually easy-to-understand map for the user and reports it to the user. The input is the acquired location data, and the output is a visual information display.

[0682] Step 4:

[0683] The server constantly monitors acquired data using dynamic monitoring devices and performs a process to detect anomalies. Based on user-defined criteria, it analyzes location data and, if suspicious movement is detected, generates an anomaly warning and sends it to the terminal. The input is location data, and the output is a warning notification.

[0684] Step 5:

[0685] The user's daily behavior patterns are analyzed on the server side using behavioral analysis and notification methods. Sensor data acquired using TensorFlow, etc., is analyzed to create reminders about items the user needs. For example, when the user leaves the house, a confirmation reminder such as "Did you bring your keys?" is sent. The input is behavioral pattern data, and the output is a reminder notification.

[0686] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0687] This invention provides an information provision system that takes into account the management of items within the home and the emotional state of the user. The embodiments thereof are described below.

[0688] This system includes an information acquisition device for obtaining the location of objects in real time. This device works in conjunction with cameras and sensors within the home to monitor the location and movement of items. The acquired location information is stored in a cloud-based database via a data storage device and can be retrieved as needed.

[0689] When a user gives a voice command to the device, the voice recognition device converts the command into text data and sends it to the server. The server searches a database based on this text data to locate the specified item. Furthermore, the device displays a visual map based on this information and reports the item's location to the user.

[0690] A distinctive feature of this invention is the addition of a function that analyzes the user's emotional state using an emotion engine. The emotion engine recognizes emotions from the user's voice and facial expressions, and adjusts the content and method of reporting based on that state. For example, if the user is feeling stressed, the device will provide information in a gentler voice and with simpler instructions. Specifically, if the device detects that the user is in a hurry, it will immediately highlight important location information to support efficient action.

[0691] Furthermore, the emotion engine adjusts the operation of the suggestion device based on the user's emotions. For example, when the user is feeling down, it offers ideas to refresh their life, such as suggesting the rearrangement of items. In this way, it provides information in a form that takes the user's psychological stability into consideration, realizing a system that is useful for managing items and improving quality of life.

[0692] The following describes the processing flow.

[0693] Step 1:

[0694] The user asks the device by voice, "Where is XX?" The device receives this voice and converts it into text data using a speech recognition device.

[0695] Step 2:

[0696] The terminal sends the converted text data to the server. The server analyzes this data and extracts the names of specific items.

[0697] Step 3:

[0698] The server searches a database in the cloud to retrieve the current location information of the requested item. This information includes location details.

[0699] Step 4:

[0700] The server sends the acquired location information back to the device. The device receives this information and displays it on a map in an easy-to-understand format for the user.

[0701] Step 5:

[0702] The device uses its built-in emotion engine to analyze the user's voice and facial expressions and evaluate their emotional state. This emotional state influences how notifications are delivered in the next step.

[0703] Step 6:

[0704] When the device recognizes the user's emotional state, it adjusts the information it presents accordingly. For example, if the user is stressed, the device will deliver information in a calm voice and in a soothing manner.

[0705] Step 7:

[0706] If the emotion engine detects a stressed state, the server will choose a storage suggestion method that is less burdensome for the user and propose it through the terminal.

[0707] Step 8:

[0708] The user checks information from the device and, based on that, retrieves the desired item or organizes items according to storage suggestions. Through this process, the user can maintain an efficient and emotionally comfortable environment.

[0709] (Example 2)

[0710] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0711] Managing household items is often cumbersome, and users frequently lose track of their belongings. Furthermore, the quality of information provided to users needs to change depending on their emotional state. These challenges necessitate a system that efficiently manages items while maintaining the user's psychological well-being.

[0712] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0713] In this invention, the server includes detection means for acquiring the location of an object, storage means for storing the acquired location information on a storage medium on a remote server, and emotion analysis means for analyzing the user's emotional state and adjusting the content and method of reporting. This enables efficient management of household items and the provision of information tailored to the user's emotional state.

[0714] "Detection means" refers to a device or method arranged to acquire the position of an object, and which has the function of collecting accurate positional information of the object.

[0715] "Storage means" refers to a device or method for temporarily or permanently storing acquired information or data on a storage medium on a remote server.

[0716] "Search means" refers to a device or method for analyzing data stored on a storage medium, extracting necessary information, and providing it to the user.

[0717] "Analysis means" refers to a device or method for analyzing user instructions and data and converting them into necessary information.

[0718] "Emotional analysis means" refers to a device or method for analyzing a user's voice, facial expressions, etc., to understand their emotional state and adjust the content and method of reporting accordingly.

[0719] "Monitoring means" refers to a device or method for continuously observing the movement or state of an object and detecting any abnormalities.

[0720] A "proposal means" is a device or method that presents the user with the most appropriate action or option depending on the situation.

[0721] "Learning means" refers to a device or method for continuously observing a user's behavioral patterns and predicting future behavior based on that data.

[0722] The embodiments for carrying out the present invention are described below.

[0723] This system utilizes a combination of devices and technologies to manage household items and provide information based on the user's emotional state. Specifically, it employs the following hardware and software:

[0724] The server is the central hub for integrated management of information within the home. Located on a remote server, it functions as a data storage medium. When a voice command is received from a user via a terminal, the server processes it using analysis tools and retrieves the location information of necessary items from its database.

[0725] The server also analyzes the user's voice and facial expressions using emotion analysis tools. This is done through speech recognition and facial expression analysis software on the user's device. For example, if the server identifies that the user is stressed, it adjusts the information provided accordingly.

[0726] The terminal functions as an interface with the user. It is equipped with a voice recognition device that converts voice commands uttered by the user into text data. This information is sent to a server, and the information returned from the server is displayed visually. A visual map is displayed on the terminal, reporting the location of specific items to the user.

[0727] Users operate the system by giving voice commands to the device. For example, by asking "Where is the remote control?", the system can instantly obtain its location information. Furthermore, the system analyzes the user's emotional state and provides more appropriate information based on that analysis.

[0728] As a concrete example of this operation, if a user is in a hurry and gives a voice command such as "Where is it?", the server will quickly obtain the latest location information, and the device will highlight it and report it to the user.

[0729] An example of a prompt is, "Consider an approach to quickly and gently provide location information for everyday items when a user is experiencing emotional stress." Based on this prompt, the generative AI model generates an appropriate method of providing information.

[0730] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0731] Step 1:

[0732] The user gives voice commands about an item to the terminal. This voice is converted into text data by the terminal's voice recognition device. The input is the user's voice data, and the output is generated as text data. For example, if the user says "Find the remote control," the terminal converts this into text data.

[0733] Step 2:

[0734] The terminal sends the converted text data to the server. The server receives this text data as input and searches a database in the cloud. To obtain the location information of a specific item, the server analyzes the text data and extracts relevant information using search tools. The output is the location information of the specified item. Specifically, at this stage, the server queries the database based on the instruction "find the remote control" and locates the remote control.

[0735] Step 3:

[0736] The server transmits the acquired location information to the terminal. The terminal receives this location information as input and creates a visual map. The visual output clearly displays the location of the items. Specifically, the terminal displays a map with the remote control's location highlighted.

[0737] Step 4:

[0738] To provide information based on the user's emotional state, the device utilizes emotion analysis tools along with location information from the server. The server takes the user's emotion data as input and determines the appropriate reporting method. For example, if the device determines that the user is in a hurry, it will highlight important location information and provide guidance in a gentle voice. The output is a personalized report.

[0739] Step 5:

[0740] If necessary, the server generates suggestions for managing and rearranging items using suggestion mechanisms. These suggestions are based on the user's current state and past behavioral patterns. Input is the user's emotional state and behavioral pattern data, and output is specific suggestions. A specific action would be to display rearrangement suggestions when the user is feeling stressed.

[0741] (Application Example 2)

[0742] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0743] When managing the location of items within the home, current systems provide information without considering the user's emotional state, resulting in a failure to alleviate the user's psychological burden. In particular, there is a need for information that not only identifies the location of items but also takes the user's feelings into consideration. Furthermore, improving information provision based on the user's emotions is a crucial element in improving the quality of life within the home.

[0744] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0745] In this invention, the server includes data acquisition means for obtaining the location of an object, data storage means for storing the acquired location data in a cloud storage device, search means for searching for the location data of an object from the cloud storage device and reporting it to the user, voice recognition means for analyzing the user's voice instructions, monitoring means for monitoring the movement of an object and detecting anomalies, and emotion analysis means for analyzing the user's emotional state and adjusting the information provision method. This enables more efficient management of items within the home and comfortable information provision tailored to the user's emotions.

[0746] A "data acquisition means" is a device that detects the position of an object and collects the necessary data.

[0747] A "data storage means" is a device that securely stores and manages collected data in a cloud-based storage device.

[0748] A "search method" is a device that searches for specific location data from storage devices in the cloud and reports it to the user through visual or audio means.

[0749] A "voice recognition means" is a device that analyzes the user's voice and converts the instructions into text data.

[0750] A "monitoring device" is a device that continuously tracks the movement of an object and detects any anomalies.

[0751] An "emotion analysis device" is a device that determines emotions from the user's voice and facial expressions, and improves the method of providing information based on the results.

[0752] To implement this invention, a system is utilized that manages household items and provides information while taking into account the user's emotional state. Specifically, it is configured as follows:

[0753] The server is equipped with data acquisition means and uses cameras and sensors installed in the home to acquire object location data in real time. This information is securely stored in cloud storage via data storage means. The data in the cloud can be quickly retrieved using search means as needed and the information is provided to the terminal.

[0754] The terminal uses voice recognition to accurately analyze the user's voice instructions. The analyzed voice instructions are converted into text data and used for location retrieval. After this information is obtained, the terminal uses search tools to visually display the object's location and provides an audio report. Furthermore, it uses emotion analysis tools to analyze the user's voice and facial expressions, providing flexible information tailored to the user's emotional state.

[0755] When a user searches for their keys at home, if they say something like, "I'm looking for my keys," the system will display the key's location in real time. If the user is feeling stressed, it will also gently inform them, "They're on the table by the front door."

[0756] An example of a prompt message is, "Real-time location of household items and gentle voice guidance tailored to the user's feelings." Through this system, the aim is to significantly improve the convenience of managing household items and make users' lives richer and more comfortable.

[0757] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0758] Step 1:

[0759] The server acquires data from cameras and sensors placed within the home. Its inputs are camera video data and sensor information, which it analyzes and outputs as object location data. Specifically, it uses image processing technology to analyze the shape and color of objects and determines the location of specific items.

[0760] Step 2:

[0761] The server saves the acquired object location data to a cloud storage device using a data storage mechanism. The input here is the location data obtained in step 1, and the output is the updated location information in the cloud database. During saving, the data is organized and compressed to enable efficient searching.

[0762] Step 3:

[0763] The user gives voice commands to the device. The input is the user's voice commands, which are converted into text data by a speech recognition system. Specifically, the voice waveform is converted into digital data, its acoustic characteristics are analyzed, and the content of the command is converted into a string of characters.

[0764] Step 4:

[0765] The terminal sends text data to the server and uses a search mechanism to retrieve the location information of the target object from a cloud-based recording device. The input here is a text-based instruction, and the output is the object's location information retrieved from the cloud. The server automatically queries the database and performs calculations to extract relevant information.

[0766] Step 5:

[0767] The device evaluates the user's emotional state using emotion analysis tools. Input is additional voice or facial expression data from the user, and based on this, it outputs an emotional state (relaxed, stressed, etc.). It performs processing to calculate multiple emotional indicators using voice tone analysis and facial expression recognition technology.

[0768] Step 6:

[0769] The terminal provides appropriate voice guidance, taking into account the location information of the acquired objects and the user's emotional state. The input is the information obtained in steps 4 and 5, and the output is the voice and visual guidance provided to the user. Specifically, voice guidance is generated using speech synthesis technology, and a visual map is displayed on the screen as needed.

[0770] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the controlled object 443 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0771] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0772] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the robot 414.

[0773] Furthermore, the emotion identification model 59, acting as an emotion engine, may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to a specific mapping, which is an emotion map (see Figure 9). Similarly, the emotion identification model 59 may also determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[0774] Figure 9 shows an emotion map 400 in which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. The closer to the center of the concentric circles, the more primitive the emotions are located. Further out of the concentric circles, emotions representing states and actions arising from mental states are located. Emotion is a concept that includes feelings and mental states. On the left side of the concentric circles, emotions that are generally generated from reactions occurring in the brain are located. On the right side of the concentric circles, emotions that are generally induced by situational judgment are located. Above and below the concentric circles, emotions that are generally generated from reactions occurring in the brain and induced by situational judgment are located. In addition, the emotion of "pleasure" is located on the upper side of the concentric circles, and the emotion of "displeasure" is located on the lower side. Thus, in the emotion map 400, multiple emotions are mapped based on the structure in which emotions arise, and emotions that are likely to occur simultaneously are mapped close together.

[0775] These emotions are distributed at the 3 o'clock position on the Emotion Map 400, and usually fluctuate between feelings of security and anxiety. In the right half of the Emotion Map 400, situational awareness takes precedence over internal feelings, resulting in a calm impression.

[0776] The inside of the Emotion Map 400 represents inner thoughts, while the outside represents actions. Therefore, the further you go from the outside of the Emotion Map 400, the more visible (expressed in actions) your emotions become.

[0777] Here, human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. Similarly, in robots, cars, motorcycles, etc., emotions can be created based on various balances, such as posture and battery level. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. The emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on a system for analyzing brain physiological signals of speech emotion recognition and emotion, Tokushima University, doctoral dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map contains emotions belonging to a region called "response," where sensation is dominant. The right half of the emotion map contains emotions belonging to a region called "situation," where situational awareness is dominant.

[0778] The emotion map defines two emotions that promote learning. One is the emotion around the middle of the negative "repentance" and "reflection" on the situation side. In other words, it is when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is the emotion around the positive "desire" on the reaction side. In other words, it is when the robot has positive feelings such as "I want more" or "I want to know more."

[0779] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​representing each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple training data sets, which are combinations of user input and emotion values ​​representing each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions located close together have similar values, as shown in the emotion map 900 in Figure 10. Figure 10 shows an example where multiple emotions such as "reassured," "calm," and "confident" have similar emotion values.

[0780] The above description primarily focuses on the functions of the data processing device 12 in relation to this disclosure. However, the system related to this disclosure is not necessarily implemented on a server. The system related to this disclosure may be implemented as a general information processing system. This disclosure may be implemented, for example, as a software program that runs on a personal computer or as an application that runs on a smartphone. The method related to this disclosure may be provided to users in SaaS (Software as a Service) format.

[0781] In the above embodiment, an example was given in which a specific process is performed by a single computer 22. However, the technology of this disclosure is not limited thereto, and a distributed processing of the specific process may be performed by multiple computers, including computer 22. For example, a data generation model 58 may be provided in an external device of the data processing device 12, and the external device may generate data according to the input data.

[0782] In the above embodiment, an example was given in which the specific processing program 56 is stored in the storage 32, but the technology of this disclosure is not limited thereto. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-temporary storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-temporary storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes specific processing according to the specific processing program 56.

[0783] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[0784] Furthermore, it is not necessary to store the entirety of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store the entirety of the specific processing program 56 in the storage 32; it is acceptable to store only a portion of the specific processing program 56.

[0785] The following types of processors can be used as hardware resources to perform specific processing. Examples of processors include a CPU, a general-purpose processor that functions as a hardware resource to perform specific processing by executing software, i.e., a program. Other examples of processors include dedicated electrical circuits, such as FPGAs (Field-Programmable Gate Arrays), PLDs (Programmable Logic Devices), or ASICs (Application Specific Integrated Circuits), which have circuit configurations specifically designed to perform specific processing. All of these processors have built-in or connected memory, and all of them perform specific processing by using memory.

[0786] The hardware resource that performs a specific process may consist of one of these various processors, or it may consist of a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Alternatively, the hardware resource that performs a specific process may consist of a single processor.

[0787] Examples of configurations using a single processor include, firstly, a configuration in which one or more CPUs and software are combined to form a single processor, and this processor functions as a hardware resource that performs a specific process. Secondly, there is a configuration using a processor that realizes the functions of the entire system, including multiple hardware resources that perform a specific process, on a single IC chip, as exemplified by SoCs (System-on-a-chip). In this way, a specific process is realized using one or more of the above types of processors as hardware resources.

[0788] Furthermore, the hardware structure of these various processors can more specifically utilize electrical circuits that combine circuit elements such as semiconductor devices. Also, the specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps can be deleted, new steps added, or the processing order rearranged, as long as it does not deviate from the main purpose.

[0789] The descriptions and illustrations presented above are detailed explanations of the technical aspects of this disclosure and are merely examples of the technical aspects. For example, the above descriptions of the structure, function, operation, and effect are examples of the structure, function, operation, and effect of the technical aspects of this disclosure. Therefore, it goes without saying that you may delete unnecessary parts, add new elements, or replace elements in the descriptions and illustrations presented above, as long as you do not deviate from the essence of the technical aspects of this disclosure. Furthermore, in order to avoid confusion and facilitate understanding of the technical aspects of this disclosure, explanations of common technical knowledge and the like that do not require special explanation to enable the implementation of the technical aspects of this disclosure have been omitted from the descriptions and illustrations presented above.

[0790] All documents, patent applications, and technical standards described herein are incorporated by reference to the same extent as if each individual document, patent application, and technical standard were specifically and individually noted as being incorporated by reference.

[0791] The following is further disclosed regarding the embodiments described above.

[0792] (Claim 1)

[0793] An information acquisition device for obtaining the position of an object,

[0794] A data storage device for saving acquired location information to a database in the cloud,

[0795] A search device for retrieving object location information from a cloud-based database and reporting it to the user,

[0796] A voice recognition device for analyzing user voice commands,

[0797] A system including monitoring devices for monitoring the movement of objects and detecting anomalies.

[0798] (Claim 2)

[0799] The system according to claim 1, wherein the information acquisition device further includes a suggestion device for recommending the optimal storage method for household items.

[0800] (Claim 3)

[0801] The system according to claim 1, wherein the information acquisition device further includes a learning device for learning user behavior patterns and generating predictive notifications for items.

[0802] "Example 1"

[0803] (Claim 1)

[0804] A detection means for obtaining the position of an item,

[0805] A recording means for saving acquired location information to a storage medium on the network,

[0806] A query mechanism for retrieving location information of an item from a storage medium on a network and reporting it to the user,

[0807] A voice processing means for analyzing the user's voice commands,

[0808] A monitoring system for monitoring the movement of goods and detecting abnormalities,

[0809] A learning method for learning user behavior patterns and generating predictive notifications,

[0810] A system including a proposal mechanism for suggesting the optimal arrangement method for items.

[0811] (Claim 2)

[0812] The system according to claim 1, wherein the monitoring means has the function of immediately detecting the inappropriate movement of important articles and sending a warning.

[0813] (Claim 3)

[0814] The system according to claim 1, wherein the above-mentioned voice processing means has a function for converting user instructions into text and transmitting them to a location information search means.

[0815] "Application Example 1"

[0816] (Claim 1)

[0817] A position detection means for obtaining the position of an object,

[0818] Information storage means for storing acquired location information in a remote data area,

[0819] An information retrieval means for extracting object location information from a remote data area and transmitting it to the user,

[0820] A voice interpretation means for interpreting the user's voice instructions,

[0821] A motion monitoring means for monitoring the movement of an object and recognizing anomalies,

[0822] A system that analyzes users' daily behavior and includes behavioral analysis and notification mechanisms to remind them of necessary items.

[0823] (Claim 2)

[0824] The system according to claim 1, wherein the information acquisition device further includes arrangement suggestion means for suggesting an optimized storage method for items in a living space.

[0825] (Claim 3)

[0826] The system according to claim 1, wherein the information acquisition device further includes learning and notification generation means for learning the user's behavioral habits and creating notifications for items.

[0827] "Example 2 of combining an emotion engine"

[0828] (Claim 1)

[0829] A detection means for obtaining the position of an object,

[0830] A storage means for saving acquired location information to a storage medium on a remote server,

[0831] A search method for retrieving the location information of an object from a storage medium on a remote server and reporting it to the user,

[0832] An analysis means for analyzing the user's voice commands,

[0833] A means of emotional analysis to analyze the emotional state of users and adjust the content and method of reporting,

[0834] A system that includes monitoring means for monitoring the movement of objects and detecting anomalies.

[0835] (Claim 2)

[0836] The system according to claim 1, wherein the detection means further includes a suggestion means for recommending an optimal storage method for items in a living space.

[0837] (Claim 3)

[0838] The system according to claim 1, wherein the detection means further includes a learning means for learning user behavior patterns and generating predictive notifications for items.

[0839] "Application example 2 when combining with an emotional engine"

[0840] (Claim 1)

[0841] A means for acquiring data to obtain the position of an object,

[0842] A data storage means for saving acquired location data to a storage device in the cloud,

[0843] A search method for retrieving object location data from cloud-based storage devices and reporting it to the user,

[0844] A speech recognition means for analyzing the user's voice instructions,

[0845] A monitoring means for monitoring the movement of an object and detecting anomalies,

[0846] A system that includes emotion analysis means for analyzing the emotional state of users and adjusting the method of information provision.

[0847] (Claim 2)

[0848] The system according to claim 1, wherein the information acquisition means further includes a suggestion means for recommending the optimal storage method for household items.

[0849] (Claim 3)

[0850] The system according to claim 1, wherein the information acquisition means further includes a learning means for learning user behavior patterns and generating predictive notifications for items. [Explanation of Symbols]

[0851] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Devices 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robots< / url:> < / url:> < / url:> < / url:>

Claims

1. An information acquisition device for obtaining the position of an object, A data storage device for saving acquired location information to a database in the cloud, A search device for retrieving object location information from a cloud-based database and reporting it to the user, A voice recognition device for analyzing user voice commands, A system including monitoring devices for monitoring the movement of objects and detecting anomalies.

2. The system according to claim 1, wherein the information acquisition device further includes a suggestion device for recommending the optimal storage method for household items.

3. The system according to claim 1, wherein the information acquisition device further includes a learning device for learning user behavior patterns and generating predictive notifications for items.

Images