system

Abstract

Provide a system. 【Solution means】 Means for acquiring the behavior of an animal in real time using a sensing device arranged in a home area; Means for analyzing the acquired image data and specifying the usage frequency of each device; Means for performing a usage evaluation and generating a list of devices determined to be unnecessary; Means for automatically providing device information by communicating with an information market based on the generated list of devices; Means for transmitting the device to a designated market based on the provided device information; Means for selecting unnecessary devices and contacting a recycler or provider to arrange for collection or provision; Means for automatically patrolling a home mobile device and monitoring the usage status of the device; Means for transferring the information provided by the sensing device to a terminal device in real time and generating statistical information; A system including the above.

Description

Technical Field

[0001] The technology of the present disclosure relates to a system.

Background Art

[0002] Patent Document 1 discloses a persona chatbot control method performed by at least one processor, including steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to an explanation of a chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] Due to the increase in pet supplies, the space in the home is compressed, and there is a problem that the quality of life of pets and their owners decreases. In addition, there is a problem that unused items are left unattended, which requires unnecessary time and effort for tidying up and increases the environmental load. Furthermore, since the disposal of unnecessary items is troublesome, there is also a problem that the processing is postponed.

Means for Solving the Problems

[0005] This invention uses an observation device installed in the home to acquire real-time data on an animal's daily activities and analyzes the video data to identify the frequency of use of each item. Based on this analysis, the usefulness of the items is evaluated, and a list of items deemed unnecessary is generated. The generated list is communicated to an electronic marketplace to automatically provide item information, and the items are then listed on the designated marketplace, reducing the effort required from the user. Furthermore, the system contacts recycling companies or donation recipients to arrange for the collection or donation of items deemed unnecessary. This system supports the proper decluttering of pet supplies and makes it possible to provide a comfortable living environment for both pet owners and their pets.

[0006] An "observation device" is a device installed in a home that collects video and data to capture the daily activities of animals in real time.

[0007] "Video data" refers to digital data that records animal activity acquired by observation devices, and is an information source that is analyzed based on image processing.

[0008] "Frequency of use" refers to the number of times or the duration of use of a particular item by an animal, and is an indicator used to evaluate the importance of an item.

[0009] "Usefulness" is an evaluation criterion that indicates the degree to which an item is necessary or valuable to the animal or its owner.

[0010] An "electronic marketplace" refers to an online platform where goods are bought and sold, and it allows for the automatic provision and listing of product information.

[0011] A "collection company" is a company that provides a service of picking up unwanted items from households and is responsible for their proper disposal.

[0012] A "recipient of donations" refers to an organization or facility that can make effective use of unwanted items, and is responsible for reusing items or providing them to people in need. [Brief explanation of the drawing]

[0013] [Figure 1] This is a conceptual diagram showing an example of the configuration of a data processing system according to the first embodiment. [Figure 2] This is a conceptual diagram showing an example of the essential functions of a data processing device and a smart device according to the first embodiment. [Figure 3] This is a conceptual diagram showing an example of the configuration of a data processing system according to the second embodiment. [Figure 4] This is a conceptual diagram showing an example of the main functions of a data processing device and smart glasses according to the second embodiment. [Figure 5] This is a conceptual diagram showing an example of the configuration of a data processing system according to the third embodiment. [Figure 6] This is a conceptual diagram showing an example of the main functions of a data processing device and a headset-type terminal according to the third embodiment. [Figure 7] This is a conceptual diagram showing an example of the configuration of a data processing system according to the fourth embodiment. [Figure 8] This is a conceptual diagram showing an example of the main functions of a data processing device and a robot according to the fourth embodiment. [Figure 9] This shows an emotion map where multiple emotions are mapped. [Figure 10] This shows an emotion map where multiple emotions are mapped. [Figure 11] This is a sequence diagram showing the processing flow of the data processing system in Example 1. [Figure 12] This is a sequence diagram showing the processing flow of the data processing system in Application Example 1. [Figure 13] This is a sequence diagram showing the processing flow of the data processing system in Example 2, which incorporates an emotion engine. [Figure 14] This is a sequence diagram showing the processing flow of the data processing system in Application Example 2, which combines an emotion engine. [Modes for carrying out the invention]

[0014] An example of an embodiment of the system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0015] First, the terms used in the following description will be explained.

[0016] In the following embodiments, the numbered processor (hereinafter simply referred to as "processor") may be a single arithmetic unit or a combination of multiple arithmetic units. Also, the processor may be a single type of arithmetic unit or a combination of multiple types of arithmetic units. Examples of arithmetic units include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), an APU (Accelerated Processing Unit), and the like.

[0017] In the following embodiments, the numbered RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a work memory by the processor.

[0018] In the following embodiments, the numbered storage is one or more non-volatile storage devices that store various programs and various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), or magnetic tapes, etc.

[0019] In the following embodiments, the signed communication interface (I / F) is an interface that includes a communication processor and an antenna, etc. The communication interface manages communication between multiple computers. Examples of communication standards applicable to the communication interface include wireless communication standards such as 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), or Bluetooth (registered trademark).

[0020] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." That is, "A and / or B" means that it may be A alone, or B alone, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" applies when expressing three or more things linked by "and / or."

[0021] [First Embodiment]

[0022] Figure 1 shows an example of the configuration of the data processing system 10 according to the first embodiment.

[0023] As shown in Figure 1, the data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0024] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0025] The smart device 14 comprises a computer 36, a reception device 38, an output device 40, a camera 42, and a communication interface 44. The computer 36 comprises a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The reception device 38, output device 40, and camera 42 are also connected to the bus 52.

[0026] The reception device 38 is equipped with a touch panel 38A and a microphone 38B, etc., and receives user input. The touch panel 38A receives user input by detecting contact with an object (e.g., a pen or finger). The microphone 38B receives user input by detecting the user's voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0027] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form perceptible to the user 20 (e.g., audio and / or text). The display 40A displays visible information such as text and images according to instructions from the processor 46. The speaker 40B outputs audio according to instructions from the processor 46. The camera 42 is a small digital camera equipped with an optical system such as a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0028] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various types of information between processor 46 and processor 28 via network 54.

[0029] Figure 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0030] As shown in Figure 2, in the data processing device 12, a specific processing is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" related to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0031] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0032] In the smart device 14, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The reception output program 60 is used in conjunction with a specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0033] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0034] This invention is a system that automatically observes and analyzes the usage of animal supplies and supports the organization and disposal of unnecessary items. This invention is mainly implemented using an observation device, a server, a terminal, and means for interacting with the user.

[0035] First, the device captures video data of the animal's daily activities through an observation device installed in the home. This data is compressed and processed, and then transmitted to a server via the network. The server receives the video data and uses a dedicated image processing algorithm and machine learning model to analyze the frequency of use of each item.

[0036] Next, the server evaluates the usefulness of animal products based on the analysis results. This takes into account statistical information, including the number of times and duration of use within a certain period. The usefulness evaluation generates a list of items deemed unnecessary. This list is stored in the database and provided to the user as a report as needed.

[0037] Through the generated report, users can view a list of items deemed unnecessary. For example, the list might include pet toys that haven't been used at all in the past month. In this case, the user can decide through the system whether to automatically list these items on a flea market site or have them taken by a recycling company or donation organization.

[0038] This system integrates with necessary network communication protocols to provide information on selected items to the online marketplace. Furthermore, the server automatically handles communication with recycling companies and donation recipients, ensuring that unwanted items are reused in a socially meaningful way. The goal is to reduce environmental impact while providing a comfortable living space for pets and their owners.

[0039] The following describes the processing flow.

[0040] Step 1:

[0041] The device activates the in-home observation equipment and records the animal's daily activities as video in real time. The acquired video data is temporarily stored in local storage.

[0042] Step 2:

[0043] The terminal compresses video data at predetermined time intervals and transmits it to the server via the internet. Security protocols are applied during data transmission to protect data confidentiality.

[0044] Step 3:

[0045] The server executes image processing algorithms to analyze the received video data. In this process, it identifies the objects that the animals have come into contact with and measures how often they are used.

[0046] Step 4:

[0047] The server evaluates the usefulness of each item based on usage frequency data. This evaluation is calculated, for example, based on the number of times it has been used in the past month.

[0048] Step 5:

[0049] The server generates a list of items deemed unnecessary based on the evaluation results. This list is prepared for notification to the user.

[0050] Step 6:

[0051] The user receives a notification and checks the item list through the system. The list displays the frequency of use and evaluation results for each item.

[0052] Step 7:

[0053] Users select an action regarding unwanted items. Options include automatically listing them on a flea market site or donating them to a recycling company or charity.

[0054] Step 8:

[0055] The server provides item information to the electronic marketplace or related businesses based on the user's selected action. The process is automated, including data transmission and scheduling of collection dates and times.

[0056] Step 9:

[0057] The server finally reports the processing results of the selected items to the user. This completes the entire system process.

[0058] (Example 1)

[0059] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0060] There is a need to efficiently identify and dispose of or reuse unnecessary items used in the home. In particular, it is difficult to accurately grasp the usage of everyday items through human observation, leading to the problem of accumulated waste. Furthermore, the disposal of unwanted items is time-consuming for users, often resulting in missed opportunities for social reuse.

[0061] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0062] In this invention, the server includes means for acquiring biological activity using an observation device, means for analyzing the acquired information and identifying the frequency of use of items, and means for evaluating the usefulness of items and generating a list of items deemed unnecessary. This makes it possible to accurately grasp usage patterns based on biological activity, enabling efficient identification and appropriate disposal of unnecessary items.

[0063] An "observation device" is a device installed in a home to collect information on the activity of living organisms, and its role is to acquire images and data.

[0064] "Acquired information" refers to data collected by observation devices, including content that specifically indicates the activity status of organisms.

[0065] "Items used" refers to various items that living organisms use on a daily basis, and their frequency of use and usefulness are the subjects of evaluation.

[0066] "Frequency of use" is an indicator obtained through the analysis of acquired information, showing how often each item was used within a specific period.

[0067] "Usefulness" is a criterion for evaluating how important a product is to the life of an organism, and is judged based on the frequency and duration of use.

[0068] An "item list" is a list generated based on an evaluation of the usefulness of the items used, and is used to identify items that have been deemed unnecessary.

[0069] An "information processing device" refers to a computing device that analyzes acquired information and processes various types of data through communication.

[0070] A "user" refers to an entity that utilizes the system, reviews the generated item list, and decides how to dispose of unwanted items.

[0071] This invention is a system for monitoring the activity of living organisms in the home, identifying unwanted items, and efficiently disposing of or reusing them. This system is primarily implemented using an observation device, a server, a terminal, and a user interface.

[0072] The device captures real-time images of the daily activities of living organisms through observation equipment installed in the home and acquires that information. Here, the camera device functions as an observation device, collecting the captured images as data. This data is compressed for efficient storage and transfer, and then transmitted to a server via the network.

[0073] The server analyzes the received information using a dedicated image processing algorithm and machine learning model. This analysis process utilizes open-source "machine learning libraries" used as machine learning frameworks, and specifically combines them with "image processing libraries" to identify organisms and objects being used in the video. This allows for the statistical calculation of the frequency and duration of use of each item.

[0074] Next, the server evaluates the usefulness of the items based on the analysis results and generates a list of items deemed unnecessary according to certain criteria. This list includes items that are not used or are used infrequently. This generated list is stored in a database and provided to the user.

[0075] Users can review unwanted items through the generated list and decide how to dispose of them. For example, a list might include pet toys that haven't been used for over a month. In this case, the user can use the system to decide whether to sell those items to a recycling shop, or to have them collected and disposed of or donated.

[0076] This system can connect selected item information with external markets and services via necessary communication protocols. Furthermore, the server automatically contacts recycling companies and donation recipients to ensure the socially beneficial use of unwanted items. This reduces the environmental impact while maintaining a comfortable living environment for pets and their owners.

[0077] Examples of prompts to input into a generative AI model:

[0078] "Please describe in detail the process of analyzing the frequency of use of pet supplies and optimizing the selection of unnecessary items."

[0079] This configuration allows the present invention to streamline the management of items within the home and promote the effective use of resources.

[0080] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0081] Step 1:

[0082] The device uses an observation system installed in the home to film the activity of living organisms and acquire video data. The input is the behavior and environment of the organisms, and the output is the filmed video clips. Specifically, the camera monitors 24 hours a day, creates clips at regular intervals, and records them.

[0083] Step 2:

[0084] If the video data acquired by the device is saved at its original quality, the file size becomes large. Therefore, to efficiently transfer the data over the network, the data is compressed using a lightweight codec. The input is raw video data, and the output is compressed video data. Specifically, video compression software is used to reduce the amount of data.

[0085] Step 3:

[0086] The terminal sends compressed video data to the server over the network. The input is the compressed video data, and the output is the server's response confirming the sent data. Specifically, the terminal uses a data transmission protocol to steadily upload the video to the server.

[0087] Step 4:

[0088] To analyze the video data received by the server, a dedicated image processing algorithm is used to identify organisms and objects within the video. The input is compressed video data, and the output is information about each identified item and its usage. Specifically, an image recognition library is used to identify items and obtain their location information.

[0089] Step 5:

[0090] The server applies a machine learning model to the identified information to evaluate the frequency and duration of use of items. The input is usage data for the identified items, and the output is statistical information regarding usage frequency and duration. Specifically, it uses a machine learning framework to analyze usage patterns and aggregate the results.

[0091] Step 6:

[0092] Based on the evaluation results, the server extracts items deemed to have low usefulness and generates a list of unnecessary items. The input is statistical information on used items, and the output is a list of unnecessary items. Specifically, it compares items with pre-set usefulness criteria and lists items that do not meet the criteria.

[0093] Step 7:

[0094] The server generates a list of unwanted items, stores it in a database, and provides it to the user. The input is the list of unwanted items, and the output is a report for the user. Specifically, the system periodically notifies the user of the list via an automated messaging system.

[0095] Step 8:

[0096] The user views the provided report and selects a disposal method for unwanted items. The input is the user's selection information, and the output is a record of the selected disposal method. Specifically, the user interface allows for options such as automatically listing unwanted items for sale or requesting collection.

[0097] Step 9:

[0098] The server processes items based on user instructions. Input is the user's disposal instructions, and output is the result of the executed disposal activities. Specifically, it sends data to flea market sites and contacts recycling companies to ensure the items are properly disposed of.

[0099] (Application Example 1)

[0100] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0101] The number of pet-related devices and items used in homes is increasing, making it difficult to manually manage their usage individually. Furthermore, delays in the disposal of unwanted devices and items can lead to wasted space and increased environmental burden. Therefore, efficient device management and the promotion of reuse are needed to realize a sustainable society.

[0102] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0103] In this invention, the server includes means for acquiring animal behavior in real time using sensing devices placed in the home area, means for analyzing the acquired image data and identifying the frequency of use of each device, and means for performing usage evaluations and generating a list of devices deemed unnecessary. This makes it possible to efficiently manage the use of devices and items in the home, quickly identify unnecessary items, and dispose of them in a reusable form.

[0104] The "household area" refers to the space within and around a house, where animals live on a daily basis.

[0105] A "sensing device" is a device that detects animal behavior in real time and acquires that information, and includes cameras and sensors.

[0106] "Behavior" refers to the activities and actions that animals perform within their home environment, including playing, eating, and resting.

[0107] "Acquired image data" refers to digital images and video information captured and collected by sensing devices.

[0108] "Frequency of use" is an indicator that shows the number of times or duration a particular device is used within a certain period of time.

[0109] "Usage evaluation" is the process of determining the usefulness and necessity of a device based on its usage frequency and other relevant data.

[0110] The "device list" is a list of identified devices, specifically those deemed unnecessary.

[0111] "Real-time transfer" refers to the process of immediately sending acquired data to a processing unit, meaning that information is exchanged almost simultaneously.

[0112] "Statistical information" refers to numerical summaries generated through data analysis, where information is organized according to certain rules.

[0113] The system that realizes this application consists of a home pet care robot that observes the animal's behavior through a sensing device. This robot is equipped with a camera and a communication module, which captures the animal's activity in real time and transmits the acquired data to a server in the cloud.

[0114] The server analyzes the received video data using the image processing library OpenCV to determine the usage frequency of each device. A machine learning model implemented in Python is used for data identification and classification.

[0115] Users can view analysis results from the server in real time using an application installed on their device. The application is designed using the mobile framework Flutter® and is responsible for notifying users of the generated list of devices and their usage evaluations.

[0116] As a concrete example, a robot can observe a cat toy for a week and generate a report indicating that it is an unused item. The user can then use an application to add the toy to a list and automatically list it on a flea market website.

[0117] An example of a prompt message is: "Based on data collected by a home pet care robot, think of ways to identify pet supplies that are no longer needed and notify the user of this information to help them manage them."

[0118] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0119] Step 1:

[0120] The terminal activates a home pet care robot, and the robot's camera captures the animal's behavior in real time. The input is visual data of the animal's behavior, and the output is a digitized video file of this data. The camera continuously records video and prepares to transmit the data to the next process.

[0121] Step 2:

[0122] The server receives digital video data transmitted from the terminal and uses the OpenCV library to identify specific actions or device usage within the video. The input is a digitized video file, and the output is analytical data indicating device usage. This process involves analyzing each frame to extract which devices the animal is using and to what extent.

[0123] Step 3:

[0124] The server uses a machine learning model implemented in Python based on the analyzed data to generate device usage frequency data. The input is analyzed data showing the device usage status, and the output is statistical information regarding usage frequency. As part of the data calculation, the number of times the device is used per hour is aggregated and the frequency is quantified.

[0125] Step 4:

[0126] The server uses statistical information to list infrequently used devices and generates a list of unnecessary devices. The input is usage statistics, and the output is a list of unnecessary devices. This list is created by identifying devices whose usage count over a month falls below a certain threshold.

[0127] Step 5:

[0128] The device receives a list of unnecessary devices sent from the server and notifies the user using a Flutter application. The input is a list of unnecessary devices, and the output is notification information for the user. This operation includes displaying the list and displaying a notification alert.

[0129] Step 6:

[0130] The user reviews the list through the application and decides whether to automatically list unwanted equipment on a flea market site or arrange for its collection. The input is the user's selection, and the output is the selected action. The user presses the list or collection arrangement button, and the result is fed back to the server.

[0131] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0132] This invention provides a system for streamlining the use and management of animal products, particularly enabling operation that takes user emotions into consideration. The system is implemented around an observation device, a server, a terminal, and an emotion engine.

[0133] First, the terminal monitors the animal's activity through an observation device installed in the home and acquires video data. This data is compressed and converted before being sent to a server via the network. The server analyzes the received data to identify the frequency of use of items used by the animal. Next, based on these results, the usefulness of the items is evaluated, and a list of items deemed unnecessary is generated.

[0134] In parallel, the server uses an emotion engine to recognize the emotions a user feels when accessing the system. This involves acquiring emotion data through interfaces such as cameras and microphones. The emotion engine analyzes the user's facial expressions and tone of voice to identify emotional states such as "pleasant" or "unpleasant."

[0135] When users review the list of items generated by the system, they receive an interactive experience tailored to their emotional state. For example, if the emotion engine determines that the user is feeling anxious about handling an item, the server adjusts the dialogue to provide suggestions and information that will make them feel more at ease. Specifically, this could include alternatives such as donating the item to a local community.

[0136] Furthermore, based on past emotional data, the system provides decision-making support tailored to the individual user's preferences. This reduces psychological resistance to disposing of unwanted items and promotes smoother decision-making.

[0137] In this way, the present invention provides a flexible interface that reflects the user's emotions and effectively supports the management of animal supplies.

[0138] The following describes the processing flow.

[0139] Step 1:

[0140] The device activates the in-home observation equipment and records the animal's daily activities as video in real time. The video data is temporarily stored in local storage.

[0141] Step 2:

[0142] The terminal compresses video data at predetermined time intervals and transmits it to the server via a secure connection. Security protocols are applied during data transmission to protect the data.

[0143] Step 3:

[0144] The server uses image processing algorithms to automatically identify items that animals have interacted with from the received video data. This allows the frequency of use of each item to be recorded.

[0145] Step 4:

[0146] The server evaluates the usefulness of items based on recorded usage frequency and lists items that do not meet certain evaluation criteria as unnecessary.

[0147] Step 5:

[0148] The server uses an emotion engine to recognize the user's emotions while they are interacting with the system. This recognition includes facial expression data and voice data captured by cameras and microphones.

[0149] Step 6:

[0150] The emotion engine determines the user's emotions through facial expression and voice analysis, detecting comfort and discomfort. Based on these results, the system dynamically adjusts its response.

[0151] Step 7:

[0152] Users review a list of unwanted items presented on the system and receive recommendations and alternatives tailored to their emotional state. For example, if the emotion engine detects the user's anxiety, the server suggests actions such as recommending donation destinations.

[0153] Step 8:

[0154] Based on the user's selected action, the server either lists the item on an online marketplace or sends the item information to a recycling company or donation recipient.

[0155] Step 9:

[0156] The server provides feedback to the user regarding the processed items and reports that the process is complete. The entire system, including consideration for emotions, creates an environment where users can comfortably declutter.

[0157] (Example 2)

[0158] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0159] In recent years, the number of households keeping pets has increased, leading to a greater complexity in managing pet-related items. In particular, users often experience emotional burdens when selecting and disposing of pet supplies. Furthermore, the lack of adequate guidance on proper disposal methods for unnecessary items has resulted in inefficient management.

[0160] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0161] In this invention, the server includes means for monitoring animal activity using observation devices installed in the home and acquiring data, means for analyzing the acquired data and identifying the frequency of use of items, and means for analyzing the user's emotions using an emotion engine and adjusting the interface. This enables the user to efficiently manage pet supplies and properly dispose of unwanted items while reducing emotional burden.

[0162] An "observation device" is a device installed in a home to monitor and record the behavior of animals, and may include cameras and sensors.

[0163] "Means of acquiring data" refers to the processes and technical means for collecting information in real time from observation devices.

[0164] "Means of data analysis" refers to algorithms and software used to process acquired information and analyze its content for use in specific purposes.

[0165] "Frequency of use of an item" is an indicator that shows how often an animal used a particular item during a specific period of time.

[0166] The "unnecessary items list" is a list of items that have been determined to be of low usefulness based on usage frequency analysis.

[0167] An "emotion engine" is a system or technology that recognizes and analyzes a user's emotions and adjusts the interface based on those emotions.

[0168] "Means of adjusting the interface" refers to a mechanism that changes the display method and dialogue format in order to provide optimal information in accordance with the user's emotional state.

[0169] A "means of supporting decision-making" refers to a system that provides information and suggestions to enable users to make choices and take actions effectively.

[0170] This invention is a system for streamlining the management of pet supplies in the home, aiming to enable operation that takes user emotions into consideration. The system consists of an observation device, a server, a terminal, and an emotion engine.

[0171] The device monitors animal activity using in-home observation equipment and acquires video data in real time. The observation equipment has built-in cameras and sensors to accurately record animal movements and behavior. This data is compressed and converted on the device and transmitted to a server via the internet. MPEG format is used as the data compression technology in this process.

[0172] The server analyzes the received data. Image processing software is used for the analysis, and image algorithms are employed to identify which items the animals used. Based on the identified frequency of use, the usefulness of the items is evaluated, and a list of items deemed unnecessary is generated.

[0173] In parallel, the server uses an emotion engine to acquire user emotion data. This emotion is recognized by analyzing data obtained through the camera and microphone. Facial recognition software and voice analysis tools analyze facial expressions and voice tone to identify emotions.

[0174] When users review the list of items generated by the system, they can receive interactive suggestions that correspond to their emotional state. For example, if a user expresses anxiety, the server may suggest donating to a local community to provide reassurance. In this way, users can be helped to dispose of unwanted items appropriately while reducing their psychological burden.

[0175] An example of a prompt for a generative AI model is: "Please identify my pet supplies that I don't use often or that others might need. Also, please provide gentle suggestions on how to dispose of them based on sentiment data."

[0176] This system efficiently supports the management of animal supplies by providing an approach that takes user emotions into consideration.

[0177] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0178] Step 1:

[0179] The terminal monitors animal activity in real time via an in-home observation device and acquires video data. In this step, a camera sensor records the animal's movements, and this video data is input to the terminal. The terminal compresses the received video data in a format such as MPEG and converts it into a format that can be sent to the server. Compressed video data is generated as output.

[0180] Step 2:

[0181] The server receives compressed video data sent from the terminal and begins analysis. Here, image processing algorithms are used to analyze animal behavior from the video and identify the items used. The input is compressed video data, and the output generates data on the identified items and their frequency of use.

[0182] Step 3:

[0183] The server executes a usefulness evaluation algorithm based on item usage frequency data. This data processing quantifies the usage frequency of each item and generates a list of items deemed unnecessary. The input is usage frequency data for identified items, and the output is a list of unnecessary items.

[0184] Step 4:

[0185] The server uses an emotion engine to analyze the user's facial expressions and tone of voice to determine their emotional state. Emotional data is collected through the camera and microphone and processed by analysis tools. Inputs include the user's facial expressions and voice data, and the output identifies the user's emotional state, such as "pleasant" or "unpleasant."

[0186] Step 5:

[0187] The user reviews a list of unwanted items generated by the system, and the server suggests appropriate disposal methods based on the user's emotional state. These suggestions may include options such as donating to the local community. User input consists of viewing the item list and selecting disposal methods, while the output provides personalized suggestions based on the user's emotional state.

[0188] Step 6:

[0189] The server references past emotional data to provide decision-making support tailored to the user's preferences. This reduces the user's psychological burden and facilitates the disposal of unwanted items. The input for this step is past emotional data, and the output is personalized decision-making support information.

[0190] (Application Example 2)

[0191] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0192] In modern households, pet supplies are diverse, and their management is crucial, but there is a lack of efficient management methods that take user feelings into consideration. As a result, unnecessary supplies accumulate in homes, occupying space and potentially increasing the environmental burden. Furthermore, there is a need for methods to dispose of unnecessary supplies without causing emotional distress to users. This invention aims to solve these problems.

[0193] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0194] In this invention, the server includes means for acquiring animal behavior in real time using a detection device placed in the home environment, means for analyzing the acquired visual information to identify the frequency of use of each item, and means for analyzing the user's emotional state. This makes it possible to manage pet supplies based on the user's emotional state.

[0195] A "detection device" is a device installed in a home to acquire information about an animal's behavior in real time.

[0196] "Visual information" refers to data that visually records the behavior and environment of animals.

[0197] "Item usage frequency" is an indicator that shows how often animal products are used.

[0198] An "e-commerce marketplace" is an online platform for exchanging product information and conducting transactions.

[0199] "User emotional state" refers to data that indicates the user's psychological or emotional condition.

[0200] "Sentiment analysis" is the process of analyzing a user's emotional state and generating suggestions that are tailored to that user.

[0201] To realize this invention, a detection device, a server, and a terminal for the user are required, all of which are installed in the home.

[0202] The server acquires real-time information on animal behavior through detection devices placed in the home environment. These detection devices include cameras and microphones, and monitor animal activity within the home, recording it as visual information. This information is transmitted to the server via the network, where it is analyzed. OpenCV is used as the image processing library for this analysis, and data processing is performed to identify the frequency of use of pet supplies.

[0203] The server evaluates the usefulness of items based on usage frequency data and generates a list of items deemed unnecessary. Furthermore, it analyzes the user's emotional state using an emotion engine. The emotional state is recognized through the user's facial expressions and voice, and the Google® Cloud Sentiment Analysis API is used for emotion analysis. Prompts are generated and suggestions are presented that are tailored to the user's emotions.

[0204] The user's device serves to display this information to the user. The user uses a dedicated smartphone app to check their pet's activity status and a list of unwanted items, and disposes of unwanted items based on the suggested best options. For example, they can donate unwanted pet beds to a local animal shelter. This allows the user to manage their belongings efficiently while gaining emotional satisfaction.

[0205] Examples of prompt statements that can be provided to the generation AI model include the following:

[0206] "Please suggest ways to manage animal supplies based on the user's feelings. If there are unused supplies, please suggest donation or reuse in a gentle manner."

[0207] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0208] Step 1:

[0209] The device acquires real-time animal activity data from detection devices installed in the home. This data includes video and audio data. The acquired data is compressed and converted into a format suitable for later analysis. The output is appropriately formatted visual information.

[0210] Step 2:

[0211] The server receives visual information transmitted from the terminal and performs analysis using the image processing library OpenCV. The main purpose of the analysis is to identify the frequency of use of animal supplies. Specifically, it uses object recognition technology to identify supplies in the video and measures the frequency of use of each. The output is data showing the frequency of use of each supply.

[0212] Step 3:

[0213] Based on the analysis results, the server evaluates the usefulness of each item and creates a list of items deemed unnecessary. This evaluation uses defined criteria, primarily selecting items with low usage frequency. The output is a list of items deemed unnecessary.

[0214] Step 4:

[0215] The server retrieves facial expressions and voice data from the user's device to analyze the user's emotional state. This data is sent to Google Cloud's Sentiment Analysis API for emotional analysis. Specifically, the API determines emotions such as "pleasant" or "unpleasant" based on facial expressions and voice tone. The output is data indicating the user's emotional state.

[0216] Step 5:

[0217] The server generates suggestions for the user based on the sentiment analysis results. The generated prompts are then processed through a generative AI model to provide user-appropriate donation and reuse suggestions. The output is information containing suggestions for the user.

[0218] Step 6:

[0219] Users review suggestions provided by the server via their device screen and make decisions regarding the disposal of unwanted items. Specifically, they select recipients for donation or choose to reuse items. The goal is to efficiently process unwanted items. The output consists of processing instructions based on the user's selections.

[0220] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0221] Data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of data generation model 58 is ChatGPT (registered trademark) (Internet search).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0222] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart device 14.

[0223] [Second Embodiment]

[0224] Figure 3 shows an example of the configuration of the data processing system 210 according to the second embodiment.

[0225] As shown in Figure 3, the data processing system 210 includes a data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0226] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0227] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication interface 44. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, and camera 42 are also connected to the bus 52.

[0228] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0229] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0230] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0231] Figure 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Figure 4, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0232] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0233] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0234] In the smart glasses 214, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0235] Next, the identification processing performed by the identification processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0236] This invention is a system that automatically observes and analyzes the usage of animal supplies and supports the organization and disposal of unnecessary items. This invention is mainly implemented using an observation device, a server, a terminal, and means for interacting with the user.

[0237] First, the device captures video data of the animal's daily activities through an observation device installed in the home. This data is compressed and processed, and then transmitted to a server via the network. The server receives the video data and uses a dedicated image processing algorithm and machine learning model to analyze the frequency of use of each item.

[0238] Next, the server evaluates the usefulness of animal products based on the analysis results. This takes into account statistical information, including the number of times and duration of use within a certain period. The usefulness evaluation generates a list of items deemed unnecessary. This list is stored in the database and provided to the user as a report as needed.

[0239] Through the generated report, users can view a list of items deemed unnecessary. For example, the list might include pet toys that haven't been used at all in the past month. In this case, the user can decide through the system whether to automatically list these items on a flea market site or have them taken by a recycling company or donation organization.

[0240] This system integrates with necessary network communication protocols to provide information on selected items to the online marketplace. Furthermore, the server automatically handles communication with recycling companies and donation recipients, ensuring that unwanted items are reused in a socially meaningful way. The goal is to reduce environmental impact while providing a comfortable living space for pets and their owners.

[0241] The following describes the processing flow.

[0242] Step 1:

[0243] The device activates the in-home observation equipment and records the animal's daily activities as video in real time. The acquired video data is temporarily stored in local storage.

[0244] Step 2:

[0245] The terminal compresses video data at predetermined time intervals and transmits it to the server via the internet. Security protocols are applied during data transmission to protect data confidentiality.

[0246] Step 3:

[0247] The server executes image processing algorithms to analyze the received video data. In this process, it identifies the objects that the animals have come into contact with and measures how often they are used.

[0248] Step 4:

[0249] The server evaluates the usefulness of each item based on usage frequency data. This evaluation is calculated, for example, based on the number of times it has been used in the past month.

[0250] Step 5:

[0251] The server generates a list of items deemed unnecessary based on the evaluation results. This list is prepared for notification to the user.

[0252] Step 6:

[0253] The user receives a notification and checks the item list through the system. The list displays the frequency of use and evaluation results for each item.

[0254] Step 7:

[0255] Users select an action regarding unwanted items. Options include automatically listing them on a flea market site or donating them to a recycling company or charity.

[0256] Step 8:

[0257] The server provides item information to the electronic marketplace or related businesses based on the user's selected action. The process is automated, including data transmission and scheduling of collection dates and times.

[0258] Step 9:

[0259] The server finally reports the processing results of the selected items to the user. This completes the entire system process.

[0260] (Example 1)

[0261] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0262] There is a need to efficiently identify and dispose of or reuse unnecessary items used in the home. In particular, it is difficult to accurately grasp the usage of everyday items through human observation, leading to the problem of accumulated waste. Furthermore, the disposal of unwanted items is time-consuming for users, often resulting in missed opportunities for social reuse.

[0263] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0264] In this invention, the server includes means for acquiring biological activity using an observation device, means for analyzing the acquired information and identifying the frequency of use of items, and means for evaluating the usefulness of items and generating a list of items deemed unnecessary. This makes it possible to accurately grasp usage patterns based on biological activity, enabling efficient identification and appropriate disposal of unnecessary items.

[0265] An "observation device" is a device installed in a home to collect information on the activity of living organisms, and its role is to acquire images and data.

[0266] "Acquired information" refers to data collected by observation devices, including content that specifically indicates the activity status of organisms.

[0267] "Items used" refers to various items that living organisms use on a daily basis, and their frequency of use and usefulness are the subjects of evaluation.

[0268] "Frequency of use" is an indicator obtained through the analysis of acquired information, showing how often each item was used within a specific period.

[0269] "Usefulness" is a criterion for evaluating how important a product is to the life of an organism, and is judged based on the frequency and duration of use.

[0270] An "item list" is a list generated based on an evaluation of the usefulness of the items used, and is used to identify items that have been deemed unnecessary.

[0271] An "information processing device" refers to a computing device that analyzes acquired information and processes various types of data through communication.

[0272] A "user" refers to an entity that utilizes the system, reviews the generated item list, and decides how to dispose of unwanted items.

[0273] This invention is a system for monitoring the activity of living organisms in the home, identifying unwanted items, and efficiently disposing of or reusing them. This system is primarily implemented using an observation device, a server, a terminal, and a user interface.

[0274] The device captures real-time images of the daily activities of living organisms through observation equipment installed in the home and acquires that information. Here, the camera device functions as an observation device, collecting the captured images as data. This data is compressed for efficient storage and transfer, and then transmitted to a server via the network.

[0275] The server analyzes the received information using a dedicated image processing algorithm and machine learning model. This analysis process utilizes open-source "machine learning libraries" used as machine learning frameworks, and specifically combines them with "image processing libraries" to identify organisms and objects being used in the video. This allows for the statistical calculation of the frequency and duration of use of each item.

[0276] Next, the server evaluates the usefulness of the items based on the analysis results and generates a list of items deemed unnecessary according to certain criteria. This list includes items that are not used or are used infrequently. This generated list is stored in a database and provided to the user.

[0277] Users can review unwanted items through the generated list and decide how to dispose of them. For example, a list might include pet toys that haven't been used for over a month. In this case, the user can use the system to decide whether to sell those items to a recycling shop, or to have them collected and disposed of or donated.

[0278] This system can connect selected item information with external markets and services via necessary communication protocols. Furthermore, the server automatically contacts recycling companies and donation recipients to ensure the socially beneficial use of unwanted items. This reduces the environmental impact while maintaining a comfortable living environment for pets and their owners.

[0279] Examples of prompts to input into a generative AI model:

[0280] "Please describe in detail the process of analyzing the frequency of use of pet supplies and optimizing the selection of unnecessary items."

[0281] This configuration allows the present invention to streamline the management of items within the home and promote the effective use of resources.

[0282] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0283] Step 1:

[0284] The terminal uses an observation device installed in the home to photograph the activities of organisms and obtain video data. The input is the behavior and environment of the organisms, and the output is the photographed video clip. As a specific operation, the camera conducts 24-hour monitoring, creates clips at regular intervals, and records them.

[0285] Step 2:

[0286] If the terminal stores the acquired video data in its original quality, the size will be large. Therefore, to efficiently transfer it through the network, the data is compressed with a lightweight codec. The input is the raw video data, and the output is the compressed video data. Specifically, video compression software is used to reduce the data volume.

[0287] Step 3:

[0288] The terminal transmits the compressed video data to the server via the network. The input is the compressed video data, and the output is the response of data confirmation sent to the server. As a specific operation, the terminal steadily uploads the video to the server using the data transmission protocol.

[0289] Step 4:

[0290] To analyze the video data received by the server, a dedicated image processing algorithm is used to identify organisms and used items in the video. The input is the compressed video data, and the output is the information on each identified item and its usage status. As a specific task, an image recognition library is utilized to identify the items and obtain their location information.

[0291] Step 5:

[0292] The server applies a machine learning model to the identified information to evaluate the usage frequency and usage time of the used items. The input is the usage data of the identified items, and the output is the statistical information on the usage frequency and usage time. As a specific operation, a machine learning framework is used to analyze the usage patterns and aggregate the results.

[0293] Step 6:

[0294] Based on the evaluation results, the server extracts items deemed to have low usefulness and generates a list of unnecessary items. The input is statistical information on used items, and the output is a list of unnecessary items. Specifically, it compares items with pre-set usefulness criteria and lists items that do not meet the criteria.

[0295] Step 7:

[0296] The server generates a list of unwanted items, stores it in a database, and provides it to the user. The input is the list of unwanted items, and the output is a report for the user. Specifically, the system periodically notifies the user of the list via an automated messaging system.

[0297] Step 8:

[0298] The user views the provided report and selects a disposal method for unwanted items. The input is the user's selection information, and the output is a record of the selected disposal method. Specifically, the user interface allows for options such as automatically listing unwanted items for sale or requesting collection.

[0299] Step 9:

[0300] The server processes items based on user instructions. Input is the user's disposal instructions, and output is the result of the executed disposal activities. Specifically, it sends data to flea market sites and contacts recycling companies to ensure the items are properly disposed of.

[0301] (Application Example 1)

[0302] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0303] The number of devices and items for animals used within the home has increased, making it difficult to manually manage their usage individually. Additionally, the disposal of unnecessary devices and items is delayed, potentially leading to wasted space and increased environmental impact. Therefore, there is a need to promote efficient device management and reuse towards the realization of a sustainable society.

[0304] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following respective means.

[0305] In this invention, the server includes means for acquiring the behavior of an animal in real time using a sensing device arranged in the home area, means for analyzing the acquired image data to specify the usage frequency of each device, and means for performing a usage evaluation and generating a list of devices determined to be unnecessary. Thereby, it becomes possible to efficiently manage the usage of devices and items within the home, quickly identify unnecessary items, and dispose of them in a reusable form.

[0306] The "home area" refers to the space of the house and its surroundings, which is the place where the animal lives daily.

[0307] The "sensing device" is a device for sensing the behavior of an animal in real time and acquiring its information, and includes a camera and a sensor.

[0308] "Behavior" refers to the activities and actions performed by the animal within the home area, including playing, eating, resting, etc.

[0309] The "acquired image data" refers to the information of digital images and videos captured and collected by the sensing device.

[0310] The "usage frequency" is an index indicating the number of times or the time during which a specific device is used within a certain period.

[0311] The "usage evaluation" is a process of determining the usefulness and necessity of a device based on the usage frequency of the device and other related data.

[0312] The "device list" is a list of identified devices, specifically those deemed unnecessary.

[0313] "Real-time transfer" refers to the process of immediately sending acquired data to a processing unit, meaning that information is exchanged almost simultaneously.

[0314] "Statistical information" refers to numerical summaries generated through data analysis, where information is organized according to certain rules.

[0315] The system that realizes this application consists of a home pet care robot that observes the animal's behavior through a sensing device. This robot is equipped with a camera and a communication module, which captures the animal's activity in real time and transmits the acquired data to a server in the cloud.

[0316] The server analyzes the received video data using the image processing library OpenCV to determine the usage frequency of each device. A machine learning model implemented in Python is used for data identification and classification.

[0317] Users can view analysis results from the server in real time using an application installed on their device. The application, designed using the mobile framework Flutter, is responsible for notifying users of the generated device list and its usage evaluation.

[0318] As a concrete example, a robot can observe a cat toy for a week and generate a report indicating that it is an unused item. The user can then use an application to add the toy to a list and automatically list it on a flea market website.

[0319] An example of a prompt message is: "Based on data collected by a home pet care robot, think of ways to identify pet supplies that are no longer needed and notify the user of this information to help them manage them."

[0320] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0321] Step 1:

[0322] The terminal activates a home pet care robot, and the robot's camera captures the animal's behavior in real time. The input is visual data of the animal's behavior, and the output is a digitized video file of this data. The camera continuously records video and prepares to transmit the data to the next process.

[0323] Step 2:

[0324] The server receives digital video data transmitted from the terminal and uses the OpenCV library to identify specific actions or device usage within the video. The input is a digitized video file, and the output is analytical data indicating device usage. This process involves analyzing each frame to extract which devices the animal is using and to what extent.

[0325] Step 3:

[0326] The server uses a machine learning model implemented in Python based on the analyzed data to generate device usage frequency data. The input is analyzed data showing the device usage status, and the output is statistical information regarding usage frequency. As part of the data calculation, the number of times the device is used per hour is aggregated and the frequency is quantified.

[0327] Step 4:

[0328] The server uses statistical information to list infrequently used devices and generates a list of unnecessary devices. The input is usage statistics, and the output is a list of unnecessary devices. This list is created by identifying devices whose usage count over a month falls below a certain threshold.

[0329] Step 5:

[0330] The device receives a list of unnecessary devices sent from the server and notifies the user using a Flutter application. The input is a list of unnecessary devices, and the output is notification information for the user. This operation includes displaying the list and displaying a notification alert.

[0331] Step 6:

[0332] The user reviews the list through the application and decides whether to automatically list unwanted equipment on a flea market site or arrange for its collection. The input is the user's selection, and the output is the selected action. The user presses the list or collection arrangement button, and the result is fed back to the server.

[0333] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0334] This invention provides a system for streamlining the use and management of animal products, particularly enabling operation that takes user emotions into consideration. The system is implemented around an observation device, a server, a terminal, and an emotion engine.

[0335] First, the terminal monitors the animal's activity through an observation device installed in the home and acquires video data. This data is compressed and converted before being sent to a server via the network. The server analyzes the received data to identify the frequency of use of items used by the animal. Next, based on these results, the usefulness of the items is evaluated, and a list of items deemed unnecessary is generated.

[0336] In parallel, the server uses an emotion engine to recognize the emotions a user feels when accessing the system. This involves acquiring emotion data through interfaces such as cameras and microphones. The emotion engine analyzes the user's facial expressions and tone of voice to identify emotional states such as "pleasant" or "unpleasant."

[0337] When users review the list of items generated by the system, they receive an interactive experience tailored to their emotional state. For example, if the emotion engine determines that the user is feeling anxious about handling an item, the server adjusts the dialogue to provide suggestions and information that will make them feel more at ease. Specifically, this could include alternatives such as donating the item to a local community.

[0338] Furthermore, based on past emotional data, the system provides decision-making support tailored to the individual user's preferences. This reduces psychological resistance to disposing of unwanted items and promotes smoother decision-making.

[0339] In this way, the present invention provides a flexible interface that reflects the user's emotions and effectively supports the management of animal supplies.

[0340] The following describes the processing flow.

[0341] Step 1:

[0342] The device activates the in-home observation equipment and records the animal's daily activities as video in real time. The video data is temporarily stored in local storage.

[0343] Step 2:

[0344] The terminal compresses video data at predetermined time intervals and transmits it to the server via a secure connection. Security protocols are applied during data transmission to protect the data.

[0345] Step 3:

[0346] The server uses image processing algorithms to automatically identify items that animals have interacted with from the received video data. This allows the frequency of use of each item to be recorded.

[0347] Step 4:

[0348] The server evaluates the usefulness of items based on recorded usage frequency and lists items that do not meet certain evaluation criteria as unnecessary.

[0349] Step 5:

[0350] The server uses an emotion engine to recognize the user's emotions while they are interacting with the system. This recognition includes facial expression data and voice data captured by cameras and microphones.

[0351] Step 6:

[0352] The emotion engine determines the user's emotions through facial expression and voice analysis, detecting comfort and discomfort. Based on these results, the system dynamically adjusts its response.

[0353] Step 7:

[0354] Users review a list of unwanted items presented on the system and receive recommendations and alternatives tailored to their emotional state. For example, if the emotion engine detects the user's anxiety, the server suggests actions such as recommending donation destinations.

[0355] Step 8:

[0356] Based on the user's selected action, the server either lists the item on an online marketplace or sends the item information to a recycling company or donation recipient.

[0357] Step 9:

[0358] The server provides feedback to the user regarding the processed items and reports that the process is complete. The entire system, including consideration for emotions, creates an environment where users can comfortably declutter.

[0359] (Example 2)

[0360] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0361] In recent years, the number of households keeping pets has increased, leading to a greater complexity in managing pet-related items. In particular, users often experience emotional burdens when selecting and disposing of pet supplies. Furthermore, the lack of adequate guidance on proper disposal methods for unnecessary items has resulted in inefficient management.

[0362] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0363] In this invention, the server includes means for monitoring animal activity using observation devices installed in the home and acquiring data, means for analyzing the acquired data and identifying the frequency of use of items, and means for analyzing the user's emotions using an emotion engine and adjusting the interface. This enables the user to efficiently manage pet supplies and properly dispose of unwanted items while reducing emotional burden.

[0364] An "observation device" is a device installed in a home to monitor and record the behavior of animals, and may include cameras and sensors.

[0365] "Means of acquiring data" refers to the processes and technical means for collecting information in real time from observation devices.

[0366] "Means of data analysis" refers to algorithms and software used to process acquired information and analyze its content for use in specific purposes.

[0367] "Frequency of use of an item" is an indicator that shows how often an animal used a particular item during a specific period of time.

[0368] The "unnecessary items list" is a list of items that have been determined to be of low usefulness based on usage frequency analysis.

[0369] An "emotion engine" is a system or technology that recognizes and analyzes a user's emotions and adjusts the interface based on those emotions.

[0370] "Means of adjusting the interface" refers to a mechanism that changes the display method and dialogue format in order to provide optimal information in accordance with the user's emotional state.

[0371] A "means of supporting decision-making" refers to a system that provides information and suggestions to enable users to make choices and take actions effectively.

[0372] This invention is a system for streamlining the management of pet supplies in the home, aiming to enable operation that takes user emotions into consideration. The system consists of an observation device, a server, a terminal, and an emotion engine.

[0373] The device monitors animal activity using in-home observation equipment and acquires video data in real time. The observation equipment has built-in cameras and sensors to accurately record animal movements and behavior. This data is compressed and converted on the device and transmitted to a server via the internet. MPEG format is used as the data compression technology in this process.

[0374] The server analyzes the received data. Image processing software is used for the analysis, and image algorithms are employed to identify which items the animals used. Based on the identified frequency of use, the usefulness of the items is evaluated, and a list of items deemed unnecessary is generated.

[0375] In parallel, the server uses an emotion engine to acquire user emotion data. This emotion is recognized by analyzing data obtained through the camera and microphone. Facial recognition software and voice analysis tools analyze facial expressions and voice tone to identify emotions.

[0376] When users review the list of items generated by the system, they can receive interactive suggestions that correspond to their emotional state. For example, if a user expresses anxiety, the server may suggest donating to a local community to provide reassurance. In this way, users can be helped to dispose of unwanted items appropriately while reducing their psychological burden.

[0377] An example of a prompt for a generative AI model is: "Please identify my pet supplies that I don't use often or that others might need. Also, please provide gentle suggestions on how to dispose of them based on sentiment data."

[0378] This system efficiently supports the management of animal supplies by providing an approach that takes user emotions into consideration.

[0379] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0380] Step 1:

[0381] The terminal monitors animal activity in real time via an in-home observation device and acquires video data. In this step, a camera sensor records the animal's movements, and this video data is input to the terminal. The terminal compresses the received video data in a format such as MPEG and converts it into a format that can be sent to the server. Compressed video data is generated as output.

[0382] Step 2:

[0383] The server receives compressed video data sent from the terminal and begins analysis. Here, image processing algorithms are used to analyze animal behavior from the video and identify the items used. The input is compressed video data, and the output generates data on the identified items and their frequency of use.

[0384] Step 3:

[0385] The server executes a usefulness evaluation algorithm based on item usage frequency data. This data processing quantifies the usage frequency of each item and generates a list of items deemed unnecessary. The input is usage frequency data for identified items, and the output is a list of unnecessary items.

[0386] Step 4:

[0387] The server uses an emotion engine to analyze the user's facial expressions and tone of voice to determine their emotional state. Emotional data is collected through the camera and microphone and processed by analysis tools. Inputs include the user's facial expressions and voice data, and the output identifies the user's emotional state, such as "pleasant" or "unpleasant."

[0388] Step 5:

[0389] The user reviews a list of unwanted items generated by the system, and the server suggests appropriate disposal methods based on the user's emotional state. These suggestions may include options such as donating to the local community. User input consists of viewing the item list and selecting disposal methods, while the output provides personalized suggestions based on the user's emotional state.

[0390] Step 6:

[0391] The server references past emotional data to provide decision-making support tailored to the user's preferences. This reduces the user's psychological burden and facilitates the disposal of unwanted items. The input for this step is past emotional data, and the output is personalized decision-making support information.

[0392] (Application Example 2)

[0393] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0394] In modern households, pet supplies are diverse, and their management is crucial, but there is a lack of efficient management methods that take user feelings into consideration. As a result, unnecessary supplies accumulate in homes, occupying space and potentially increasing the environmental burden. Furthermore, there is a need for methods to dispose of unnecessary supplies without causing emotional distress to users. This invention aims to solve these problems.

[0395] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0396] In this invention, the server includes means for acquiring animal behavior in real time using a detection device placed in the home environment, means for analyzing the acquired visual information to identify the frequency of use of each item, and means for analyzing the user's emotional state. This makes it possible to manage pet supplies based on the user's emotional state.

[0397] A "detection device" is a device installed in a home to acquire information about an animal's behavior in real time.

[0398] "Visual information" refers to data that visually records the behavior and environment of animals.

[0399] "Item usage frequency" is an indicator that shows how often animal products are used.

[0400] An "e-commerce marketplace" is an online platform for exchanging product information and conducting transactions.

[0401] "User emotional state" refers to data that indicates the user's psychological or emotional condition.

[0402] "Sentiment analysis" is the process of analyzing a user's emotional state and generating suggestions that are tailored to that user.

[0403] To realize this invention, a detection device, a server, and a terminal for the user are required, all of which are installed in the home.

[0404] The server acquires real-time information on animal behavior through detection devices placed in the home environment. These detection devices include cameras and microphones, and monitor animal activity within the home, recording it as visual information. This information is transmitted to the server via the network, where it is analyzed. OpenCV is used as the image processing library for this analysis, and data processing is performed to identify the frequency of use of pet supplies.

[0405] The server evaluates the usefulness of items based on usage frequency data and generates a list of items deemed unnecessary. Furthermore, it analyzes the user's emotional state using an emotion engine. The emotional state is recognized from the user's facial expressions and voice, and the Google Cloud Sentiment Analysis API is used for emotion analysis. Prompts are generated and suggestions are presented that are tailored to the user's emotions.

[0406] The user's device serves to display this information to the user. The user uses a dedicated smartphone app to check their pet's activity status and a list of unwanted items, and disposes of unwanted items based on the suggested best options. For example, they can donate unwanted pet beds to a local animal shelter. This allows the user to manage their belongings efficiently while gaining emotional satisfaction.

[0407] Examples of prompt statements that can be provided to the generation AI model include the following:

[0408] "Please suggest ways to manage animal supplies based on the user's feelings. If there are unused supplies, please suggest donation or reuse in a gentle manner."

[0409] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0410] Step 1:

[0411] The device acquires real-time animal activity data from detection devices installed in the home. This data includes video and audio data. The acquired data is compressed and converted into a format suitable for later analysis. The output is appropriately formatted visual information.

[0412] Step 2:

[0413] The server receives visual information transmitted from the terminal and performs analysis using the image processing library OpenCV. The main purpose of the analysis is to identify the frequency of use of animal supplies. Specifically, it uses object recognition technology to identify supplies in the video and measures the frequency of use of each. The output is data showing the frequency of use of each supply.

[0414] Step 3:

[0415] Based on the analysis results, the server evaluates the usefulness of each item and creates a list of items deemed unnecessary. This evaluation uses defined criteria, primarily selecting items with low usage frequency. The output is a list of items deemed unnecessary.

[0416] Step 4:

[0417] The server retrieves facial expressions and voice data from the user's device to analyze the user's emotional state. This data is sent to Google Cloud's Sentiment Analysis API for emotional analysis. Specifically, the API determines emotions such as "pleasant" or "unpleasant" based on facial expressions and voice tone. The output is data indicating the user's emotional state.

[0418] Step 5:

[0419] The server generates suggestions for the user based on the sentiment analysis results. The generated prompts are then processed through a generative AI model to provide user-appropriate donation and reuse suggestions. The output is information containing suggestions for the user.

[0420] Step 6:

[0421] Users review suggestions provided by the server via their device screen and make decisions regarding the disposal of unwanted items. Specifically, they select recipients for donation or choose to reuse items. The goal is to efficiently process unwanted items. The output consists of processing instructions based on the user's selections.

[0422] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0423] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0424] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart glasses 214.

[0425] [Third Embodiment]

[0426] Figure 5 shows an example of the configuration of the data processing system 310 according to the third embodiment.

[0427] As shown in Figure 5, the data processing system 310 includes a data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0428] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0429] The headset terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a display 343. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and display 343 are also connected to the bus 52.

[0430] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0431] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0432] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0433] Figure 6 shows an example of the main functions of the data processing device 12 and the headset terminal 314. As shown in Figure 6, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0434] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0435] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0436] In the headset terminal 314, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0437] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the headset terminal 314 will be referred to as the "terminal".

[0438] This invention is a system that automatically observes and analyzes the usage of animal supplies and supports the organization and disposal of unnecessary items. This invention is mainly implemented using an observation device, a server, a terminal, and means for interacting with the user.

[0439] First, the device captures video data of the animal's daily activities through an observation device installed in the home. This data is compressed and processed, and then transmitted to a server via the network. The server receives the video data and uses a dedicated image processing algorithm and machine learning model to analyze the frequency of use of each item.

[0440] Next, the server evaluates the usefulness of animal products based on the analysis results. This takes into account statistical information, including the number of times and duration of use within a certain period. The usefulness evaluation generates a list of items deemed unnecessary. This list is stored in the database and provided to the user as a report as needed.

[0441] Through the generated report, users can view a list of items deemed unnecessary. For example, the list might include pet toys that haven't been used at all in the past month. In this case, the user can decide through the system whether to automatically list these items on a flea market site or have them taken by a recycling company or donation organization.

[0442] This system integrates with necessary network communication protocols to provide information on selected items to the online marketplace. Furthermore, the server automatically handles communication with recycling companies and donation recipients, ensuring that unwanted items are reused in a socially meaningful way. The goal is to reduce environmental impact while providing a comfortable living space for pets and their owners.

[0443] The following describes the processing flow.

[0444] Step 1:

[0445] The device activates the in-home observation equipment and records the animal's daily activities as video in real time. The acquired video data is temporarily stored in local storage.

[0446] Step 2:

[0447] The terminal compresses video data at predetermined time intervals and transmits it to the server via the internet. Security protocols are applied during data transmission to protect data confidentiality.

[0448] Step 3:

[0449] The server executes image processing algorithms to analyze the received video data. In this process, it identifies the objects that the animals have come into contact with and measures how often they are used.

[0450] Step 4:

[0451] The server evaluates the usefulness of each item based on usage frequency data. This evaluation is calculated, for example, based on the number of times it has been used in the past month.

[0452] Step 5:

[0453] The server generates a list of items deemed unnecessary based on the evaluation results. This list is prepared for notification to the user.

[0454] Step 6:

[0455] The user receives a notification and checks the item list through the system. The list displays the frequency of use and evaluation results for each item.

[0456] Step 7:

[0457] Users select an action regarding unwanted items. Options include automatically listing them on a flea market site or donating them to a recycling company or charity.

[0458] Step 8:

[0459] The server provides item information to the electronic marketplace or related businesses based on the user's selected action. The process is automated, including data transmission and scheduling of collection dates and times.

[0460] Step 9:

[0461] The server finally reports the processing results of the selected items to the user. This completes the entire system process.

[0462] (Example 1)

[0463] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0464] There is a need to efficiently identify and dispose of or reuse unnecessary items used in the home. In particular, it is difficult to accurately grasp the usage of everyday items through human observation, leading to the problem of accumulated waste. Furthermore, the disposal of unwanted items is time-consuming for users, often resulting in missed opportunities for social reuse.

[0465] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0466] In this invention, the server includes means for acquiring biological activity using an observation device, means for analyzing the acquired information and identifying the frequency of use of items, and means for evaluating the usefulness of items and generating a list of items deemed unnecessary. This makes it possible to accurately grasp usage patterns based on biological activity, enabling efficient identification and appropriate disposal of unnecessary items.

[0467] An "observation device" is a device installed in a home to collect information on the activity of living organisms, and its role is to acquire images and data.

[0468] "Acquired information" refers to data collected by observation devices, including content that specifically indicates the activity status of organisms.

[0469] "Items used" refers to various items that living organisms use on a daily basis, and their frequency of use and usefulness are the subjects of evaluation.

[0470] "Frequency of use" is an indicator obtained through the analysis of acquired information, showing how often each item was used within a specific period.

[0471] "Usefulness" is a criterion for evaluating how important a product is to the life of an organism, and is judged based on the frequency and duration of use.

[0472] An "item list" is a list generated based on an evaluation of the usefulness of the items used, and is used to identify items that have been deemed unnecessary.

[0473] An "information processing device" refers to a computing device that analyzes acquired information and processes various types of data through communication.

[0474] A "user" refers to an entity that utilizes the system, reviews the generated item list, and decides how to dispose of unwanted items.

[0475] This invention is a system for monitoring the activity of living organisms in the home, identifying unwanted items, and efficiently disposing of or reusing them. This system is primarily implemented using an observation device, a server, a terminal, and a user interface.

[0476] The device captures real-time images of the daily activities of living organisms through observation equipment installed in the home and acquires that information. Here, the camera device functions as an observation device, collecting the captured images as data. This data is compressed for efficient storage and transfer, and then transmitted to a server via the network.

[0477] The server analyzes the received information using a dedicated image processing algorithm and machine learning model. This analysis process utilizes open-source "machine learning libraries" used as machine learning frameworks, and specifically combines them with "image processing libraries" to identify organisms and objects being used in the video. This allows for the statistical calculation of the frequency and duration of use of each item.

[0478] Next, the server evaluates the usefulness of the items based on the analysis results and generates a list of items deemed unnecessary according to certain criteria. This list includes items that are not used or are used infrequently. This generated list is stored in a database and provided to the user.

[0479] Users can review unwanted items through the generated list and decide how to dispose of them. For example, a list might include pet toys that haven't been used for over a month. In this case, the user can use the system to decide whether to sell those items to a recycling shop, or to have them collected and disposed of or donated.

[0480] This system can connect selected item information with external markets and services via necessary communication protocols. Furthermore, the server automatically contacts recycling companies and donation recipients to ensure the socially beneficial use of unwanted items. This reduces the environmental impact while maintaining a comfortable living environment for pets and their owners.

[0481] Examples of prompts to input into a generative AI model:

[0482] "Please describe in detail the process of analyzing the frequency of use of pet supplies and optimizing the selection of unnecessary items."

[0483] This configuration allows the present invention to streamline the management of items within the home and promote the effective use of resources.

[0484] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0485] Step 1:

[0486] The device uses an observation system installed in the home to film the activity of living organisms and acquire video data. The input is the behavior and environment of the organisms, and the output is the filmed video clips. Specifically, the camera monitors 24 hours a day, creates clips at regular intervals, and records them.

[0487] Step 2:

[0488] If the video data acquired by the device is saved at its original quality, the file size becomes large. Therefore, to efficiently transfer the data over the network, the data is compressed using a lightweight codec. The input is raw video data, and the output is compressed video data. Specifically, video compression software is used to reduce the amount of data.

[0489] Step 3:

[0490] The terminal sends compressed video data to the server over the network. The input is the compressed video data, and the output is the server's response confirming the sent data. Specifically, the terminal uses a data transmission protocol to steadily upload the video to the server.

[0491] Step 4:

[0492] To analyze the video data received by the server, a dedicated image processing algorithm is used to identify organisms and objects within the video. The input is compressed video data, and the output is information about each identified item and its usage. Specifically, an image recognition library is used to identify items and obtain their location information.

[0493] Step 5:

[0494] The server applies a machine learning model to the identified information to evaluate the frequency and duration of use of items. The input is usage data for the identified items, and the output is statistical information regarding usage frequency and duration. Specifically, it uses a machine learning framework to analyze usage patterns and aggregate the results.

[0495] Step 6:

[0496] Based on the evaluation results, the server extracts items deemed to have low usefulness and generates a list of unnecessary items. The input is statistical information on used items, and the output is a list of unnecessary items. Specifically, it compares items with pre-set usefulness criteria and lists items that do not meet the criteria.

[0497] Step 7:

[0498] The server generates a list of unwanted items, stores it in a database, and provides it to the user. The input is the list of unwanted items, and the output is a report for the user. Specifically, the system periodically notifies the user of the list via an automated messaging system.

[0499] Step 8:

[0500] The user views the provided report and selects a disposal method for unwanted items. The input is the user's selection information, and the output is a record of the selected disposal method. Specifically, the user interface allows for options such as automatically listing unwanted items for sale or requesting collection.

[0501] Step 9:

[0502] The server processes items based on user instructions. Input is the user's disposal instructions, and output is the result of the executed disposal activities. Specifically, it sends data to flea market sites and contacts recycling companies to ensure the items are properly disposed of.

[0503] (Application Example 1)

[0504] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0505] The number of pet-related devices and items used in homes is increasing, making it difficult to manually manage their usage individually. Furthermore, delays in the disposal of unwanted devices and items can lead to wasted space and increased environmental burden. Therefore, efficient device management and the promotion of reuse are needed to realize a sustainable society.

[0506] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0507] In this invention, the server includes means for acquiring animal behavior in real time using sensing devices placed in the home area, means for analyzing the acquired image data and identifying the frequency of use of each device, and means for performing usage evaluations and generating a list of devices deemed unnecessary. This makes it possible to efficiently manage the use of devices and items in the home, quickly identify unnecessary items, and dispose of them in a reusable form.

[0508] The "household area" refers to the space within and around a house, where animals live on a daily basis.

[0509] A "sensing device" is a device that detects animal behavior in real time and acquires that information, and includes cameras and sensors.

[0510] "Behavior" refers to the activities and actions that animals perform within their home environment, including playing, eating, and resting.

[0511] "Acquired image data" refers to digital images and video information captured and collected by sensing devices.

[0512] "Frequency of use" is an indicator that shows the number of times or duration a particular device is used within a certain period of time.

[0513] "Usage evaluation" is the process of determining the usefulness and necessity of a device based on its usage frequency and other relevant data.

[0514] The "device list" is a list of identified devices, specifically those deemed unnecessary.

[0515] "Real-time transfer" refers to the process of immediately sending acquired data to a processing unit, meaning that information is exchanged almost simultaneously.

[0516] "Statistical information" refers to numerical summaries generated through data analysis, where information is organized according to certain rules.

[0517] The system that realizes this application consists of a home pet care robot that observes the animal's behavior through a sensing device. This robot is equipped with a camera and a communication module, which captures the animal's activity in real time and transmits the acquired data to a server in the cloud.

[0518] The server analyzes the received video data using the image processing library OpenCV to determine the usage frequency of each device. A machine learning model implemented in Python is used for data identification and classification.

[0519] Users can view analysis results from the server in real time using an application installed on their device. The application, designed using the mobile framework Flutter, is responsible for notifying users of the generated device list and its usage evaluation.

[0520] As a concrete example, a robot can observe a cat toy for a week and generate a report indicating that it is an unused item. The user can then use an application to add the toy to a list and automatically list it on a flea market website.

[0521] An example of a prompt message is: "Based on data collected by a home pet care robot, think of ways to identify pet supplies that are no longer needed and notify the user of this information to help them manage them."

[0522] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0523] Step 1:

[0524] The terminal activates a home pet care robot, and the robot's camera captures the animal's behavior in real time. The input is visual data of the animal's behavior, and the output is a digitized video file of this data. The camera continuously records video and prepares to transmit the data to the next process.

[0525] Step 2:

[0526] The server receives digital video data transmitted from the terminal and uses the OpenCV library to identify specific actions or device usage within the video. The input is a digitized video file, and the output is analytical data indicating device usage. This process involves analyzing each frame to extract which devices the animal is using and to what extent.

[0527] Step 3:

[0528] The server uses a machine learning model implemented in Python based on the analyzed data to generate device usage frequency data. The input is analyzed data showing the device usage status, and the output is statistical information regarding usage frequency. As part of the data calculation, the number of times the device is used per hour is aggregated and the frequency is quantified.

[0529] Step 4:

[0530] The server uses statistical information to list infrequently used devices and generates a list of unnecessary devices. The input is usage statistics, and the output is a list of unnecessary devices. This list is created by identifying devices whose usage count over a month falls below a certain threshold.

[0531] Step 5:

[0532] The device receives a list of unnecessary devices sent from the server and notifies the user using a Flutter application. The input is a list of unnecessary devices, and the output is notification information for the user. This operation includes displaying the list and displaying a notification alert.

[0533] Step 6:

[0534] The user reviews the list through the application and decides whether to automatically list unwanted equipment on a flea market site or arrange for its collection. The input is the user's selection, and the output is the selected action. The user presses the list or collection arrangement button, and the result is fed back to the server.

[0535] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0536] This invention provides a system for streamlining the use and management of animal products, particularly enabling operation that takes user emotions into consideration. The system is implemented around an observation device, a server, a terminal, and an emotion engine.

[0537] First, the terminal monitors the animal's activity through an observation device installed in the home and acquires video data. This data is compressed and converted before being sent to a server via the network. The server analyzes the received data to identify the frequency of use of items used by the animal. Next, based on these results, the usefulness of the items is evaluated, and a list of items deemed unnecessary is generated.

[0538] In parallel, the server uses an emotion engine to recognize the emotions a user feels when accessing the system. This involves acquiring emotion data through interfaces such as cameras and microphones. The emotion engine analyzes the user's facial expressions and tone of voice to identify emotional states such as "pleasant" or "unpleasant."

[0539] When users review the list of items generated by the system, they receive an interactive experience tailored to their emotional state. For example, if the emotion engine determines that the user is feeling anxious about handling an item, the server adjusts the dialogue to provide suggestions and information that will make them feel more at ease. Specifically, this could include alternatives such as donating the item to a local community.

[0540] Furthermore, based on past emotional data, the system provides decision-making support tailored to the individual user's preferences. This reduces psychological resistance to disposing of unwanted items and promotes smoother decision-making.

[0541] In this way, the present invention provides a flexible interface that reflects the user's emotions and effectively supports the management of animal supplies.

[0542] The following describes the processing flow.

[0543] Step 1:

[0544] The device activates the in-home observation equipment and records the animal's daily activities as video in real time. The video data is temporarily stored in local storage.

[0545] Step 2:

[0546] The terminal compresses video data at predetermined time intervals and transmits it to the server via a secure connection. Security protocols are applied during data transmission to protect the data.

[0547] Step 3:

[0548] The server uses image processing algorithms to automatically identify items that animals have interacted with from the received video data. This allows the frequency of use of each item to be recorded.

[0549] Step 4:

[0550] The server evaluates the usefulness of items based on recorded usage frequency and lists items that do not meet certain evaluation criteria as unnecessary.

[0551] Step 5:

[0552] The server uses an emotion engine to recognize the user's emotions while they are interacting with the system. This recognition includes facial expression data and voice data captured by cameras and microphones.

[0553] Step 6:

[0554] The emotion engine determines the user's emotions through facial expression and voice analysis, detecting comfort and discomfort. Based on these results, the system dynamically adjusts its response.

[0555] Step 7:

[0556] Users review a list of unwanted items presented on the system and receive recommendations and alternatives tailored to their emotional state. For example, if the emotion engine detects the user's anxiety, the server suggests actions such as recommending donation destinations.

[0557] Step 8:

[0558] Based on the user's selected action, the server either lists the item on an online marketplace or sends the item information to a recycling company or donation recipient.

[0559] Step 9:

[0560] The server provides feedback to the user regarding the processed items and reports that the process is complete. The entire system, including consideration for emotions, creates an environment where users can comfortably declutter.

[0561] (Example 2)

[0562] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0563] In recent years, the number of households keeping pets has increased, leading to a greater complexity in managing pet-related items. In particular, users often experience emotional burdens when selecting and disposing of pet supplies. Furthermore, the lack of adequate guidance on proper disposal methods for unnecessary items has resulted in inefficient management.

[0564] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0565] In this invention, the server includes means for monitoring animal activity using observation devices installed in the home and acquiring data, means for analyzing the acquired data and identifying the frequency of use of items, and means for analyzing the user's emotions using an emotion engine and adjusting the interface. This enables the user to efficiently manage pet supplies and properly dispose of unwanted items while reducing emotional burden.

[0566] An "observation device" is a device installed in a home to monitor and record the behavior of animals, and may include cameras and sensors.

[0567] "Means of acquiring data" refers to the processes and technical means for collecting information in real time from observation devices.

[0568] "Means of data analysis" refers to algorithms and software used to process acquired information and analyze its content for use in specific purposes.

[0569] "Frequency of use of an item" is an indicator that shows how often an animal used a particular item during a specific period of time.

[0570] The "unnecessary items list" is a list of items that have been determined to be of low usefulness based on usage frequency analysis.

[0571] An "emotion engine" is a system or technology that recognizes and analyzes a user's emotions and adjusts the interface based on those emotions.

[0572] "Means of adjusting the interface" refers to a mechanism that changes the display method and dialogue format in order to provide optimal information in accordance with the user's emotional state.

[0573] A "means of supporting decision-making" refers to a system that provides information and suggestions to enable users to make choices and take actions effectively.

[0574] This invention is a system for streamlining the management of pet supplies in the home, aiming to enable operation that takes user emotions into consideration. The system consists of an observation device, a server, a terminal, and an emotion engine.

[0575] The device monitors animal activity using in-home observation equipment and acquires video data in real time. The observation equipment has built-in cameras and sensors to accurately record animal movements and behavior. This data is compressed and converted on the device and transmitted to a server via the internet. MPEG format is used as the data compression technology in this process.

[0576] The server analyzes the received data. Image processing software is used for the analysis, and image algorithms are employed to identify which items the animals used. Based on the identified frequency of use, the usefulness of the items is evaluated, and a list of items deemed unnecessary is generated.

[0577] In parallel, the server uses an emotion engine to acquire user emotion data. This emotion is recognized by analyzing data obtained through the camera and microphone. Facial recognition software and voice analysis tools analyze facial expressions and voice tone to identify emotions.

[0578] When users review the list of items generated by the system, they can receive interactive suggestions that correspond to their emotional state. For example, if a user expresses anxiety, the server may suggest donating to a local community to provide reassurance. In this way, users can be helped to dispose of unwanted items appropriately while reducing their psychological burden.

[0579] An example of a prompt for a generative AI model is: "Please identify my pet supplies that I don't use often or that others might need. Also, please provide gentle suggestions on how to dispose of them based on sentiment data."

[0580] This system efficiently supports the management of animal supplies by providing an approach that takes user emotions into consideration.

[0581] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0582] Step 1:

[0583] The terminal monitors animal activity in real time via an in-home observation device and acquires video data. In this step, a camera sensor records the animal's movements, and this video data is input to the terminal. The terminal compresses the received video data in a format such as MPEG and converts it into a format that can be sent to the server. Compressed video data is generated as output.

[0584] Step 2:

[0585] The server receives compressed video data sent from the terminal and begins analysis. Here, image processing algorithms are used to analyze animal behavior from the video and identify the items used. The input is compressed video data, and the output generates data on the identified items and their frequency of use.

[0586] Step 3:

[0587] The server executes a usefulness evaluation algorithm based on item usage frequency data. This data processing quantifies the usage frequency of each item and generates a list of items deemed unnecessary. The input is usage frequency data for identified items, and the output is a list of unnecessary items.

[0588] Step 4:

[0589] The server uses an emotion engine to analyze the user's facial expressions and tone of voice to determine their emotional state. Emotional data is collected through the camera and microphone and processed by analysis tools. Inputs include the user's facial expressions and voice data, and the output identifies the user's emotional state, such as "pleasant" or "unpleasant."

[0590] Step 5:

[0591] The user reviews a list of unwanted items generated by the system, and the server suggests appropriate disposal methods based on the user's emotional state. These suggestions may include options such as donating to the local community. User input consists of viewing the item list and selecting disposal methods, while the output provides personalized suggestions based on the user's emotional state.

[0592] Step 6:

[0593] The server references past emotional data to provide decision-making support tailored to the user's preferences. This reduces the user's psychological burden and facilitates the disposal of unwanted items. The input for this step is past emotional data, and the output is personalized decision-making support information.

[0594] (Application Example 2)

[0595] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0596] In modern households, pet supplies are diverse, and their management is crucial, but there is a lack of efficient management methods that take user feelings into consideration. As a result, unnecessary supplies accumulate in homes, occupying space and potentially increasing the environmental burden. Furthermore, there is a need for methods to dispose of unnecessary supplies without causing emotional distress to users. This invention aims to solve these problems.

[0597] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0598] In this invention, the server includes means for acquiring animal behavior in real time using a detection device placed in the home environment, means for analyzing the acquired visual information to identify the frequency of use of each item, and means for analyzing the user's emotional state. This makes it possible to manage pet supplies based on the user's emotional state.

[0599] A "detection device" is a device installed in a home to acquire information about an animal's behavior in real time.

[0600] "Visual information" refers to data that visually records the behavior and environment of animals.

[0601] "Item usage frequency" is an indicator that shows how often animal products are used.

[0602] An "e-commerce marketplace" is an online platform for exchanging product information and conducting transactions.

[0603] "User emotional state" refers to data that indicates the user's psychological or emotional condition.

[0604] "Sentiment analysis" is the process of analyzing a user's emotional state and generating suggestions that are tailored to that user.

[0605] To realize this invention, a detection device, a server, and a terminal for the user are required, all of which are installed in the home.

[0606] The server acquires real-time information on animal behavior through detection devices placed in the home environment. These detection devices include cameras and microphones, and monitor animal activity within the home, recording it as visual information. This information is transmitted to the server via the network, where it is analyzed. OpenCV is used as the image processing library for this analysis, and data processing is performed to identify the frequency of use of pet supplies.

[0607] The server evaluates the usefulness of items based on usage frequency data and generates a list of items deemed unnecessary. Furthermore, it analyzes the user's emotional state using an emotion engine. The emotional state is recognized from the user's facial expressions and voice, and the Google Cloud Sentiment Analysis API is used for emotion analysis. Prompts are generated and suggestions are presented that are tailored to the user's emotions.

[0608] The user's device serves to display this information to the user. The user uses a dedicated smartphone app to check their pet's activity status and a list of unwanted items, and disposes of unwanted items based on the suggested best options. For example, they can donate unwanted pet beds to a local animal shelter. This allows the user to manage their belongings efficiently while gaining emotional satisfaction.

[0609] Examples of prompt statements that can be provided to the generation AI model include the following:

[0610] "Please suggest ways to manage animal supplies based on the user's feelings. If there are unused supplies, please suggest donation or reuse in a gentle manner."

[0611] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0612] Step 1:

[0613] The device acquires real-time animal activity data from detection devices installed in the home. This data includes video and audio data. The acquired data is compressed and converted into a format suitable for later analysis. The output is appropriately formatted visual information.

[0614] Step 2:

[0615] The server receives visual information transmitted from the terminal and performs analysis using the image processing library OpenCV. The main purpose of the analysis is to identify the frequency of use of animal supplies. Specifically, it uses object recognition technology to identify supplies in the video and measures the frequency of use of each. The output is data showing the frequency of use of each supply.

[0616] Step 3:

[0617] Based on the analysis results, the server evaluates the usefulness of each item and creates a list of items deemed unnecessary. This evaluation uses defined criteria, primarily selecting items with low usage frequency. The output is a list of items deemed unnecessary.

[0618] Step 4:

[0619] The server retrieves facial expressions and voice data from the user's device to analyze the user's emotional state. This data is sent to Google Cloud's Sentiment Analysis API for emotional analysis. Specifically, the API determines emotions such as "pleasant" or "unpleasant" based on facial expressions and voice tone. The output is data indicating the user's emotional state.

[0620] Step 5:

[0621] The server generates suggestions for the user based on the sentiment analysis results. The generated prompts are then processed through a generative AI model to provide user-appropriate donation and reuse suggestions. The output is information containing suggestions for the user.

[0622] Step 6:

[0623] Users review suggestions provided by the server via their device screen and make decisions regarding the disposal of unwanted items. Specifically, they select recipients for donation or choose to reuse items. The goal is to efficiently process unwanted items. The output consists of processing instructions based on the user's selections.

[0624] The specific processing unit 290 transmits the result of the specific processing to the headset terminal 314. In the headset terminal 314, the control unit 46A causes the speaker 240 and display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0625] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0626] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and specific processing may also be performed by the headset terminal 314.

[0627] [Fourth Embodiment]

[0628] Figure 7 shows an example of the configuration of the data processing system 410 according to the fourth embodiment.

[0629] As shown in Figure 7, the data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[0630] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0631] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a controlled object 443. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and controlled object 443 are also connected to the bus 52.

[0632] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0633] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0634] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0635] The controlled object 443 includes a display device, LEDs in the eyes, and motors that drive the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the robot 414's emotions can be expressed by controlling these motors. Furthermore, the robot 414's facial expressions can also be expressed by controlling the illumination state of the LEDs in its eyes.

[0636] Figure 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Figure 8, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0637] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0638] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0639] In robot 414, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0640] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0641] This invention is a system that automatically observes and analyzes the usage of animal supplies and supports the organization and disposal of unnecessary items. This invention is mainly implemented using an observation device, a server, a terminal, and means for interacting with the user.

[0642] First, the device captures video data of the animal's daily activities through an observation device installed in the home. This data is compressed and processed, and then transmitted to a server via the network. The server receives the video data and uses a dedicated image processing algorithm and machine learning model to analyze the frequency of use of each item.

[0643] Next, the server evaluates the usefulness of animal products based on the analysis results. This takes into account statistical information, including the number of times and duration of use within a certain period. The usefulness evaluation generates a list of items deemed unnecessary. This list is stored in the database and provided to the user as a report as needed.

[0644] Through the generated report, users can view a list of items deemed unnecessary. For example, the list might include pet toys that haven't been used at all in the past month. In this case, the user can decide through the system whether to automatically list these items on a flea market site or have them taken by a recycling company or donation organization.

[0645] This system integrates with necessary network communication protocols to provide information on selected items to the online marketplace. Furthermore, the server automatically handles communication with recycling companies and donation recipients, ensuring that unwanted items are reused in a socially meaningful way. The goal is to reduce environmental impact while providing a comfortable living space for pets and their owners.

[0646] The following describes the processing flow.

[0647] Step 1:

[0648] The device activates the in-home observation equipment and records the animal's daily activities as video in real time. The acquired video data is temporarily stored in local storage.

[0649] Step 2:

[0650] The terminal compresses video data at predetermined time intervals and transmits it to the server via the internet. Security protocols are applied during data transmission to protect data confidentiality.

[0651] Step 3:

[0652] The server executes image processing algorithms to analyze the received video data. In this process, it identifies the objects that the animals have come into contact with and measures how often they are used.

[0653] Step 4:

[0654] The server evaluates the usefulness of each item based on usage frequency data. This evaluation is calculated, for example, based on the number of times it has been used in the past month.

[0655] Step 5:

[0656] The server generates a list of items deemed unnecessary based on the evaluation results. This list is prepared for notification to the user.

[0657] Step 6:

[0658] The user receives a notification and checks the item list through the system. The list displays the frequency of use and evaluation results for each item.

[0659] Step 7:

[0660] Users select an action regarding unwanted items. Options include automatically listing them on a flea market site or donating them to a recycling company or charity.

[0661] Step 8:

[0662] The server provides item information to the electronic marketplace or related businesses based on the user's selected action. The process is automated, including data transmission and scheduling of collection dates and times.

[0663] Step 9:

[0664] The server finally reports the processing results of the selected items to the user. This completes the entire system process.

[0665] (Example 1)

[0666] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0667] There is a need to efficiently identify and dispose of or reuse unnecessary items used in the home. In particular, it is difficult to accurately grasp the usage of everyday items through human observation, leading to the problem of accumulated waste. Furthermore, the disposal of unwanted items is time-consuming for users, often resulting in missed opportunities for social reuse.

[0668] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0669] In this invention, the server includes means for acquiring biological activity using an observation device, means for analyzing the acquired information and identifying the frequency of use of items, and means for evaluating the usefulness of items and generating a list of items deemed unnecessary. This makes it possible to accurately grasp usage patterns based on biological activity, enabling efficient identification and appropriate disposal of unnecessary items.

[0670] An "observation device" is a device installed in a home to collect information on the activity of living organisms, and its role is to acquire images and data.

[0671] "Acquired information" refers to data collected by observation devices, including content that specifically indicates the activity status of organisms.

[0672] "Items used" refers to various items that living organisms use on a daily basis, and their frequency of use and usefulness are the subjects of evaluation.

[0673] "Frequency of use" is an indicator obtained through the analysis of acquired information, showing how often each item was used within a specific period.

[0674] "Usefulness" is a criterion for evaluating how important a product is to the life of an organism, and is judged based on the frequency and duration of use.

[0675] An "item list" is a list generated based on an evaluation of the usefulness of the items used, and is used to identify items that have been deemed unnecessary.

[0676] An "information processing device" refers to a computing device that analyzes acquired information and processes various types of data through communication.

[0677] A "user" refers to an entity that utilizes the system, reviews the generated item list, and decides how to dispose of unwanted items.

[0678] This invention is a system for monitoring the activity of living organisms in the home, identifying unwanted items, and efficiently disposing of or reusing them. This system is primarily implemented using an observation device, a server, a terminal, and a user interface.

[0679] The device captures real-time images of the daily activities of living organisms through observation equipment installed in the home and acquires that information. Here, the camera device functions as an observation device, collecting the captured images as data. This data is compressed for efficient storage and transfer, and then transmitted to a server via the network.

[0680] The server analyzes the received information using a dedicated image processing algorithm and machine learning model. This analysis process utilizes open-source "machine learning libraries" used as machine learning frameworks, and specifically combines them with "image processing libraries" to identify organisms and objects being used in the video. This allows for the statistical calculation of the frequency and duration of use of each item.

[0681] Next, the server evaluates the usefulness of the items based on the analysis results and generates a list of items deemed unnecessary according to certain criteria. This list includes items that are not used or are used infrequently. This generated list is stored in a database and provided to the user.

[0682] Users can review unwanted items through the generated list and decide how to dispose of them. For example, a list might include pet toys that haven't been used for over a month. In this case, the user can use the system to decide whether to sell those items to a recycling shop, or to have them collected and disposed of or donated.

[0683] This system can connect selected item information with external markets and services via necessary communication protocols. Furthermore, the server automatically contacts recycling companies and donation recipients to ensure the socially beneficial use of unwanted items. This reduces the environmental impact while maintaining a comfortable living environment for pets and their owners.

[0684] Examples of prompts to input into a generative AI model:

[0685] "Please describe in detail the process of analyzing the frequency of use of pet supplies and optimizing the selection of unnecessary items."

[0686] This configuration allows the present invention to streamline the management of items within the home and promote the effective use of resources.

[0687] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0688] Step 1:

[0689] The device uses an observation system installed in the home to film the activity of living organisms and acquire video data. The input is the behavior and environment of the organisms, and the output is the filmed video clips. Specifically, the camera monitors 24 hours a day, creates clips at regular intervals, and records them.

[0690] Step 2:

[0691] If the video data acquired by the device is saved at its original quality, the file size becomes large. Therefore, to efficiently transfer the data over the network, the data is compressed using a lightweight codec. The input is raw video data, and the output is compressed video data. Specifically, video compression software is used to reduce the amount of data.

[0692] Step 3:

[0693] The terminal sends compressed video data to the server over the network. The input is the compressed video data, and the output is the server's response confirming the sent data. Specifically, the terminal uses a data transmission protocol to steadily upload the video to the server.

[0694] Step 4:

[0695] To analyze the video data received by the server, a dedicated image processing algorithm is used to identify organisms and objects within the video. The input is compressed video data, and the output is information about each identified item and its usage. Specifically, an image recognition library is used to identify items and obtain their location information.

[0696] Step 5:

[0697] The server applies a machine learning model to the identified information to evaluate the frequency and duration of use of items. The input is usage data for the identified items, and the output is statistical information regarding usage frequency and duration. Specifically, it uses a machine learning framework to analyze usage patterns and aggregate the results.

[0698] Step 6:

[0699] Based on the evaluation results, the server extracts items deemed to have low usefulness and generates a list of unnecessary items. The input is statistical information on used items, and the output is a list of unnecessary items. Specifically, it compares items with pre-set usefulness criteria and lists items that do not meet the criteria.

[0700] Step 7:

[0701] The server generates a list of unwanted items, stores it in a database, and provides it to the user. The input is the list of unwanted items, and the output is a report for the user. Specifically, the system periodically notifies the user of the list via an automated messaging system.

[0702] Step 8:

[0703] The user views the provided report and selects a disposal method for unwanted items. The input is the user's selection information, and the output is a record of the selected disposal method. Specifically, the user interface allows for options such as automatically listing unwanted items for sale or requesting collection.

[0704] Step 9:

[0705] The server processes items based on user instructions. Input is the user's disposal instructions, and output is the result of the executed disposal activities. Specifically, it sends data to flea market sites and contacts recycling companies to ensure the items are properly disposed of.

[0706] (Application Example 1)

[0707] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0708] The number of pet-related devices and items used in homes is increasing, making it difficult to manually manage their usage individually. Furthermore, delays in the disposal of unwanted devices and items can lead to wasted space and increased environmental burden. Therefore, efficient device management and the promotion of reuse are needed to realize a sustainable society.

[0709] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0710] In this invention, the server includes means for acquiring animal behavior in real time using sensing devices placed in the home area, means for analyzing the acquired image data and identifying the frequency of use of each device, and means for performing usage evaluations and generating a list of devices deemed unnecessary. This makes it possible to efficiently manage the use of devices and items in the home, quickly identify unnecessary items, and dispose of them in a reusable form.

[0711] The "household area" refers to the space within and around a house, where animals live on a daily basis.

[0712] A "sensing device" is a device that detects animal behavior in real time and acquires that information, and includes cameras and sensors.

[0713] "Behavior" refers to the activities and actions that animals perform within their home environment, including playing, eating, and resting.

[0714] "Acquired image data" refers to digital images and video information captured and collected by sensing devices.

[0715] "Frequency of use" is an indicator that shows the number of times or duration a particular device is used within a certain period of time.

[0716] "Usage evaluation" is the process of determining the usefulness and necessity of a device based on its usage frequency and other relevant data.

[0717] The "device list" is a list of identified devices, specifically those deemed unnecessary.

[0718] "Real-time transfer" refers to the process of immediately sending acquired data to a processing unit, meaning that information is exchanged almost simultaneously.

[0719] "Statistical information" refers to numerical summaries generated through data analysis, where information is organized according to certain rules.

[0720] The system that realizes this application consists of a home pet care robot that observes the animal's behavior through a sensing device. This robot is equipped with a camera and a communication module, which captures the animal's activity in real time and transmits the acquired data to a server in the cloud.

[0721] The server analyzes the received video data using the image processing library OpenCV to determine the usage frequency of each device. A machine learning model implemented in Python is used for data identification and classification.

[0722] Users can view analysis results from the server in real time using an application installed on their device. The application, designed using the mobile framework Flutter, is responsible for notifying users of the generated device list and its usage evaluation.

[0723] As a concrete example, a robot can observe a cat toy for a week and generate a report indicating that it is an unused item. The user can then use an application to add the toy to a list and automatically list it on a flea market website.

[0724] An example of a prompt message is: "Based on data collected by a home pet care robot, think of ways to identify pet supplies that are no longer needed and notify the user of this information to help them manage them."

[0725] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0726] Step 1:

[0727] The terminal activates a home pet care robot, and the robot's camera captures the animal's behavior in real time. The input is visual data of the animal's behavior, and the output is a digitized video file of this data. The camera continuously records video and prepares to transmit the data to the next process.

[0728] Step 2:

[0729] The server receives digital video data transmitted from the terminal and uses the OpenCV library to identify specific actions or device usage within the video. The input is a digitized video file, and the output is analytical data indicating device usage. This process involves analyzing each frame to extract which devices the animal is using and to what extent.

[0730] Step 3:

[0731] The server uses a machine learning model implemented in Python based on the analyzed data to generate device usage frequency data. The input is analyzed data showing the device usage status, and the output is statistical information regarding usage frequency. As part of the data calculation, the number of times the device is used per hour is aggregated and the frequency is quantified.

[0732] Step 4:

[0733] The server uses statistical information to list infrequently used devices and generates a list of unnecessary devices. The input is usage statistics, and the output is a list of unnecessary devices. This list is created by identifying devices whose usage count over a month falls below a certain threshold.

[0734] Step 5:

[0735] The device receives a list of unnecessary devices sent from the server and notifies the user using a Flutter application. The input is a list of unnecessary devices, and the output is notification information for the user. This operation includes displaying the list and displaying a notification alert.

[0736] Step 6:

[0737] The user reviews the list through the application and decides whether to automatically list unwanted equipment on a flea market site or arrange for its collection. The input is the user's selection, and the output is the selected action. The user presses the list or collection arrangement button, and the result is fed back to the server.

[0738] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0739] This invention provides a system for streamlining the use and management of animal products, particularly enabling operation that takes user emotions into consideration. The system is implemented around an observation device, a server, a terminal, and an emotion engine.

[0740] First, the terminal monitors the animal's activity through an observation device installed in the home and acquires video data. This data is compressed and converted before being sent to a server via the network. The server analyzes the received data to identify the frequency of use of items used by the animal. Next, based on these results, the usefulness of the items is evaluated, and a list of items deemed unnecessary is generated.

[0741] In parallel, the server uses an emotion engine to recognize the emotions a user feels when accessing the system. This involves acquiring emotion data through interfaces such as cameras and microphones. The emotion engine analyzes the user's facial expressions and tone of voice to identify emotional states such as "pleasant" or "unpleasant."

[0742] When users review the list of items generated by the system, they receive an interactive experience tailored to their emotional state. For example, if the emotion engine determines that the user is feeling anxious about handling an item, the server adjusts the dialogue to provide suggestions and information that will make them feel more at ease. Specifically, this could include alternatives such as donating the item to a local community.

[0743] Furthermore, based on past emotional data, the system provides decision-making support tailored to the individual user's preferences. This reduces psychological resistance to disposing of unwanted items and promotes smoother decision-making.

[0744] In this way, the present invention provides a flexible interface that reflects the user's emotions and effectively supports the management of animal supplies.

[0745] The following describes the processing flow.

[0746] Step 1:

[0747] The device activates the in-home observation equipment and records the animal's daily activities as video in real time. The video data is temporarily stored in local storage.

[0748] Step 2:

[0749] The terminal compresses video data at predetermined time intervals and transmits it to the server via a secure connection. Security protocols are applied during data transmission to protect the data.

[0750] Step 3:

[0751] The server uses image processing algorithms to automatically identify items that animals have interacted with from the received video data. This allows the frequency of use of each item to be recorded.

[0752] Step 4:

[0753] The server evaluates the usefulness of items based on recorded usage frequency and lists items that do not meet certain evaluation criteria as unnecessary.

[0754] Step 5:

[0755] The server uses an emotion engine to recognize the user's emotions while they are interacting with the system. This recognition includes facial expression data and voice data captured by cameras and microphones.

[0756] Step 6:

[0757] The emotion engine determines the user's emotions through facial expression and voice analysis, detecting comfort and discomfort. Based on these results, the system dynamically adjusts its response.

[0758] Step 7:

[0759] Users review a list of unwanted items presented on the system and receive recommendations and alternatives tailored to their emotional state. For example, if the emotion engine detects the user's anxiety, the server suggests actions such as recommending donation destinations.

[0760] Step 8:

[0761] Based on the user's selected action, the server either lists the item on an online marketplace or sends the item information to a recycling company or donation recipient.

[0762] Step 9:

[0763] The server provides feedback to the user regarding the processed items and reports that the process is complete. The entire system, including consideration for emotions, creates an environment where users can comfortably declutter.

[0764] (Example 2)

[0765] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0766] In recent years, the number of households keeping pets has increased, leading to a greater complexity in managing pet-related items. In particular, users often experience emotional burdens when selecting and disposing of pet supplies. Furthermore, the lack of adequate guidance on proper disposal methods for unnecessary items has resulted in inefficient management.

[0767] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0768] In this invention, the server includes means for monitoring animal activity using observation devices installed in the home and acquiring data, means for analyzing the acquired data and identifying the frequency of use of items, and means for analyzing the user's emotions using an emotion engine and adjusting the interface. This enables the user to efficiently manage pet supplies and properly dispose of unwanted items while reducing emotional burden.

[0769] An "observation device" is a device installed in a home to monitor and record the behavior of animals, and may include cameras and sensors.

[0770] "Means of acquiring data" refers to the processes and technical means for collecting information in real time from observation devices.

[0771] "Means of data analysis" refers to algorithms and software used to process acquired information and analyze its content for use in specific purposes.

[0772] "Frequency of use of an item" is an indicator that shows how often an animal used a particular item during a specific period of time.

[0773] The "unnecessary items list" is a list of items that have been determined to be of low usefulness based on usage frequency analysis.

[0774] An "emotion engine" is a system or technology that recognizes and analyzes a user's emotions and adjusts the interface based on those emotions.

[0775] "Means of adjusting the interface" refers to a mechanism that changes the display method and dialogue format in order to provide optimal information in accordance with the user's emotional state.

[0776] A "means of supporting decision-making" refers to a system that provides information and suggestions to enable users to make choices and take actions effectively.

[0777] This invention is a system for streamlining the management of pet supplies in the home, aiming to enable operation that takes user emotions into consideration. The system consists of an observation device, a server, a terminal, and an emotion engine.

[0778] The device monitors animal activity using in-home observation equipment and acquires video data in real time. The observation equipment has built-in cameras and sensors to accurately record animal movements and behavior. This data is compressed and converted on the device and transmitted to a server via the internet. MPEG format is used as the data compression technology in this process.

[0779] The server analyzes the received data. Image processing software is used for the analysis, and image algorithms are employed to identify which items the animals used. Based on the identified frequency of use, the usefulness of the items is evaluated, and a list of items deemed unnecessary is generated.

[0780] In parallel, the server uses an emotion engine to acquire user emotion data. This emotion is recognized by analyzing data obtained through the camera and microphone. Facial recognition software and voice analysis tools analyze facial expressions and voice tone to identify emotions.

[0781] When users review the list of items generated by the system, they can receive interactive suggestions that correspond to their emotional state. For example, if a user expresses anxiety, the server may suggest donating to a local community to provide reassurance. In this way, users can be helped to dispose of unwanted items appropriately while reducing their psychological burden.

[0782] An example of a prompt for a generative AI model is: "Please identify my pet supplies that I don't use often or that others might need. Also, please provide gentle suggestions on how to dispose of them based on sentiment data."

[0783] This system efficiently supports the management of animal supplies by providing an approach that takes user emotions into consideration.

[0784] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0785] Step 1:

[0786] The terminal monitors animal activity in real time via an in-home observation device and acquires video data. In this step, a camera sensor records the animal's movements, and this video data is input to the terminal. The terminal compresses the received video data in a format such as MPEG and converts it into a format that can be sent to the server. Compressed video data is generated as output.

[0787] Step 2:

[0788] The server receives compressed video data sent from the terminal and begins analysis. Here, image processing algorithms are used to analyze animal behavior from the video and identify the items used. The input is compressed video data, and the output generates data on the identified items and their frequency of use.

[0789] Step 3:

[0790] The server executes a usefulness evaluation algorithm based on item usage frequency data. This data processing quantifies the usage frequency of each item and generates a list of items deemed unnecessary. The input is usage frequency data for identified items, and the output is a list of unnecessary items.

[0791] Step 4:

[0792] The server uses an emotion engine to analyze the user's facial expressions and tone of voice to determine their emotional state. Emotional data is collected through the camera and microphone and processed by analysis tools. Inputs include the user's facial expressions and voice data, and the output identifies the user's emotional state, such as "pleasant" or "unpleasant."

[0793] Step 5:

[0794] The user reviews a list of unwanted items generated by the system, and the server suggests appropriate disposal methods based on the user's emotional state. These suggestions may include options such as donating to the local community. User input consists of viewing the item list and selecting disposal methods, while the output provides personalized suggestions based on the user's emotional state.

[0795] Step 6:

[0796] The server references past emotional data to provide decision-making support tailored to the user's preferences. This reduces the user's psychological burden and facilitates the disposal of unwanted items. The input for this step is past emotional data, and the output is personalized decision-making support information.

[0797] (Application Example 2)

[0798] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0799] In modern households, pet supplies are diverse, and their management is crucial, but there is a lack of efficient management methods that take user feelings into consideration. As a result, unnecessary supplies accumulate in homes, occupying space and potentially increasing the environmental burden. Furthermore, there is a need for methods to dispose of unnecessary supplies without causing emotional distress to users. This invention aims to solve these problems.

[0800] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0801] In this invention, the server includes means for acquiring animal behavior in real time using a detection device placed in the home environment, means for analyzing the acquired visual information to identify the frequency of use of each item, and means for analyzing the user's emotional state. This makes it possible to manage pet supplies based on the user's emotional state.

[0802] A "detection device" is a device installed in a home to acquire information about an animal's behavior in real time.

[0803] "Visual information" refers to data that visually records the behavior and environment of animals.

[0804] "Item usage frequency" is an indicator that shows how often animal products are used.

[0805] An "e-commerce marketplace" is an online platform for exchanging product information and conducting transactions.

[0806] "User emotional state" refers to data that indicates the user's psychological or emotional condition.

[0807] "Sentiment analysis" is the process of analyzing a user's emotional state and generating suggestions that are tailored to that user.

[0808] To realize this invention, a detection device, a server, and a terminal for the user are required, all of which are installed in the home.

[0809] The server acquires real-time information on animal behavior through detection devices placed in the home environment. These detection devices include cameras and microphones, and monitor animal activity within the home, recording it as visual information. This information is transmitted to the server via the network, where it is analyzed. OpenCV is used as the image processing library for this analysis, and data processing is performed to identify the frequency of use of pet supplies.

[0810] The server evaluates the usefulness of items based on usage frequency data and generates a list of items deemed unnecessary. Furthermore, it analyzes the user's emotional state using an emotion engine. The emotional state is recognized from the user's facial expressions and voice, and the Google Cloud Sentiment Analysis API is used for emotion analysis. Prompts are generated and suggestions are presented that are tailored to the user's emotions.

[0811] The user's device serves to display this information to the user. The user uses a dedicated smartphone app to check their pet's activity status and a list of unwanted items, and disposes of unwanted items based on the suggested best options. For example, they can donate unwanted pet beds to a local animal shelter. This allows the user to manage their belongings efficiently while gaining emotional satisfaction.

[0812] Examples of prompt statements that can be provided to the generation AI model include the following:

[0813] "Please suggest ways to manage animal supplies based on the user's feelings. If there are unused supplies, please suggest donation or reuse in a gentle manner."

[0814] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0815] Step 1:

[0816] The device acquires real-time animal activity data from detection devices installed in the home. This data includes video and audio data. The acquired data is compressed and converted into a format suitable for later analysis. The output is appropriately formatted visual information.

[0817] Step 2:

[0818] The server receives visual information transmitted from the terminal and performs analysis using the image processing library OpenCV. The main purpose of the analysis is to identify the frequency of use of animal supplies. Specifically, it uses object recognition technology to identify supplies in the video and measures the frequency of use of each. The output is data showing the frequency of use of each supply.

[0819] Step 3:

[0820] Based on the analysis results, the server evaluates the usefulness of each item and creates a list of items deemed unnecessary. This evaluation uses defined criteria, primarily selecting items with low usage frequency. The output is a list of items deemed unnecessary.

[0821] Step 4:

[0822] The server retrieves facial expressions and voice data from the user's device to analyze the user's emotional state. This data is sent to Google Cloud's Sentiment Analysis API for emotional analysis. Specifically, the API determines emotions such as "pleasant" or "unpleasant" based on facial expressions and voice tone. The output is data indicating the user's emotional state.

[0823] Step 5:

[0824] The server generates suggestions for the user based on the sentiment analysis results. The generated prompts are then processed through a generative AI model to provide user-appropriate donation and reuse suggestions. The output is information containing suggestions for the user.

[0825] Step 6:

[0826] Users review suggestions provided by the server via their device screen and make decisions regarding the disposal of unwanted items. Specifically, they select recipients for donation or choose to reuse items. The goal is to efficiently process unwanted items. The output consists of processing instructions based on the user's selections.

[0827] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the controlled object 443 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0828] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0829] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the robot 414.

[0830] Furthermore, the emotion identification model 59, acting as an emotion engine, may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to a specific mapping, which is an emotion map (see Figure 9). Similarly, the emotion identification model 59 may also determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[0831] Figure 9 shows an emotion map 400 in which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. The closer to the center of the concentric circles, the more primitive the emotions are located. Further out of the concentric circles, emotions representing states and actions arising from mental states are located. Emotion is a concept that includes feelings and mental states. On the left side of the concentric circles, emotions that are generally generated from reactions occurring in the brain are located. On the right side of the concentric circles, emotions that are generally induced by situational judgment are located. Above and below the concentric circles, emotions that are generally generated from reactions occurring in the brain and induced by situational judgment are located. In addition, the emotion of "pleasure" is located on the upper side of the concentric circles, and the emotion of "displeasure" is located on the lower side. Thus, in the emotion map 400, multiple emotions are mapped based on the structure in which emotions arise, and emotions that are likely to occur simultaneously are mapped close together.

[0832] These emotions are distributed at the 3 o'clock position on the Emotion Map 400, and usually fluctuate between feelings of security and anxiety. In the right half of the Emotion Map 400, situational awareness takes precedence over internal feelings, resulting in a calm impression.

[0833] The inside of the Emotion Map 400 represents inner thoughts, while the outside represents actions. Therefore, the further you go from the outside of the Emotion Map 400, the more visible (expressed in actions) your emotions become.

[0834] Here, human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. Similarly, in robots, cars, motorcycles, etc., emotions can be created based on various balances, such as posture and battery level. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. The emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on a system for analyzing brain physiological signals of speech emotion recognition and emotion, Tokushima University, doctoral dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map contains emotions belonging to a region called "response," where sensation is dominant. The right half of the emotion map contains emotions belonging to a region called "situation," where situational awareness is dominant.

[0835] The emotion map defines two emotions that promote learning. One is the emotion around the middle of the negative "repentance" and "reflection" on the situation side. In other words, it is when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is the emotion around the positive "desire" on the reaction side. In other words, it is when the robot has positive feelings such as "I want more" or "I want to know more."

[0836] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​representing each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple training data sets, which are combinations of user input and emotion values ​​representing each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions located close together have similar values, as shown in the emotion map 900 in Figure 10. Figure 10 shows an example where multiple emotions such as "reassured," "calm," and "confident" have similar emotion values.

[0837] The above description primarily focuses on the functions of the data processing device 12 in relation to this disclosure. However, the system related to this disclosure is not necessarily implemented on a server. The system related to this disclosure may be implemented as a general information processing system. This disclosure may be implemented, for example, as a software program that runs on a personal computer or as an application that runs on a smartphone. The method related to this disclosure may be provided to users in SaaS (Software as a Service) format.

[0838] In the above embodiment, an example was given in which a specific process is performed by a single computer 22. However, the technology of this disclosure is not limited thereto, and a distributed processing of the specific process may be performed by multiple computers, including computer 22. For example, a data generation model 58 may be provided in an external device of the data processing device 12, and the external device may generate data according to the input data.

[0839] In the above embodiment, an example was given in which the specific processing program 56 is stored in the storage 32, but the technology of this disclosure is not limited thereto. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-temporary storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-temporary storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes specific processing according to the specific processing program 56.

[0840] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[0841] Furthermore, it is not necessary to store the entirety of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store the entirety of the specific processing program 56 in the storage 32; it is acceptable to store only a portion of the specific processing program 56.

[0842] The following types of processors can be used as hardware resources to perform specific processing. Examples of processors include a CPU, a general-purpose processor that functions as a hardware resource to perform specific processing by executing software, i.e., a program. Other examples of processors include dedicated electrical circuits, such as FPGAs (Field-Programmable Gate Arrays), PLDs (Programmable Logic Devices), or ASICs (Application Specific Integrated Circuits), which have circuit configurations specifically designed to perform specific processing. All of these processors have built-in or connected memory, and all of them perform specific processing by using memory.

[0843] The hardware resource that performs a specific process may consist of one of these various processors, or it may consist of a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Alternatively, the hardware resource that performs a specific process may consist of a single processor.

[0844] Examples of configurations using a single processor include, firstly, a configuration in which one or more CPUs and software are combined to form a single processor, and this processor functions as a hardware resource that performs a specific process. Secondly, there is a configuration using a processor that realizes the functions of the entire system, including multiple hardware resources that perform a specific process, on a single IC chip, as exemplified by SoCs (System-on-a-chip). In this way, a specific process is realized using one or more of the above types of processors as hardware resources.

[0845] Furthermore, the hardware structure of these various processors can more specifically utilize electrical circuits that combine circuit elements such as semiconductor devices. Also, the specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps can be deleted, new steps added, or the processing order rearranged, as long as it does not deviate from the main purpose.

[0846] The descriptions and illustrations presented above are detailed explanations of the technical aspects of this disclosure and are merely examples of the technical aspects. For example, the above descriptions of the structure, function, operation, and effect are examples of the structure, function, operation, and effect of the technical aspects of this disclosure. Therefore, it goes without saying that you may delete unnecessary parts, add new elements, or replace elements in the descriptions and illustrations presented above, as long as you do not deviate from the essence of the technical aspects of this disclosure. Furthermore, in order to avoid confusion and facilitate understanding of the technical aspects of this disclosure, explanations of common technical knowledge and the like that do not require special explanation to enable the implementation of the technical aspects of this disclosure have been omitted from the descriptions and illustrations presented above.

[0847] All documents, patent applications, and technical standards described herein are incorporated by reference to the same extent as if each individual document, patent application, and technical standard were specifically and individually noted as being incorporated by reference.

[0848] The following is further disclosed regarding the embodiments described above.

[0849] (Claim 1)

[0850] A means of acquiring the daily activities of animals in real time using observation equipment installed in the home,

[0851] A means for analyzing acquired video data and identifying the frequency of use of each item,

[0852] A means for evaluating the usefulness of items and generating a list of items deemed unnecessary,

[0853] A means of automatically providing item information by communicating with an electronic marketplace based on the generated item list,

[0854] A means of listing goods in a designated market based on the provided goods information,

[0855] A means of selecting unwanted items and contacting collection companies or donation recipients to arrange for their collection or donation,

[0856] A system that includes this.

[0857] (Claim 2)

[0858] The system according to claim 1, further comprising means for adjusting the listing or processing procedure of an item based on user feedback obtained.

[0859] (Claim 3)

[0860] The system according to claim 1, further comprising means for explaining the usefulness of items to the user and guiding them into dialogue based on the generated list of items.

[0861] "Example 1"

[0862] (Claim 1)

[0863] A means of acquiring the activity of living organisms using an observation device,

[0864] A means of analyzing the acquired information to determine the frequency of use of the items,

[0865] A means for evaluating the usefulness of used items and generating a list of items deemed unnecessary,

[0866] A means for automatically providing item information by communicating with an information processing device based on the generated item list,

[0867] A means of listing items in a designated market based on the provided item information,

[0868] A means of selecting unwanted items and contacting collection companies or donation recipients to arrange for collection or donation,

[0869] A means for calculating the frequency and duration of use of an item based on the acquired information,

[0870] A means to allow users to choose how to dispose of unwanted items,

[0871] A system that includes this.

[0872] (Claim 2)

[0873] The system according to claim 1, further comprising means for adjusting the listing or processing procedures for items based on user feedback obtained.

[0874] (Claim 3)

[0875] The system according to claim 1, further comprising means for explaining the usefulness of items to the user and guiding the conversation based on the generated item list.

[0876] "Application Example 1"

[0877] (Claim 1)

[0878] A means of acquiring animal behavior in real time using sensing devices placed in the home area,

[0879] A means for analyzing acquired image data and determining the frequency of use of each device,

[0880] A means for conducting usage evaluations and generating a list of devices deemed unnecessary,

[0881] A means of automatically providing device information by communicating with the information market based on the generated list of devices,

[0882] Means for transmitting the device to a designated market based on the provided device information,

[0883] A means of selecting unnecessary equipment, contacting recycling companies or suppliers to arrange for collection or provision,

[0884] A means by which a mobile device for home use automatically patrols and monitors the usage status of the device,

[0885] A means for transferring information provided by a sensing device to a terminal device in real time and generating statistical information,

[0886] A system that includes this.

[0887] (Claim 2)

[0888] The system according to claim 1, further comprising means for adjusting the transmission or processing method of the device based on user feedback obtained.

[0889] (Claim 3)

[0890] The system according to claim 1, further comprising means for explaining the evaluation of device usage to the user and guiding the conversation based on the generated list of devices.

[0891] "Example 2 of combining an emotion engine"

[0892] (Claim 1)

[0893] A means of monitoring animal activity and acquiring data using observation devices installed in homes,

[0894] A means of analyzing acquired data to determine the frequency of use of items,

[0895] A means for evaluating the usefulness of items based on their specified frequency of use and generating a list of unnecessary items,

[0896] A means of analyzing the user's emotions using an emotion engine based on the generated item list and adjusting the interface accordingly,

[0897] A means of supporting user-optimized decision-making by referring to past emotional data,

[0898] A system that includes this.

[0899] (Claim 2)

[0900] The system according to claim 1, further comprising means for adjusting the listing or processing procedure of an item based on acquired user sentiment data.

[0901] (Claim 3)

[0902] The system according to claim 1, further comprising means for providing the user with emotionally appropriate suggestions and guiding the conversation based on the generated list of items.

[0903] "Application example 2 when combining with an emotional engine"

[0904] (Claim 1)

[0905] A means of acquiring animal behavior in real time using detection devices placed in the home environment,

[0906] A means for analyzing acquired visual information and identifying the frequency of use of each item,

[0907] A means for evaluating the usefulness of items and creating a list of items deemed unnecessary,

[0908] A means for automatically transmitting item information by communicating with an e-commerce marketplace based on the generated item list,

[0909] A means of listing items in designated markets based on transmitted item information,

[0910] A means of selecting unwanted items, contacting collection companies or donation recipients to arrange for collection or donation,

[0911] A means of analyzing the emotional state of users,

[0912] A means for generating and presenting user-appropriate suggestions based on the results of sentiment analysis,

[0913] A system that includes this.

[0914] (Claim 2)

[0915] The system according to claim 1, further comprising means for adjusting item processing suggestions or procedures based on the emotional state of the user obtained.

[0916] (Claim 3)

[0917] The system according to claim 1, further comprising means for explaining the usefulness of items to the user and guiding the conversation based on the generated item list. [Explanation of Symbols]

[0918] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Devices 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robots< / url:> < / url:> < / url:> < / url:>

Claims

1. A means of acquiring animal behavior in real time using sensing devices placed in the home area, A means for analyzing acquired image data and determining the frequency of use of each device, A means for conducting usage evaluations and generating a list of devices deemed unnecessary, A means of automatically providing device information by communicating with the information market based on the generated list of devices, Means for transmitting the device to a designated market based on the provided device information, A means of selecting unnecessary equipment, contacting recycling companies or suppliers to arrange for collection or provision, A means by which a mobile device for home use automatically patrols and monitors the usage status of the device, A means for transferring information provided by a sensing device to a terminal device in real time and generating statistical information, A system that includes this.

2. The system according to claim 1, further comprising means for adjusting the transmission or processing method of the device based on the user's feedback obtained.

3. The system according to claim 1, further comprising means for explaining the evaluation of the use of the devices to the user and guiding the conversation based on the generated list of devices.

Images