System

Abstract

A system is provided.SOLUTION: A system, comprising: means for setting a home position for an article to be used by a child; means for detecting a position of the article with a camera; means for notifying if the article is out of the home position; and means for detecting that the article is put back in the home position and generating and notifying a reward message.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] The present invention addresses the problem of children being unable to put things away in the correct place after using them. Its purpose is to reduce the housework burden on parents and improve the efficiency of housework throughout the household, particularly in dual-income and nuclear families. For those raising children, the daily chore of cleaning up messes made by children is often a time-consuming and labor-intensive task. This can make it difficult for parents to work or increase stress within the home. [Means for solving the problem]

[0005] The present invention is a system that includes a means for setting fixed locations for items used by children, a means for detecting the locations of the items using a camera, a means for notifying the user when an item has moved away from its fixed location, and a means for detecting when the item has been returned to its fixed location and generating a reward message to notify the user. Specifically, a camera that monitors the use of the items periodically photographs the state of the room and sends the image data to a server, which then analyzes the received image data and confirms whether the item is in its set fixed location. This system helps children develop the habit of returning items to their correct locations by themselves, reducing the housework burden on parents and improving the efficiency of housework throughout the household.

[0006] "Items" refers to concrete items such as toys and household items used by children.

[0007] "Home position" refers to a predetermined, designated location where an item should be placed when not in use.

[0008] "Means" refers to a method or device used to achieve a particular end.

[0009] "Camera" refers to a photographic device used to visually record the location and condition of an item.

[0010] "Means for establishing a fixed position" refers to a method or device for establishing a predetermined position for an item within a system.

[0011] "Means for detecting location" refers to a method or device for determining the current location of an item.

[0012] "Means for notifying" refers to a method or device for notifying a user when an item has moved out of its designated position.

[0013] A "reward message" refers to a message of encouragement or praise that is sent to the user when the item is returned to the correct location.

[0014] "Means for photographing the state of a room" refers to a method or device for visually recording the overall state or partial state of a room.

[0015] "Transmitting means" refers to a method or device for sending captured image data to an external device such as a server.

[0016] "Image data" refers to data containing visual information recorded by a photographic device such as a camera.

[0017] "Server" refers to a central processing unit for receiving, analyzing, and storing image data and other information.

[0018] "Means for analyzing" refers to a method or device for making calculations or decisions based on received information. [Brief explanation of the drawings]

[0019] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9]1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0020] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0021] First, the terms used in the following description will be explained.

[0022] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0023] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0024] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0025] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0026] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0027] [First embodiment]

[0028] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0029] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0030] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0031] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0032] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0033] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0034] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0035] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0036] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0037] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0038] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0039] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0040] The present invention provides a system for helping children put away items in their correct places after use, thereby reducing the burden of housework on parents. The system includes means for setting fixed positions for items, detecting the positions of the items with a camera, notifying the parent if the item has moved away from its fixed position, detecting that the item has been returned to its fixed position, and generating a reward message to notify the parent.

[0041] Specifically, the user registers items through a terminal. The user registers children's toys and household items in the system and sets fixed locations for the items. For example, the user sets the "blue block set" to the "top left of the toy box." The terminal uses a camera to take a photo of the blue block set and its fixed location, and sends the image data to the server. The server analyzes this image data and stores information about the items and their fixed locations in a database.

[0042] If a child uses an item but does not return it to its registered location, the system detects this and notifies the user. The device's camera periodically takes pictures of the room and sends the image data to the server. The server analyzes the received image data and, if an item is not in its designated location, sends an alert to the device. The device notifies the user by voice or on the screen, saying, "Please return the blue block set to the top left of the toy box."

[0043] When the user returns the item to the correct location, the device takes a photo with the camera again and sends the image data to the server. The server analyzes the image data and, if it confirms that the item has been returned to its designated location, generates a reward message. For example, the server may generate a message such as "Good job! That's great!" and send it to the device. The device then notifies the user of the message by voice or by displaying it on its screen.

[0044] Specific examples are as follows:

[0045] Suppose a user registers a "red toy car" as "the center of the shelf." The user opens the app on their device and takes a photo of the item and its designated location to register it. If a child plays with the red toy car and leaves it on the floor after using it, the system will determine that it is not in its designated location. The device's camera takes a photo of the room, which the server analyzes. If it detects that the item has not been returned to its designated location, a notification will appear on the device saying, "Please return the red toy car to the center of the shelf." If the child complies and returns the item to the correct location, the device's camera will take another photo, and the server will confirm that it has been returned to the correct location. This will cause the server to generate a reward message, and the device will issue an audio notification saying, "Good job!"

[0046] The system of the present invention can improve the efficiency of household chores and reduce the burden on parents.

[0047] The processing flow will be explained below.

[0048] Step 1:

[0049] The user opens the app on their device to register their child's items in the system. The user selects the "Register Item" menu and enters the item name and its designated location. For example, the "blue block set" is set to the "top left of the toy box."

[0050] Step 2:

[0051] The device's camera is activated, and the user places the blue block set in front of the camera. The user presses the shutter button, and the device captures the image. The captured image data is temporarily stored in the device.

[0052] Step 3:

[0053] The device transmits the captured image data and fixed position information to the server, which receives the data and analyzes the image data and fixed position information.

[0054] Step 4:

[0055] The server stores the image data and location information in a database, and this storage operation records the item and its location in the system.

[0056] Step 5:

[0057] A child uses an object, for example, takes a blue block set out of a toy box and starts playing with it.

[0058] Step 6:

[0059] The device uses a camera to take pictures of the room at a pre-set frequency, for example, every hour, and sends the image data to a server.

[0060] Step 7:

[0061] The server analyzes the received image data and checks the current location of the item. If the blue block set is not in the "top left of the toy box," the server generates an alert.

[0062] Step 8:

[0063] The server generates an alert and sends it to the device, which then notifies the child of the alert through audio or a display on the screen. For example, it might say, "Put the blue block set back on the top left shelf of the toy box."

[0064] Step 9:

[0065] The user instructs the child to return the items to their correct locations, for example, returning the blue block set to the top left of the toy box.

[0066] Step 10:

[0067] The device then starts the camera again and takes a picture of the room. The captured image data is then sent to the server again.

[0068] Step 11:

[0069] The server analyzes the received image data and confirms that the blue block set has been returned to its original position. Based on the confirmation result, the server generates a reward message.

[0070] Step 12:

[0071] The server sends a reward message to the device, which then notifies the child of the message by voice or on-screen display. For example, "Great job! Amazing!" is displayed.

[0072] This system helps children develop the habit of returning items to their correct places, reducing the household chore burden on parents.

[0073] Example 1

[0074] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0075] Children often do not put away their belongings or household items in the correct place after playing, which creates a problem for parents, who have to spend a lot of time and effort tidying up for them. Furthermore, children have fewer opportunities to develop the habit of tidying up on their own. This increases the household chore burden and creates the problem of increased stress for parents.

[0076] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0077] In this invention, the server includes: means for a user to set a fixed position for an item; means for taking a picture of the item and its fixed position with a camera on the terminal and sending the image data to the server; means for the server to analyze the image data and store information about the item and its fixed position in a database; means for sending an alert notification to the terminal if the item has deviated from its fixed position; means for taking another picture with the camera when the user returns the item to the correct position and sending the image data to the server; and means for the server to analyze the image data, confirm that the item has been returned to its fixed position, and generate a reward message to notify the user. This automatically monitors the tidying up of items, reducing the housework burden on parents and enabling children to develop the habit of tidying up.

[0078] A "user" is a person, including an adult in the household, who operates the system and registers items and sets their locations.

[0079] "Items" refer to items to be organized, such as toys and household items used by children.

[0080] A "home location" is a specific place or area where an item is to be stored.

[0081] A "terminal" is a device that includes a computer device operated by a user and a camera mounted on that device.

[0082] A "camera" is a device that is installed in a terminal and takes pictures of items, their fixed positions, and the state of the room to generate image data.

[0083] The "server" is a computer system that analyzes image data sent from the terminal, stores the information in a database, and generates notifications and reward messages.

[0084] "Image data" refers to visual information of an object or the state of a room captured by a camera.

[0085] A "database" is an information management system located within a server that stores and manages information on analyzed items and their fixed locations.

[0086] An "alert notification" is a message sent to the user via the terminal to warn them when an item is not in its designated location.

[0087] A "reward message" is a message of praise or encouragement that is generated by the server and sent to the user via the terminal when the user returns the item to the correct location.

[0088] "Analysis" refers to the operation in which the server processes the received image data, identifies the location of the item, and makes appropriate decisions.

[0089] This invention is a system that helps children put things away in the correct place after use, reducing the household chore burden on parents. The system is mainly composed of the user, terminal, and server elements, and functions as follows.

[0090] First, the user uses the device's app to register an item. When registering an item, the user inputs the name of the item (e.g., a toy or household item) and its designated location. Next, the device's camera is used to take a photo of the item and its designated location, and the image data is sent to the server. For example, if the user wants to set the "blue block set" to the "top left of the toy box," the user takes a photo of that location with the device's camera.

[0091] The server receives image data sent from the terminal and analyzes the items and their fixed locations. The analysis results are saved in a database, and the item location information is managed as digital data. The server's analysis makes it possible to determine whether the items have been placed in the correct locations.

[0092] The device is equipped with a timer function that periodically takes photos of the room's condition, and the camera takes photos at specified times. The image data is then sent back to the server, which analyzes the received images. If an item is not in its registered location, the server generates an alert message and notifies the user via the device. For example, a specific message such as "Please return the blue block set to the top left of the toy box" is notified by voice or displayed on the screen.

[0093] When the user puts the items away in the correct place, the device's camera takes another photo of the situation and sends the image data to the server. The server analyzes this data and confirms that the items have been returned to their proper places. If successful, the server generates a reward message and notifies the user via the device. For example, a praising message such as "Good job! Amazing!" can be conveyed by voice or displayed on the screen, increasing the child's motivation to tidy up on their own.

[0094] As a concrete example, consider the case where a user registers a "red toy car" as "the center of the shelf." The user enters the name and designated location of the item into the device's app and takes a photo of the location with the device's camera. The device then sends the image data to the server, which analyzes it and stores it in a database. The device then periodically takes photos of the room, and the server analyzes the images to check whether the item is in its designated location. If the item is not in its designated location, an alert notification is sent to the user, prompting them to return the item to its correct location.

[0095] This system promotes tidiness and efficiency in the home, reduces the burden of housework on parents, and helps children develop the habit of tidying up on their own, which is expected to improve the overall home environment.

[0096] Examples of prompts to input to a generative AI model include:

[0097] "I've registered the red toy car in the center of the shelf, but what's the notification mechanism for when a child forgets to put it back?"

[0098] "Tell me the process that generates the reward message when the child returns the object to its correct location."

[0099] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0100] System program processing flow

[0101] Step 1: Registering and positioning items

[0102] Specific behavior:

[0103] 1. The user launches the app on their device and opens the item registration screen.

[0104] 2. The user selects the "Register Item" option and enters the name of the item (e.g., "Blue Block Set").

[0105] 3. The user inputs the item's fixed location (e.g., "top left of the toy box").

[0106] 4. The user takes a photo of the item and its designated location using the device's camera.

[0107] Input and Output:

[0108] Input: Name and location of the object, camera image.

[0109] Data processing: Associate item names and location information with camera images.

[0110] Output: Send image data to the server.

[0111] Step 2: Receiving and saving image data

[0112] 1. The server receives the image data sent from the terminal.

[0113] 2. The server analyzes the received image data and extracts information about the item and its location.

[0114] 3. The server stores the item and its location information in a database.

[0115] Input and Output:

[0116] Input: Image data sent from the device.

[0117] Data processing: Extraction of information about items and their locations through image analysis.

[0118] Output: Saves the information to the database and sends a message to the terminal confirming the success.

[0119] Step 3: Photograph the room

[0120] 1. The device takes photos of the room at specified intervals.

[0121] Input and Output:

[0122] Enter: Timer settings.

[0123] Data processing: Regular photography.

[0124] Output: Captured image data.

[0125] Step 4: Sending image data

[0126] 1. The device sends the captured image data to the server.

[0127] Input and Output:

[0128] Input: Captured image data.

[0129] Data processing: Preparing image data for transmission.

[0130] Output: Send image data to the server.

[0131] Step 5: Locate your items

[0132] 1. The server analyzes the received image data and checks whether the item is in the registered location.

[0133] 2. If the server determines that the item is not in place, it generates an alert message.

[0134] 3. The server sends an alert message to the device.

[0135] Input and Output:

[0136] Input: Image data sent to the server.

[0137] Data processing: Confirmation of item location through image analysis.

[0138] Output: Generates and sends an alert message.

[0139] Step 6: Alert Notifications

[0140] 1. The terminal notifies the user of the received alert message by voice or on-screen display.

[0141] Specific behavior:

[0142] For example, a message might appear on the device screen saying, "Please return the blue block set to the top left of the toy box."

[0143] Input and Output:

[0144] Input: The alert message sent by the server.

[0145] Data processing: Preparing messages for audio or screen display.

[0146] Output: Alert notification to the user.

[0147] Step 7: Check that the item is in place

[0148] 1. The user returns the item to its correct location.

[0149] 2. The device will take another photo of the situation with its camera.

[0150] 3. The device sends the captured image data to the server.

[0151] Input and Output:

[0152] Input: User actions, captured image data.

[0153] Data processing: Preparing image data for transmission.

[0154] Output: Send image data to the server.

[0155] Step 8: Reward Message Generation and Notification

[0156] 1. The server analyzes the received image data again and confirms that the item has been returned to its designated location.

[0157] 2. The server generates a reward message and sends it to the terminal.

[0158] 3. The terminal notifies the user of the reward message by voice or on-screen display.

[0159] Specific behavior:

[0160] For example, a message like "Great job! Awesome!" will appear on the device screen.

[0161] Input and Output:

[0162] Input: Re-captured image data.

[0163] Data processing: Image analysis and verification, reward message generation.

[0164] Output: Send reward message to device and notify user.

[0165] (Application example 1)

[0166] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0167] Currently, many homes and factories face the problem of items not being returned to their correct locations after use. This increases the household chore burden on parents at home and reduces production efficiency in factories. In particular, in industrial sites, improper management of tools and parts can increase the risk of production line shutdowns and accidents. To solve these problems, a system is needed that automatically manages and checks the location of items and notifies workers as necessary.

[0168] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0169] In this invention, the server includes a means for setting a fixed position for an item used by a user, a means for detecting the position of the item with a camera, a means for notifying when the item has deviated from the fixed position, and a means for detecting when the item has been returned to the fixed position and generating a reward message to notify the user. This automates the management of item positions in homes and factories, reducing the burden of housework on parents, improving factory production efficiency, and ensuring safety.

[0170] "User" refers to a person who uses an item in a home, factory, etc.

[0171] "Article" refers to a concrete object used for a specific purpose, such as a child's toy or a tool or part used in a factory.

[0172] "Home location" refers to a predetermined location to which an item should be returned before or after use.

[0173] The "means for setting" refers to a device or software that allows a user to register the fixed location of an item and save that information.

[0174] "Detection means" refers to technology for recognizing the location of an item using a device such as a camera and acquiring that information.

[0175] "Means for notifying" means an audio, visual, or other method for notifying a user if an item is displaced from its designated location.

[0176] "Means for generating and notifying reward messages" refers to devices or software that confirm that an item has been returned to its proper location and send a message of praise or encouragement to the user.

[0177] "Camera" refers to a photographing device for photographing the position of an item and acquiring image data thereof.

[0178] "Image data" refers to digital data containing image information captured by a camera.

[0179] "Server" refers to a computer system that receives and analyzes image data and generates necessary notifications and messages.

[0180] "Software" refers to programs or applications that run on a computer or other digital device.

[0181] "OpenCV" is a library for image analysis and is used to detect the position of objects.

[0182] "Notification Library" means a program library for generating and sending reward messages and alerts.

[0183] This invention is a system that supports returning used items to their correct positions using a terminal in homes and factories. The system includes means for setting a fixed position for the item, detecting it with a camera, notifying the user if the item has deviated from its fixed position, and detecting that the item has been returned to the correct position and generating and notifying the user of a reward message.

[0184] First, the user uses a terminal to register items and set their fixed locations. The user registers the items in the system and sets the fixed location for each item. This information is sent to the server and saved in the database. For example, in a factory, a "maintenance screwdriver set" is set to the "top right of the tool shelf."

[0185] Next, the camera periodically captures images of the facility and sends the image data to a server. The server analyzes the image data to check whether the items are in their designated locations. This is done using an image analysis library such as OpenCV. If, based on the analysis results, the items have not been returned to their designated locations, the server sends a notification to the terminal. For example, an alert will be displayed saying, "Please return the maintenance screwdriver set to the top right shelf of the tool shelf."

[0186] When the user returns the item to the correct position, the camera takes a picture again and sends the image data to the server. The server analyzes the image again and confirms that the item is in its correct position. Once this is confirmed, the server generates a reward message and sends a message such as "Good job!" to the device. A specific notification library is used to send the reward message.

[0187] As a specific example, if a factory worker leaves a "maintenance screwdriver set" on the floor after use, the camera captures the state, the server analyzes it and issues a notification, and when the worker returns the screwdriver set to its original position, the camera checks again, and the server generates and sends a reward message.

[0188] An example of a prompt to be input to the generative AI model is as follows:

[0189] "Design a support system for managing the location of items in a factory. This system will manage the fixed locations of certain tools and parts and use cameras to check whether the items are in the correct location. If an item deviates from its fixed location, it will send an alert and if it is returned to the correct location, it will generate a reward message to notify. The system will use OpenCV for image analysis and a notification library to send messages. It will also check the room status every minute and manage the items."

[0190] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0191] Step 1: Register items and set their locations on the device

[0192] The user uses a terminal to register items in the system and set the fixed location of each item. The input is the item name and fixed location information, and the output is a process of sending this information to the server to store it in the database.

[0193] Step 2: Capture a fixed position image with the camera

[0194] The camera on the device periodically captures images of the facility. The input is the image capture operation by the camera, and the output is the captured image data. This image data is used in the next processing step.

[0195] Step 3: Send image data to the server

[0196] The terminal sends the captured image data to the server. The input is the captured image data, and the output is the image data sent to the server.

[0197] Step 4: Analyzing the image data

[0198] The server analyzes the received image data and checks whether the item is in the set fixed position. The input is the image data and the fixed position database. The server uses OpenCV to identify the item's position and compares it with the fixed position information. The output is the result of determining whether the item is in the set fixed position.

[0199] Step 5: Detect deviations from home position

[0200] The server detects whether the item is out of place based on the image analysis results. The input is the result of image analysis. The output is to generate an alert notification if the item is out of place.

[0201] Step 6: Sending alert notifications

[0202] The server sends an alert notification to the terminal. The input is information about the item that has moved from its designated location, and the output is an alert notification to the user. For example, a message such as "Please return the maintenance screwdriver set to the top right of the tool shelf" is displayed on the terminal.

[0203] Step 7: Relocate items

[0204] When the user returns the item to the correct position, the camera takes a picture of the item again and sends the image data to the server. The input is the re-taken image data, and the output is the new image data sent to the server.

[0205] Step 8: Reanalysis and location verification

[0206] The server analyzes the received image data and checks whether the item is in its original position. The input is the new image data and the original position database. The output is the result of determining whether the item has been returned to its original position.

[0207] Step 9: Generate Reward Messages

[0208] If the server confirms that the item has been returned to the correct location, it generates a reward message. The input is the data that confirms that the item's location is correct. The output is a reward message (e.g., "Good job!").

[0209] Step 10: Sending Reward Messages

[0210] The server sends a reward message to the terminal. The input is the generated reward message, and the output is a reward notification to the user. The terminal notifies the user of this message by voice or on a screen.

[0211] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0212] The present invention relates to a system that helps parents return items used by children to their correct designated locations, thereby reducing the burden of housework on parents. Furthermore, the present invention incorporates an emotion engine that recognizes the user's emotions and adjusts reward messages based on the emotions. Specifically, the system includes means for setting a designated location for an item, detecting the location of the item with a camera, notifying the user if the item has moved away from its designated location, detecting that the item has been returned to its designated location, and generating and notifying the user of a reward message.

[0213] First, the user registers the child's items in the system through a terminal. The user inputs the name and fixed location of the item, and the fixed location information is input into the system. For example, the user sets the "blue block set" to the "top left of the toy box." The terminal uses a camera to take a photo of the blue block set and its fixed location, and sends the image data to the server. The server analyzes this image data and saves information about the item and its fixed location in a database.

[0214] Next, after the child has finished using an item, the device's camera periodically takes pictures of the room to check whether the item has been returned to its designated location. This image data is sent to the server. The server analyzes the received image data, and if an item has been moved from its designated location, it generates an alert and sends it to the device. The device notifies the child of the alert by voice or by displaying it on the screen. For example, a message such as "Please return the blue block set to the top left of the toy box" may be displayed.

[0215] Furthermore, the system incorporates an emotion engine to recognize the user's emotions. This function records the user's emotions during the operation of locating and returning the item to its original position, and analyzes the emotion data. For example, the system uses a camera or microphone to analyze the user's facial expressions and tone of voice to detect the current emotion.

[0216] If the user returns the item to its correct location, the device takes another photo with the camera and sends the image data to the server. The server analyzes the image data and, once it confirms that the item has been returned to its proper place, adjusts the reward message based on the emotion detected by the emotion engine. For example, if the child is excited, the message displayed might say, "Good job! That's amazing!", whereas if the child is calm, the message displayed might say, "Good job, let's try harder next time."

[0217] Specific examples are as follows:

[0218] Suppose a user registers a "red toy car" in the "center of the shelf." The user opens the app on their device and takes a photo of the item and its designated location to register it. If a child plays with the red toy car and then leaves it on the floor after use, the device's camera takes a photo of the room, which the server analyzes. If the device detects that the item has not been returned to its designated location, a notification appears on the device saying, "Please return the red toy car to the center of the shelf." If the user complies and returns the item to the correct location, the device's camera takes another photo, and the server confirms that it has been returned to the correct location. The emotion engine then analyzes the user's emotions and generates an appropriate reward message, such as "Good job! Amazing!", which the device notifies them by voice or on-screen display.

[0219] This allows the system to develop the habit of returning items to their correct places, reducing the household chore burden on parents and providing effective education by providing feedback based on the user's emotions.

[0220] The processing flow will be explained below.

[0221] Step 1:

[0222] The user opens the app on their device and prepares to register an item in the system. The user selects the "Register Item" menu and enters the item name and its designated location. For example, the user sets the "blue block set" to the "top left of the toy box."

[0223] Step 2:

[0224] The device activates the camera, and the user places the blue block set in front of the camera. The user presses the shutter button, and the device captures the image. The captured image data is temporarily stored in the device.

[0225] Step 3:

[0226] The device transmits the captured image data and fixed position information to the server, which receives the data and analyzes the image data and fixed position information.

[0227] Step 4:

[0228] The server stores the image data and location information in a database, and this storage operation records the item and its location in the system.

[0229] Step 5:

[0230] A child uses an object, for example, takes a blue block set out of a toy box and starts playing with it.

[0231] Step 6:

[0232] The device uses a camera to take pictures of the room at a pre-set frequency, for example, every hour, and sends the image data to a server.

[0233] Step 7:

[0234] The server analyzes the received image data and checks the current location of the item. If the blue block set is not in the "top left of the toy box," the server generates an alert.

[0235] Step 8:

[0236] The server generates an alert and sends it to the device, which then notifies the child of the alert through audio or a display on the screen. For example, it might say, "Put the blue block set back on the top left shelf of the toy box."

[0237] Step 9:

[0238] The emotion engine analyzes the user's facial expressions and tone of voice to recognize their current emotion. The emotion data is processed internally and referenced when generating reward messages.

[0239] Step 10:

[0240] The user instructs the child to return the items to their correct locations, for example, returning the blue block set to the top left of the toy box.

[0241] Step 11:

[0242] The device then starts the camera again and takes a picture of the room. The captured image data is then sent to the server again.

[0243] Step 12:

[0244] The server analyzes the received image data and verifies that the blue block set has been returned to its original position. Based on the verification result, the server adjusts the reward message based on the emotion detected by the emotion engine.

[0245] Step 13:

[0246] The server sends a reward message to the device. The reward message is tailored to the child's current emotions. For example, messages such as "Good job! Amazing!" or "Good job, let's try harder next time" are generated.

[0247] Step 14:

[0248] The device will notify the child with a message via voice or screen display, allowing the child to receive appropriate feedback and form the habit of returning items to their correct place.

[0249] The above are the specific processing steps of the system.

[0250] Example 2

[0251] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0252] Many parents are forced to do additional housework to organize items because children are unable to smoothly return them to their correct locations. Furthermore, reward messages for children are standardized and lack appropriate feedback tailored to the situation. To solve these problems, a system is needed that automatically checks where items are returned and generates reward messages tailored to the user's emotions.

[0253] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0254] In this invention, the server includes means for setting a fixed position for an item used by a user, means for detecting the position of the item with a camera, means for notifying the user when the item has deviated from its fixed position, means for recognizing the user's emotion using an emotion engine and adjusting a reward message based on the emotion, and means for detecting when the item has been returned to its fixed position and generating and notifying the user of the reward message. This allows children to develop the habit of returning items to their correct positions, reduces the housework burden on parents, and enables effective feedback according to the user's emotions.

[0255] "User" refers to a person who uses the system to register and manage items.

[0256] "Item" refers to an object owned by a user that is intended to be registered in the system and returned to a designated location.

[0257] "Home location" refers to a predetermined place where an item should be returned after use.

[0258] "Camera" refers to a device that captures the current location of an item and the state of the room, and acquires the image data.

[0259] "Server" refers to the central processing unit that analyzes image data, operates the emotion engine, locates and notifies items, and generates reward messages.

[0260] "Emotion engine" refers to software or a system for recognizing a user's emotions and tailoring reward messages based on those emotions.

[0261] "Reward message" refers to a message generated in response to returning an item to its proper location that encourages or praises the user's behavior.

[0262] "Notification" refers to a means of communication in the form of audio or screen display to notify the user if an item has not been returned to its proper location or to notify the user of a reward message.

[0263] "Image data" refers to the digital information of images taken by a camera, and is used to analyze the location of an item or the user's emotions.

[0264] The present invention relates to a system that assists parents in returning items used by children to their designated locations, thereby reducing the burden of housework. Furthermore, the present invention incorporates an emotion engine that recognizes the user's emotions and adjusts reward messages based on the emotions. Specific embodiments of the system will be described below.

[0265] This system is primarily composed of three elements: the user, the terminal, and the server. First, the user registers and manages items through the terminal. The terminal provides an interface for inputting the name of the item and its designated location. When the user sets an item and its designated location, for example, by setting a "blue block set" to the "top left of the toy box," the terminal's camera takes a photo of the item and its designated location and sends the image data to the server. The server analyzes the received image data and stores the information about the item and its designated location in a database. An image recognition algorithm is used in this process.

[0266] Next, the process begins to check whether the items are returned to their designated locations after the child has used them. The device's camera periodically takes pictures of the room and sends this image data to the server. The server analyzes the received image data and checks whether the items are in their designated locations. If the items are not returned to their designated locations, the server generates an alert and sends notification data to the device. For example, the server could generate an alert stating, "The blue block set is not in its designated location," and the device will notify the user by voice or by displaying a message on its screen.

[0267] The system also incorporates an emotion engine that recognizes the user's emotions when returning an item to its proper place. When the user returns an item to its correct location, the device's camera and microphone record the user's facial expressions and tone of voice, and send this as emotional data to the server. The server analyzes the received emotional data and classifies the user's current emotional state as "joy," "excitement," "calm," or something similar.

[0268] When the user returns the item to its correct location, the device again takes a photo of the room with its camera and sends the image data to the server. The server analyzes the image data and, once it confirms that the item has been returned to its proper location, adjusts the reward message based on the emotion detected by the emotion engine. For example, if the child is excited, the message "Good job! That's amazing!" is generated, whereas if the child is calm, the message "Good job, let's try harder next time" is generated. Finally, the device notifies the user of the generated reward message by voice or on-screen display.

[0269] As a concrete example, consider the case where a user registers a "red toy car" as "the center of the shelf." The user opens the app on their device and takes a photo of the item and its designated location to register it. If a child plays with the red toy car and then leaves it on the floor after use, the device's camera takes a photo of the room and the server analyzes the image data. If the device detects that the item has not been returned to its designated location, a notification appears on the device saying, "Please return the red toy car to the center of the shelf." If the user complies and returns the item to the correct location, the device's camera takes another photo, and the server confirms that it has been returned to the correct location. The emotion engine then analyzes the user's emotions and generates an appropriate reward message, such as "Good job! Amazing!", which the device notifies them by voice or on-screen display.

[0270] An example of a prompt to be input to a generative AI model is, "Please describe the workflow of a system that helps children return toys to their proper places after playing with them. Please provide a detailed process, including user input, confirmation of the item's proper place, sentiment analysis, and reward message generation."

[0271] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0272] Step 1:

[0273] Product registration

[0274] The user opens the app on their device and inputs the items and their designated locations. For example, the "blue block set" is set to the "top left of the toy box."

[0275] Input: Item name and location information

[0276] The device uses the camera to take a photo of the item and the set fixed position, and sends the captured image data to the server.

[0277] Output: Image data

[0278] Specific operation: The app screen displays the instruction "Take a photo of the blue block set," and the user operates the device's camera to take a photo.

[0279] Step 2:

[0280] Data storage and analysis

[0281] The server analyzes the received image data and stores information about the item and its fixed location in a database.

[0282] Input: Image data

[0283] The server uses image recognition algorithms to identify the items and their locations and stores this information in a database.

[0284] Output: Database entry for item and location

[0285] Specific operation: As the server analyzes the image data, a progress bar will be displayed in the device app, and a confirmation notification will be sent to the user when the analysis is complete.

[0286] Step 3:

[0287] Checking the location of items

[0288] After the child uses the item, the terminal periodically takes pictures of the room and sends the image data to the server.

[0289] Input: Image data of the room condition

[0290] The server analyzes the received image data and checks whether the item is in the set fixed position.

[0291] Output: Item location status (in place or not)

[0292] Specific operation: The device displays a notification saying "Taking a photo" at the time of shooting, and the captured image is sent to the server.

[0293] Step 4:

[0294] Alert Generation

[0295] If the item is not returned to its original location, the server generates an alert and sends notification data to the terminal.

[0296] Input: Item location status

[0297] The server generates an alert message and sends it to the terminal.

[0298] Output: Alert message

[0299] Specific actions: For example, an alert saying "The blue block set is not in its proper place" will be displayed on the device, and a voice message will be played saying "Please return the blue block set to the top left of the toy box."

[0300] Step 5:

[0301] sentiment analysis

[0302] When the user returns the item to its correct location, the device's camera and microphone record the user's facial expressions and tone of voice, and send this data to the server as emotional data.

[0303] Input: User's facial expression data, voice data

[0304] The server analyzes the received emotional data and identifies the current emotional state.

[0305] Output: User's emotional state (e.g., happy, excited, calm)

[0306] What it does: The camera indicator light will turn on to let the user know that sentiment analysis is taking place.

[0307] Step 6:

[0308] Reward message generation and notification

[0309] The terminal then takes another photo with the camera to check if the item has been returned to the correct position, and sends the image data to the server.

[0310] Input: Image data of the returned item

[0311] The server analyzes the image data and, once it determines that the item has been returned to its proper location, adjusts the reward message based on the emotion detected by the emotion engine.

[0312] Output: Reward message

[0313] Specific behavior: For example, if a child is excited, the device will generate a message saying "Good job! That's amazing!" and notify the child by voice or displaying a message on the screen.

[0314] (Application example 2)

[0315] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0316] In traditional factories, parts placement errors and inaccurate storage were common, resulting in problems such as reduced work efficiency and mistakes. Preventing these mistakes required human supervision and alerts, which required a significant amount of effort and time. Furthermore, the lack of a system for properly understanding employees' emotions and motivation and providing feedback accordingly hindered improvements in work efficiency.

[0317] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes a means for setting a fixed position for items used by children, a means for detecting the position of the item with a camera, a means for notifying when the item has deviated from its fixed position, a means for detecting when the item has been returned to its fixed position and for generating and notifying a reward message, and a means for recognizing the emotion of the worker and adjusting the reward message based on the emotion. This makes it possible to prevent mistakes in part placement and a decrease in work efficiency, as well as to appropriately grasp the emotion of the worker and provide feedback accordingly.

[0318] The "means for setting a fixed location for an item used by a child" is a function that allows a user to set a predetermined location within the system where a particular item should be stored or placed.

[0319] The "means for detecting the position of an item using a camera" is a function for capturing the current position of an item using a camera and transmitting the position information to the system.

[0320] The "means for notifying when an article deviates from its fixed position" is a function for sending an alert to the user when an article deviates from its set fixed position.

[0321] The "means for detecting that an item has been returned to its designated location and generating and notifying a reward message" is a function for confirming that an item has been returned to the correct location and sending a reward message to the user based on that.

[0322] "Means for recognizing the emotions of workers and adjusting reward messages based on those emotions" refers to a function that uses sensors such as cameras and microphones to detect the emotional state of workers and adjust the content and tone of reward messages to match those emotions.

[0323] A "camera that monitors the usage status of an item" is a camera that monitors the location and status of an item while it is in use, and sends this information to a server as image data.

[0324] "Means for analyzing image data received by the server and confirming whether the item is in the set fixed position" is a function for analyzing image data sent to the server and confirming whether the item is in the set fixed position.

[0325] This invention relates to a system that aims to accurately manage the location of parts in a factory and improve work efficiency. Specifically, the system supports workers wearing smart glasses, confirms that items are returned to their designated locations, and provides appropriate feedback according to the worker's emotions.

[0326] First, we will explain the process of setting the fixed positions of items. The user uses smart glasses to register the fixed positions of each part in the system. Specifically, the user takes a photo of the part and its fixed position using the smart glasses' camera and sends the image data to the server. At that time, the user enters the part's name and fixed position, and the server stores this information in a database.

[0327] Next, we will explain how to monitor the status of parts when workers use them during work. The camera in the smart glasses periodically captures images of the workspace and sends the image data to a server. The server analyzes the received image data and generates a notification if an item deviates from its designated position. As a specific example, a message such as "Part A has deviated from its designated position. Please return it to the correct position" is displayed on the smart glasses.

[0328] Once the worker returns the item to its correct location, the smart glasses' camera takes another photo and sends the image data to the server. The server analyzes the image data and confirms that the item has been returned to its proper place. At this time, an emotion engine is used to detect the worker's emotions and tailor reward messages based on their emotions. For example, if the worker is focused, a short message like "Good job!" is displayed, whereas if the worker is nervous, a message like "Great job! Try again next time" is displayed.

[0329] For emotion recognition, the system uses a camera and microphone to analyze facial expressions and vocal tone, and employs an emotion recognition model. It processes image data using OpenCV and uses object detection algorithms such as HOGDescriptor to identify the location of objects. It also utilizes an external emotion recognition library.

[0330] Examples of prompts include:

[0331] "Detect the location of part A and notify it to return to its original position."

[0332] "Recognize workers' emotions and provide appropriate reward messages."

[0333] This system can prevent mistakes in part placement and improve work efficiency, as well as promote worker motivation and provide a more effective work environment.

[0334] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0335] Step 1:

[0336] The user registers the fixed location of an item. The user takes a photo of the item and its fixed location through the smart glasses and inputs the item's name and fixed location information. The camera in the smart glasses captures this and the image data is sent to the server. The input data is the item's name and location data, and the output is the registered fixed location information stored on the server.

[0337] Step 2:

[0338] The server analyzes the received image data and stores the fixed location information of the items in a database. Specifically, OpenCV is used to extract the features of the items in the image and store them in the database. The input data is image data and location information, and the output is the fixed location information of the items stored in the database.

[0339] Step 3:

[0340] The camera in the smart glasses periodically captures images of the workspace and sends the captured image data to a server. The input data is the periodically captured image data, and the output is the image data sent to the server.

[0341] Step 4:

[0342] The server analyzes the received image data and checks whether the item is in its designated location. Specifically, it uses OpenCV and HOGDescriptor to detect the item's location and compares it with the database's designated location information. The input data is the captured image data, and the output is the result of determining whether the item is in its designated location.

[0343] Step 5:

[0344] If an item is out of place, the server generates an alert message and notifies the smart glasses. Specifically, it displays a message such as "Part A is out of place. Please return it to the correct position." The input data is the judgment result, and the output is the alert message displayed on the smart glasses.

[0345] Step 6:

[0346] When the worker returns the item to the correct position, the camera in the smart glasses takes a picture again and sends the image data to the server. The input data is the recaptured image data, and the output is the image data sent to the server.

[0347] Step 7:

[0348] The server analyzes the image data again to confirm that the item has been returned to the correct location. If confirmed, it uses an emotion engine to recognize the worker's emotion and generates a reward message based on that emotion. The input data is the received image data and emotion data again, and the output is a reward message based on the emotion.

[0349] Step 8:

[0350] The server sends the generated reward message to the smart glasses and notifies the worker. Specifically, a message such as "Good job! Try your best next time" is displayed. The input data is the generated reward message, and the output is the reward message displayed on the smart glasses.

[0351] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0352] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0353] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0354] [Second embodiment]

[0355] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0356] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0357] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0358] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0359] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0360] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0361] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0362] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0363] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0364] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0365] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0366] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0367] The present invention provides a system for helping children put away items in their correct places after use, thereby reducing the burden of housework on parents. The system includes means for setting fixed positions for items, detecting the positions of the items with a camera, notifying the parent if the item has moved away from its fixed position, detecting that the item has been returned to its fixed position, and generating a reward message to notify the parent.

[0368] Specifically, the user registers items through a terminal. The user registers children's toys and household items in the system and sets fixed locations for the items. For example, the user sets the "blue block set" to the "top left of the toy box." The terminal uses a camera to take a photo of the blue block set and its fixed location, and sends the image data to the server. The server analyzes this image data and stores information about the items and their fixed locations in a database.

[0369] If a child uses an item but does not return it to its registered location, the system detects this and notifies the user. The device's camera periodically takes pictures of the room and sends the image data to the server. The server analyzes the received image data and, if an item is not in its designated location, sends an alert to the device. The device notifies the user by voice or on the screen, saying, "Please return the blue block set to the top left of the toy box."

[0370] When the user returns the item to the correct location, the device takes a photo with the camera again and sends the image data to the server. The server analyzes the image data and, if it confirms that the item has been returned to its designated location, generates a reward message. For example, the server may generate a message such as "Good job! That's great!" and send it to the device. The device then notifies the user of the message by voice or by displaying it on its screen.

[0371] Specific examples are as follows:

[0372] Suppose a user registers a "red toy car" as "the center of the shelf." The user opens the app on their device and takes a photo of the item and its designated location to register it. If a child plays with the red toy car and leaves it on the floor after using it, the system will determine that it is not in its designated location. The device's camera takes a photo of the room, which the server analyzes. If it detects that the item has not been returned to its designated location, a notification will appear on the device saying, "Please return the red toy car to the center of the shelf." If the child complies and returns the item to the correct location, the device's camera will take another photo, and the server will confirm that it has been returned to the correct location. This will cause the server to generate a reward message, and the device will issue an audio notification saying, "Good job!"

[0373] The system of the present invention can improve the efficiency of household chores and reduce the burden on parents.

[0374] The processing flow will be explained below.

[0375] Step 1:

[0376] The user opens the app on their device to register their child's items in the system. The user selects the "Register Item" menu and enters the item name and its designated location. For example, the "blue block set" is set to the "top left of the toy box."

[0377] Step 2:

[0378] The device's camera is activated, and the user places the blue block set in front of the camera. The user presses the shutter button, and the device captures the image. The captured image data is temporarily stored in the device.

[0379] Step 3:

[0380] The device transmits the captured image data and fixed position information to the server, which receives the data and analyzes the image data and fixed position information.

[0381] Step 4:

[0382] The server stores the image data and location information in a database, and this storage operation records the item and its location in the system.

[0383] Step 5:

[0384] A child uses an object, for example, takes a blue block set out of a toy box and starts playing with it.

[0385] Step 6:

[0386] The device uses a camera to take pictures of the room at a pre-set frequency, for example, every hour, and sends the image data to a server.

[0387] Step 7:

[0388] The server analyzes the received image data and checks the current location of the item. If the blue block set is not in the "top left of the toy box," the server generates an alert.

[0389] Step 8:

[0390] The server generates an alert and sends it to the device, which then notifies the child of the alert through audio or a display on the screen. For example, it might say, "Put the blue block set back on the top left shelf of the toy box."

[0391] Step 9:

[0392] The user instructs the child to return the items to their correct locations, for example, returning the blue block set to the top left of the toy box.

[0393] Step 10:

[0394] The device then starts the camera again and takes a picture of the room. The captured image data is then sent to the server again.

[0395] Step 11:

[0396] The server analyzes the received image data and confirms that the blue block set has been returned to its original position. Based on the confirmation result, the server generates a reward message.

[0397] Step 12:

[0398] The server sends a reward message to the device, which then notifies the child of the message by voice or on-screen display. For example, "Great job! Amazing!" is displayed.

[0399] This system helps children develop the habit of returning items to their correct places, reducing the household chore burden on parents.

[0400] Example 1

[0401] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0402] Children often do not put away their belongings or household items in the correct place after playing, which creates a problem for parents, who have to spend a lot of time and effort tidying up for them. Furthermore, children have fewer opportunities to develop the habit of tidying up on their own. This increases the household chore burden and creates the problem of increased stress for parents.

[0403] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0404] In this invention, the server includes: means for a user to set a fixed position for an item; means for taking a picture of the item and its fixed position with a camera on the terminal and sending the image data to the server; means for the server to analyze the image data and store information about the item and its fixed position in a database; means for sending an alert notification to the terminal if the item has deviated from its fixed position; means for taking another picture with the camera when the user returns the item to the correct position and sending the image data to the server; and means for the server to analyze the image data, confirm that the item has been returned to its fixed position, and generate a reward message to notify the user. This automatically monitors the tidying up of items, reducing the housework burden on parents and enabling children to develop the habit of tidying up.

[0405] A "user" is a person, including an adult in the household, who operates the system and registers items and sets their locations.

[0406] "Items" refer to items to be organized, such as toys and household items used by children.

[0407] A "home location" is a specific place or area where an item is to be stored.

[0408] A "terminal" is a device that includes a computer device operated by a user and a camera mounted on that device.

[0409] A "camera" is a device that is installed in a terminal and takes pictures of items, their fixed positions, and the state of the room to generate image data.

[0410] The "server" is a computer system that analyzes image data sent from the terminal, stores the information in a database, and generates notifications and reward messages.

[0411] "Image data" refers to visual information of an object or the state of a room captured by a camera.

[0412] A "database" is an information management system located within a server that stores and manages information on analyzed items and their fixed locations.

[0413] An "alert notification" is a message sent to the user via the terminal to warn them when an item is not in its designated location.

[0414] A "reward message" is a message of praise or encouragement that is generated by the server and sent to the user via the terminal when the user returns the item to the correct location.

[0415] "Analysis" refers to the operation in which the server processes the received image data, identifies the location of the item, and makes appropriate decisions.

[0416] This invention is a system that helps children put things away in the correct place after use, reducing the household chore burden on parents. The system is mainly composed of the user, terminal, and server elements, and functions as follows.

[0417] First, the user uses the device's app to register an item. When registering an item, the user inputs the name of the item (e.g., a toy or household item) and its designated location. Next, the device's camera is used to take a photo of the item and its designated location, and the image data is sent to the server. For example, if the user wants to set the "blue block set" to the "top left of the toy box," the user takes a photo of that location with the device's camera.

[0418] The server receives image data sent from the terminal and analyzes the items and their fixed locations. The analysis results are saved in a database, and the item location information is managed as digital data. The server's analysis makes it possible to determine whether the items have been placed in the correct locations.

[0419] The device is equipped with a timer function that periodically takes photos of the room's condition, and the camera takes photos at specified times. The image data is then sent back to the server, which analyzes the received images. If an item is not in its registered location, the server generates an alert message and notifies the user via the device. For example, a specific message such as "Please return the blue block set to the top left of the toy box" is notified by voice or displayed on the screen.

[0420] When the user puts the items away in the correct place, the device's camera takes another photo of the situation and sends the image data to the server. The server analyzes this data and confirms that the items have been returned to their proper places. If successful, the server generates a reward message and notifies the user via the device. For example, a praising message such as "Good job! Amazing!" can be conveyed by voice or displayed on the screen, increasing the child's motivation to tidy up on their own.

[0421] As a concrete example, consider the case where a user registers a "red toy car" as "the center of the shelf." The user enters the name and designated location of the item into the device's app and takes a photo of the location with the device's camera. The device then sends the image data to the server, which analyzes it and stores it in a database. The device then periodically takes photos of the room, and the server analyzes the images to check whether the item is in its designated location. If the item is not in its designated location, an alert notification is sent to the user, prompting them to return the item to its correct location.

[0422] This system promotes tidiness and efficiency in the home, reduces the burden of housework on parents, and helps children develop the habit of tidying up on their own, which is expected to improve the overall home environment.

[0423] Examples of prompts to input to a generative AI model include:

[0424] "I've registered the red toy car in the center of the shelf, but what's the notification mechanism for when a child forgets to put it back?"

[0425] "Tell me the process that generates the reward message when the child returns the object to its correct location."

[0426] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0427] System program processing flow

[0428] Step 1: Registering and positioning items

[0429] Specific behavior:

[0430] 1. The user launches the app on their device and opens the item registration screen.

[0431] 2. The user selects the "Register Item" option and enters the name of the item (e.g., "Blue Block Set").

[0432] 3. The user inputs the item's fixed location (e.g., "top left of the toy box").

[0433] 4. The user takes a photo of the item and its designated location using the device's camera.

[0434] Input and Output:

[0435] Input: Name and location of the object, camera image.

[0436] Data processing: Associate item names and location information with camera images.

[0437] Output: Send image data to the server.

[0438] Step 2: Receiving and saving image data

[0439] 1. The server receives the image data sent from the terminal.

[0440] 2. The server analyzes the received image data and extracts information about the item and its location.

[0441] 3. The server stores the item and its location information in a database.

[0442] Input and Output:

[0443] Input: Image data sent from the device.

[0444] Data processing: Extraction of information about items and their locations through image analysis.

[0445] Output: Saves the information to the database and sends a message to the terminal confirming the success.

[0446] Step 3: Photograph the room

[0447] 1. The device takes photos of the room at specified intervals.

[0448] Input and Output:

[0449] Enter: Timer settings.

[0450] Data processing: Regular photography.

[0451] Output: Captured image data.

[0452] Step 4: Sending image data

[0453] 1. The device sends the captured image data to the server.

[0454] Input and Output:

[0455] Input: Captured image data.

[0456] Data processing: Preparing image data for transmission.

[0457] Output: Send image data to the server.

[0458] Step 5: Locate your items

[0459] 1. The server analyzes the received image data and checks whether the item is in the registered location.

[0460] 2. If the server determines that the item is not in place, it generates an alert message.

[0461] 3. The server sends an alert message to the device.

[0462] Input and Output:

[0463] Input: Image data sent to the server.

[0464] Data processing: Confirmation of item location through image analysis.

[0465] Output: Generates and sends an alert message.

[0466] Step 6: Alert Notifications

[0467] 1. The terminal notifies the user of the received alert message by voice or on-screen display.

[0468] Specific behavior:

[0469] For example, a message might appear on the device screen saying, "Please return the blue block set to the top left of the toy box."

[0470] Input and Output:

[0471] Input: The alert message sent by the server.

[0472] Data processing: Preparing messages for audio or screen display.

[0473] Output: Alert notification to the user.

[0474] Step 7: Check that the item is in place

[0475] 1. The user returns the item to its correct location.

[0476] 2. The device will take another photo of the situation with its camera.

[0477] 3. The device sends the captured image data to the server.

[0478] Input and Output:

[0479] Input: User actions, captured image data.

[0480] Data processing: Preparing image data for transmission.

[0481] Output: Send image data to the server.

[0482] Step 8: Reward Message Generation and Notification

[0483] 1. The server analyzes the received image data again and confirms that the item has been returned to its designated location.

[0484] 2. The server generates a reward message and sends it to the terminal.

[0485] 3. The terminal notifies the user of the reward message by voice or on-screen display.

[0486] Specific behavior:

[0487] For example, a message like "Great job! Awesome!" will appear on the device screen.

[0488] Input and Output:

[0489] Input: Re-captured image data.

[0490] Data processing: Image analysis and verification, reward message generation.

[0491] Output: Send reward message to device and notify user.

[0492] (Application example 1)

[0493] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0494] Currently, many homes and factories face the problem of items not being returned to their correct locations after use. This increases the household chore burden on parents at home and reduces production efficiency in factories. In particular, in industrial sites, improper management of tools and parts can increase the risk of production line shutdowns and accidents. To solve these problems, a system is needed that automatically manages and checks the location of items and notifies workers as necessary.

[0495] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0496] In this invention, the server includes a means for setting a fixed position for an item used by a user, a means for detecting the position of the item with a camera, a means for notifying when the item has deviated from the fixed position, and a means for detecting when the item has been returned to the fixed position and generating a reward message to notify the user. This automates the management of item positions in homes and factories, reducing the burden of housework on parents, improving factory production efficiency, and ensuring safety.

[0497] "User" refers to a person who uses an item in a home, factory, etc.

[0498] "Article" refers to a concrete object used for a specific purpose, such as a child's toy or a tool or part used in a factory.

[0499] "Home location" refers to a predetermined location to which an item should be returned before or after use.

[0500] The "means for setting" refers to a device or software that allows a user to register the fixed location of an item and save that information.

[0501] "Detection means" refers to technology for recognizing the location of an item using a device such as a camera and acquiring that information.

[0502] "Means for notifying" means an audio, visual, or other method for notifying a user if an item is displaced from its designated location.

[0503] "Means for generating and notifying reward messages" refers to devices or software that confirm that an item has been returned to its proper location and send a message of praise or encouragement to the user.

[0504] "Camera" refers to a photographing device for photographing the position of an item and acquiring image data thereof.

[0505] "Image data" refers to digital data containing image information captured by a camera.

[0506] "Server" refers to a computer system that receives and analyzes image data and generates necessary notifications and messages.

[0507] "Software" refers to programs or applications that run on a computer or other digital device.

[0508] "OpenCV" is a library for image analysis and is used to detect the position of objects.

[0509] "Notification Library" means a program library for generating and sending reward messages and alerts.

[0510] This invention is a system that supports returning used items to their correct positions using a terminal in homes and factories. The system includes means for setting a fixed position for the item, detecting it with a camera, notifying the user if the item has deviated from its fixed position, and detecting that the item has been returned to the correct position and generating and notifying the user of a reward message.

[0511] First, the user uses a terminal to register items and set their fixed locations. The user registers the items in the system and sets the fixed location for each item. This information is sent to the server and saved in the database. For example, in a factory, a "maintenance screwdriver set" is set to the "top right of the tool shelf."

[0512] Next, the camera periodically captures images of the facility and sends the image data to a server. The server analyzes the image data to check whether the items are in their designated locations. This is done using an image analysis library such as OpenCV. If, based on the analysis results, the items have not been returned to their designated locations, the server sends a notification to the terminal. For example, an alert will be displayed saying, "Please return the maintenance screwdriver set to the top right shelf of the tool shelf."

[0513] When the user returns the item to the correct position, the camera takes a picture again and sends the image data to the server. The server analyzes the image again and confirms that the item is in its correct position. Once this is confirmed, the server generates a reward message and sends a message such as "Good job!" to the device. A specific notification library is used to send the reward message.

[0514] As a specific example, if a factory worker leaves a "maintenance screwdriver set" on the floor after use, the camera captures the state, the server analyzes it and issues a notification, and when the worker returns the screwdriver set to its original position, the camera checks again, and the server generates and sends a reward message.

[0515] An example of a prompt to be input to the generative AI model is as follows:

[0516] "Design a support system for managing the location of items in a factory. This system will manage the fixed locations of certain tools and parts and use cameras to check whether the items are in the correct location. If an item deviates from its fixed location, it will send an alert and if it is returned to the correct location, it will generate a reward message to notify. The system will use OpenCV for image analysis and a notification library to send messages. It will also check the room status every minute and manage the items."

[0517] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0518] Step 1: Register items and set their locations on the device

[0519] The user uses a terminal to register items in the system and set the fixed location of each item. The input is the item name and fixed location information, and the output is a process of sending this information to the server to store it in the database.

[0520] Step 2: Capture a fixed position image with the camera

[0521] The camera on the device periodically captures images of the facility. The input is the image capture operation by the camera, and the output is the captured image data. This image data is used in the next processing step.

[0522] Step 3: Send image data to the server

[0523] The terminal sends the captured image data to the server. The input is the captured image data, and the output is the image data sent to the server.

[0524] Step 4: Analyzing the image data

[0525] The server analyzes the received image data and checks whether the item is in the set fixed position. The input is the image data and the fixed position database. The server uses OpenCV to identify the item's position and compares it with the fixed position information. The output is the result of determining whether the item is in the set fixed position.

[0526] Step 5: Detect deviations from home position

[0527] The server detects whether the item is out of place based on the image analysis results. The input is the result of image analysis. The output is to generate an alert notification if the item is out of place.

[0528] Step 6: Sending alert notifications

[0529] The server sends an alert notification to the terminal. The input is information about the item that has moved from its designated location, and the output is an alert notification to the user. For example, a message such as "Please return the maintenance screwdriver set to the top right of the tool shelf" is displayed on the terminal.

[0530] Step 7: Relocate items

[0531] When the user returns the item to the correct position, the camera takes a picture of the item again and sends the image data to the server. The input is the re-taken image data, and the output is the new image data sent to the server.

[0532] Step 8: Reanalysis and location verification

[0533] The server analyzes the received image data and checks whether the item is in its original position. The input is the new image data and the original position database. The output is the result of determining whether the item has been returned to its original position.

[0534] Step 9: Generate Reward Messages

[0535] If the server confirms that the item has been returned to the correct location, it generates a reward message. The input is the data that confirms that the item's location is correct. The output is a reward message (e.g., "Good job!").

[0536] Step 10: Sending Reward Messages

[0537] The server sends a reward message to the terminal. The input is the generated reward message, and the output is a reward notification to the user. The terminal notifies the user of this message by voice or on a screen.

[0538] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0539] The present invention relates to a system that helps parents return items used by children to their correct designated locations, thereby reducing the burden of housework on parents. Furthermore, the present invention incorporates an emotion engine that recognizes the user's emotions and adjusts reward messages based on the emotions. Specifically, the system includes means for setting a designated location for an item, detecting the location of the item with a camera, notifying the user if the item has moved away from its designated location, detecting that the item has been returned to its designated location, and generating and notifying the user of a reward message.

[0540] First, the user registers the child's items in the system through a terminal. The user inputs the name and fixed location of the item, and the fixed location information is input into the system. For example, the user sets the "blue block set" to the "top left of the toy box." The terminal uses a camera to take a photo of the blue block set and its fixed location, and sends the image data to the server. The server analyzes this image data and saves information about the item and its fixed location in a database.

[0541] Next, after the child has finished using an item, the device's camera periodically takes pictures of the room to check whether the item has been returned to its designated location. This image data is sent to the server. The server analyzes the received image data, and if an item has been moved from its designated location, it generates an alert and sends it to the device. The device notifies the child of the alert by voice or by displaying it on the screen. For example, a message such as "Please return the blue block set to the top left of the toy box" may be displayed.

[0542] Furthermore, the system incorporates an emotion engine to recognize the user's emotions. This function records the user's emotions during the operation of locating and returning the item to its original position, and analyzes the emotion data. For example, the system uses a camera or microphone to analyze the user's facial expressions and tone of voice to detect the current emotion.

[0543] If the user returns the item to its correct location, the device takes another photo with the camera and sends the image data to the server. The server analyzes the image data and, once it confirms that the item has been returned to its proper place, adjusts the reward message based on the emotion detected by the emotion engine. For example, if the child is excited, the message displayed might say, "Good job! That's amazing!", whereas if the child is calm, the message displayed might say, "Good job, let's try harder next time."

[0544] Specific examples are as follows:

[0545] Suppose a user registers a "red toy car" in the "center of the shelf." The user opens the app on their device and takes a photo of the item and its designated location to register it. If a child plays with the red toy car and then leaves it on the floor after use, the device's camera takes a photo of the room, which the server analyzes. If the device detects that the item has not been returned to its designated location, a notification appears on the device saying, "Please return the red toy car to the center of the shelf." If the user complies and returns the item to the correct location, the device's camera takes another photo, and the server confirms that it has been returned to the correct location. The emotion engine then analyzes the user's emotions and generates an appropriate reward message, such as "Good job! Amazing!", which the device notifies them by voice or on-screen display.

[0546] This allows the system to develop the habit of returning items to their correct places, reducing the household chore burden on parents and providing effective education by providing feedback based on the user's emotions.

[0547] The processing flow will be explained below.

[0548] Step 1:

[0549] The user opens the app on their device and prepares to register an item in the system. The user selects the "Register Item" menu and enters the item name and its designated location. For example, the user sets the "blue block set" to the "top left of the toy box."

[0550] Step 2:

[0551] The device activates the camera, and the user places the blue block set in front of the camera. The user presses the shutter button, and the device captures the image. The captured image data is temporarily stored in the device.

[0552] Step 3:

[0553] The device transmits the captured image data and fixed position information to the server, which receives the data and analyzes the image data and fixed position information.

[0554] Step 4:

[0555] The server stores the image data and location information in a database, and this storage operation records the item and its location in the system.

[0556] Step 5:

[0557] A child uses an object, for example, takes a blue block set out of a toy box and starts playing with it.

[0558] Step 6:

[0559] The device uses a camera to take pictures of the room at a pre-set frequency, for example, every hour, and sends the image data to a server.

[0560] Step 7:

[0561] The server analyzes the received image data and checks the current location of the item. If the blue block set is not in the "top left of the toy box," the server generates an alert.

[0562] Step 8:

[0563] The server generates an alert and sends it to the device, which then notifies the child of the alert through audio or a display on the screen. For example, it might say, "Put the blue block set back on the top left shelf of the toy box."

[0564] Step 9:

[0565] The emotion engine analyzes the user's facial expressions and tone of voice to recognize their current emotion. The emotion data is processed internally and referenced when generating reward messages.

[0566] Step 10:

[0567] The user instructs the child to return the items to their correct locations, for example, returning the blue block set to the top left of the toy box.

[0568] Step 11:

[0569] The device then starts the camera again and takes a picture of the room. The captured image data is then sent to the server again.

[0570] Step 12:

[0571] The server analyzes the received image data and verifies that the blue block set has been returned to its original position. Based on the verification result, the server adjusts the reward message based on the emotion detected by the emotion engine.

[0572] Step 13:

[0573] The server sends a reward message to the device. The reward message is tailored to the child's current emotions. For example, messages such as "Good job! Amazing!" or "Good job, let's try harder next time" are generated.

[0574] Step 14:

[0575] The device will notify the child with a message via voice or screen display, allowing the child to receive appropriate feedback and form the habit of returning items to their correct place.

[0576] The above are the specific processing steps of the system.

[0577] Example 2

[0578] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0579] Many parents are forced to do additional housework to organize items because children are unable to smoothly return them to their correct locations. Furthermore, reward messages for children are standardized and lack appropriate feedback tailored to the situation. To solve these problems, a system is needed that automatically checks where items are returned and generates reward messages tailored to the user's emotions.

[0580] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0581] In this invention, the server includes means for setting a fixed position for an item used by a user, means for detecting the position of the item with a camera, means for notifying the user when the item has deviated from its fixed position, means for recognizing the user's emotion using an emotion engine and adjusting a reward message based on the emotion, and means for detecting when the item has been returned to its fixed position and generating and notifying the user of the reward message. This allows children to develop the habit of returning items to their correct positions, reduces the housework burden on parents, and enables effective feedback according to the user's emotions.

[0582] "User" refers to a person who uses the system to register and manage items.

[0583] "Item" refers to an object owned by a user that is intended to be registered in the system and returned to a designated location.

[0584] "Home location" refers to a predetermined place where an item should be returned after use.

[0585] "Camera" refers to a device that captures the current location of an item and the state of the room, and acquires the image data.

[0586] "Server" refers to the central processing unit that analyzes image data, operates the emotion engine, locates and notifies items, and generates reward messages.

[0587] "Emotion engine" refers to software or a system for recognizing a user's emotions and tailoring reward messages based on those emotions.

[0588] "Reward message" refers to a message generated in response to returning an item to its proper location that encourages or praises the user's behavior.

[0589] "Notification" refers to a means of communication in the form of audio or screen display to notify the user if an item has not been returned to its proper location or to notify the user of a reward message.

[0590] "Image data" refers to the digital information of images taken by a camera, and is used to analyze the location of an item or the user's emotions.

[0591] The present invention relates to a system that assists parents in returning items used by children to their designated locations, thereby reducing the burden of housework. Furthermore, the present invention incorporates an emotion engine that recognizes the user's emotions and adjusts reward messages based on the emotions. Specific embodiments of the system will be described below.

[0592] This system is primarily composed of three elements: the user, the terminal, and the server. First, the user registers and manages items through the terminal. The terminal provides an interface for inputting the name of the item and its designated location. When the user sets an item and its designated location, for example, by setting a "blue block set" to the "top left of the toy box," the terminal's camera takes a photo of the item and its designated location and sends the image data to the server. The server analyzes the received image data and stores the information about the item and its designated location in a database. An image recognition algorithm is used in this process.

[0593] Next, the process begins to check whether the items are returned to their designated locations after the child has used them. The device's camera periodically takes pictures of the room and sends this image data to the server. The server analyzes the received image data and checks whether the items are in their designated locations. If the items are not returned to their designated locations, the server generates an alert and sends notification data to the device. For example, the server could generate an alert stating, "The blue block set is not in its designated location," and the device will notify the user by voice or by displaying a message on its screen.

[0594] The system also incorporates an emotion engine that recognizes the user's emotions when returning an item to its proper place. When the user returns an item to its correct location, the device's camera and microphone record the user's facial expressions and tone of voice, and send this as emotional data to the server. The server analyzes the received emotional data and classifies the user's current emotional state as "joy," "excitement," "calm," or something similar.

[0595] When the user returns the item to its correct location, the device again takes a photo of the room with its camera and sends the image data to the server. The server analyzes the image data and, once it confirms that the item has been returned to its proper location, adjusts the reward message based on the emotion detected by the emotion engine. For example, if the child is excited, the message "Good job! That's amazing!" is generated, whereas if the child is calm, the message "Good job, let's try harder next time" is generated. Finally, the device notifies the user of the generated reward message by voice or on-screen display.

[0596] As a concrete example, consider the case where a user registers a "red toy car" as "the center of the shelf." The user opens the app on their device and takes a photo of the item and its designated location to register it. If a child plays with the red toy car and then leaves it on the floor after use, the device's camera takes a photo of the room and the server analyzes the image data. If the device detects that the item has not been returned to its designated location, a notification appears on the device saying, "Please return the red toy car to the center of the shelf." If the user complies and returns the item to the correct location, the device's camera takes another photo, and the server confirms that it has been returned to the correct location. The emotion engine then analyzes the user's emotions and generates an appropriate reward message, such as "Good job! Amazing!", which the device notifies them by voice or on-screen display.

[0597] An example of a prompt to be input to a generative AI model is, "Please describe the workflow of a system that helps children return toys to their proper places after playing with them. Please provide a detailed process, including user input, confirmation of the item's proper place, sentiment analysis, and reward message generation."

[0598] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0599] Step 1:

[0600] Product registration

[0601] The user opens the app on their device and inputs the items and their designated locations. For example, the "blue block set" is set to the "top left of the toy box."

[0602] Input: Item name and location information

[0603] The device uses the camera to take a photo of the item and the set fixed position, and sends the captured image data to the server.

[0604] Output: Image data

[0605] Specific operation: The app screen displays the instruction "Take a photo of the blue block set," and the user operates the device's camera to take a photo.

[0606] Step 2:

[0607] Data storage and analysis

[0608] The server analyzes the received image data and stores information about the item and its fixed location in a database.

[0609] Input: Image data

[0610] The server uses image recognition algorithms to identify the items and their locations and stores this information in a database.

[0611] Output: Database entry for item and location

[0612] Specific operation: As the server analyzes the image data, a progress bar will be displayed in the device app, and a confirmation notification will be sent to the user when the analysis is complete.

[0613] Step 3:

[0614] Checking the location of items

[0615] After the child uses the item, the terminal periodically takes pictures of the room and sends the image data to the server.

[0616] Input: Image data of the room condition

[0617] The server analyzes the received image data and checks whether the item is in the set fixed position.

[0618] Output: Item location status (in place or not)

[0619] Specific operation: The device displays a notification saying "Taking a photo" at the time of shooting, and the captured image is sent to the server.

[0620] Step 4:

[0621] Alert Generation

[0622] If the item is not returned to its original location, the server generates an alert and sends notification data to the terminal.

[0623] Input: Item location status

[0624] The server generates an alert message and sends it to the terminal.

[0625] Output: Alert message

[0626] Specific actions: For example, an alert saying "The blue block set is not in its proper place" will be displayed on the device, and a voice message will be played saying "Please return the blue block set to the top left of the toy box."

[0627] Step 5:

[0628] sentiment analysis

[0629] When the user returns the item to its correct location, the device's camera and microphone record the user's facial expressions and tone of voice, and send this data to the server as emotional data.

[0630] Input: User's facial expression data, voice data

[0631] The server analyzes the received emotional data and identifies the current emotional state.

[0632] Output: User's emotional state (e.g., happy, excited, calm)

[0633] What it does: The camera indicator light will turn on to let the user know that sentiment analysis is taking place.

[0634] Step 6:

[0635] Reward message generation and notification

[0636] The terminal then takes another photo with the camera to check if the item has been returned to the correct position, and sends the image data to the server.

[0637] Input: Image data of the returned item

[0638] The server analyzes the image data and, once it determines that the item has been returned to its proper location, adjusts the reward message based on the emotion detected by the emotion engine.

[0639] Output: Reward message

[0640] Specific behavior: For example, if a child is excited, the device will generate a message saying "Good job! That's amazing!" and notify the child by voice or displaying a message on the screen.

[0641] (Application example 2)

[0642] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0643] In traditional factories, parts placement errors and inaccurate storage were common, resulting in problems such as reduced work efficiency and mistakes. Preventing these mistakes required human supervision and alerts, which required a significant amount of effort and time. Furthermore, the lack of a system for properly understanding employees' emotions and motivation and providing feedback accordingly hindered improvements in work efficiency.

[0644] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes a means for setting a fixed position for items used by children, a means for detecting the position of the item with a camera, a means for notifying when the item has deviated from its fixed position, a means for detecting when the item has been returned to its fixed position and for generating and notifying a reward message, and a means for recognizing the emotion of the worker and adjusting the reward message based on the emotion. This makes it possible to prevent mistakes in part placement and a decrease in work efficiency, as well as to appropriately grasp the emotion of the worker and provide feedback accordingly.

[0645] The "means for setting a fixed location for an item used by a child" is a function that allows a user to set a predetermined location within the system where a particular item should be stored or placed.

[0646] The "means for detecting the position of an item using a camera" is a function for capturing the current position of an item using a camera and transmitting the position information to the system.

[0647] The "means for notifying when an article deviates from its fixed position" is a function for sending an alert to the user when an article deviates from its set fixed position.

[0648] The "means for detecting that an item has been returned to its designated location and generating and notifying a reward message" is a function for confirming that an item has been returned to the correct location and sending a reward message to the user based on that.

[0649] "Means for recognizing the emotions of workers and adjusting reward messages based on those emotions" refers to a function that uses sensors such as cameras and microphones to detect the emotional state of workers and adjust the content and tone of reward messages to match those emotions.

[0650] A "camera that monitors the usage status of an item" is a camera that monitors the location and status of an item while it is in use, and sends this information to a server as image data.

[0651] "Means for analyzing image data received by the server and confirming whether the item is in the set fixed position" is a function for analyzing image data sent to the server and confirming whether the item is in the set fixed position.

[0652] This invention relates to a system that aims to accurately manage the location of parts in a factory and improve work efficiency. Specifically, the system supports workers wearing smart glasses, confirms that items are returned to their designated locations, and provides appropriate feedback according to the worker's emotions.

[0653] First, we will explain the process of setting the fixed positions of items. The user uses smart glasses to register the fixed positions of each part in the system. Specifically, the user takes a photo of the part and its fixed position using the smart glasses' camera and sends the image data to the server. At that time, the user enters the part's name and fixed position, and the server stores this information in a database.

[0654] Next, we will explain how to monitor the status of parts when workers use them during work. The camera in the smart glasses periodically captures images of the workspace and sends the image data to a server. The server analyzes the received image data and generates a notification if an item deviates from its designated position. As a specific example, a message such as "Part A has deviated from its designated position. Please return it to the correct position" is displayed on the smart glasses.

[0655] Once the worker returns the item to its correct location, the smart glasses' camera takes another photo and sends the image data to the server. The server analyzes the image data and confirms that the item has been returned to its proper place. At this time, an emotion engine is used to detect the worker's emotions and tailor reward messages based on their emotions. For example, if the worker is focused, a short message like "Good job!" is displayed, whereas if the worker is nervous, a message like "Great job! Try again next time" is displayed.

[0656] For emotion recognition, the system uses a camera and microphone to analyze facial expressions and vocal tone, and employs an emotion recognition model. It processes image data using OpenCV and uses object detection algorithms such as HOGDescriptor to identify the location of objects. It also utilizes an external emotion recognition library.

[0657] Examples of prompts include:

[0658] "Detect the location of part A and notify it to return to its original position."

[0659] "Recognize workers' emotions and provide appropriate reward messages."

[0660] This system can prevent mistakes in part placement and improve work efficiency, as well as promote worker motivation and provide a more effective work environment.

[0661] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0662] Step 1:

[0663] The user registers the fixed location of an item. The user takes a photo of the item and its fixed location through the smart glasses and inputs the item's name and fixed location information. The camera in the smart glasses captures this and the image data is sent to the server. The input data is the item's name and location data, and the output is the registered fixed location information stored on the server.

[0664] Step 2:

[0665] The server analyzes the received image data and stores the fixed location information of the items in a database. Specifically, OpenCV is used to extract the features of the items in the image and store them in the database. The input data is image data and location information, and the output is the fixed location information of the items stored in the database.

[0666] Step 3:

[0667] The camera in the smart glasses periodically captures images of the workspace and sends the captured image data to a server. The input data is the periodically captured image data, and the output is the image data sent to the server.

[0668] Step 4:

[0669] The server analyzes the received image data and checks whether the item is in its designated location. Specifically, it uses OpenCV and HOGDescriptor to detect the item's location and compares it with the database's designated location information. The input data is the captured image data, and the output is the result of determining whether the item is in its designated location.

[0670] Step 5:

[0671] If an item is out of place, the server generates an alert message and notifies the smart glasses. Specifically, it displays a message such as "Part A is out of place. Please return it to the correct position." The input data is the judgment result, and the output is the alert message displayed on the smart glasses.

[0672] Step 6:

[0673] When the worker returns the item to the correct position, the camera in the smart glasses takes a picture again and sends the image data to the server. The input data is the recaptured image data, and the output is the image data sent to the server.

[0674] Step 7:

[0675] The server analyzes the image data again to confirm that the item has been returned to the correct location. If confirmed, it uses an emotion engine to recognize the worker's emotion and generates a reward message based on that emotion. The input data is the received image data and emotion data again, and the output is a reward message based on the emotion.

[0676] Step 8:

[0677] The server sends the generated reward message to the smart glasses and notifies the worker. Specifically, a message such as "Good job! Try your best next time" is displayed. The input data is the generated reward message, and the output is the reward message displayed on the smart glasses.

[0678] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0679] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0680] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0681] [Third embodiment]

[0682] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0683] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[0684] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0685] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0686] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0687] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0688] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0689] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0690] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0691] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0692] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0693] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0694] The present invention provides a system for helping children put away items in their correct places after use, thereby reducing the burden of housework on parents. The system includes means for setting fixed positions for items, detecting the positions of the items with a camera, notifying the parent if the item has moved away from its fixed position, detecting that the item has been returned to its fixed position, and generating a reward message to notify the parent.

[0695] Specifically, the user registers items through a terminal. The user registers children's toys and household items in the system and sets fixed locations for the items. For example, the user sets the "blue block set" to the "top left of the toy box." The terminal uses a camera to take a photo of the blue block set and its fixed location, and sends the image data to the server. The server analyzes this image data and stores information about the items and their fixed locations in a database.

[0696] If a child uses an item but does not return it to its registered location, the system detects this and notifies the user. The device's camera periodically takes pictures of the room and sends the image data to the server. The server analyzes the received image data and, if an item is not in its designated location, sends an alert to the device. The device notifies the user by voice or on the screen, saying, "Please return the blue block set to the top left of the toy box."

[0697] When the user returns the item to the correct location, the device takes a photo with the camera again and sends the image data to the server. The server analyzes the image data and, if it confirms that the item has been returned to its designated location, generates a reward message. For example, the server may generate a message such as "Good job! That's great!" and send it to the device. The device then notifies the user of the message by voice or by displaying it on its screen.

[0698] Specific examples are as follows:

[0699] Suppose a user registers a "red toy car" as "the center of the shelf." The user opens the app on their device and takes a photo of the item and its designated location to register it. If a child plays with the red toy car and leaves it on the floor after using it, the system will determine that it is not in its designated location. The device's camera takes a photo of the room, which the server analyzes. If it detects that the item has not been returned to its designated location, a notification will appear on the device saying, "Please return the red toy car to the center of the shelf." If the child complies and returns the item to the correct location, the device's camera will take another photo, and the server will confirm that it has been returned to the correct location. This will cause the server to generate a reward message, and the device will issue an audio notification saying, "Good job!"

[0700] The system of the present invention can improve the efficiency of household chores and reduce the burden on parents.

[0701] The processing flow will be explained below.

[0702] Step 1:

[0703] The user opens the app on their device to register their child's items in the system. The user selects the "Register Item" menu and enters the item name and its designated location. For example, the "blue block set" is set to the "top left of the toy box."

[0704] Step 2:

[0705] The device's camera is activated, and the user places the blue block set in front of the camera. The user presses the shutter button, and the device captures the image. The captured image data is temporarily stored in the device.

[0706] Step 3:

[0707] The device transmits the captured image data and fixed position information to the server, which receives the data and analyzes the image data and fixed position information.

[0708] Step 4:

[0709] The server stores the image data and location information in a database, and this storage operation records the item and its location in the system.

[0710] Step 5:

[0711] A child uses an object, for example, takes a blue block set out of a toy box and starts playing with it.

[0712] Step 6:

[0713] The device uses a camera to take pictures of the room at a pre-set frequency, for example, every hour, and sends the image data to a server.

[0714] Step 7:

[0715] The server analyzes the received image data and checks the current location of the item. If the blue block set is not in the "top left of the toy box," the server generates an alert.

[0716] Step 8:

[0717] The server generates an alert and sends it to the device, which then notifies the child of the alert through audio or a display on the screen. For example, it might say, "Put the blue block set back on the top left shelf of the toy box."

[0718] Step 9:

[0719] The user instructs the child to return the items to their correct locations, for example, returning the blue block set to the top left of the toy box.

[0720] Step 10:

[0721] The device then starts the camera again and takes a picture of the room. The captured image data is then sent to the server again.

[0722] Step 11:

[0723] The server analyzes the received image data and confirms that the blue block set has been returned to its original position. Based on the confirmation result, the server generates a reward message.

[0724] Step 12:

[0725] The server sends a reward message to the device, which then notifies the child of the message by voice or on-screen display. For example, "Great job! Amazing!" is displayed.

[0726] This system helps children develop the habit of returning items to their correct places, reducing the household chore burden on parents.

[0727] Example 1

[0728] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0729] Children often do not put away their belongings or household items in the correct place after playing, which creates a problem for parents, who have to spend a lot of time and effort tidying up for them. Furthermore, children have fewer opportunities to develop the habit of tidying up on their own. This increases the household chore burden and creates the problem of increased stress for parents.

[0730] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0731] In this invention, the server includes: means for a user to set a fixed position for an item; means for taking a picture of the item and its fixed position with a camera on the terminal and sending the image data to the server; means for the server to analyze the image data and store information about the item and its fixed position in a database; means for sending an alert notification to the terminal if the item has deviated from its fixed position; means for taking another picture with the camera when the user returns the item to the correct position and sending the image data to the server; and means for the server to analyze the image data, confirm that the item has been returned to its fixed position, and generate a reward message to notify the user. This automatically monitors the tidying up of items, reducing the housework burden on parents and enabling children to develop the habit of tidying up.

[0732] A "user" is a person, including an adult in the household, who operates the system and registers items and sets their locations.

[0733] "Items" refer to items to be organized, such as toys and household items used by children.

[0734] A "home location" is a specific place or area where an item is to be stored.

[0735] A "terminal" is a device that includes a computer device operated by a user and a camera mounted on that device.

[0736] A "camera" is a device that is installed in a terminal and takes pictures of items, their fixed positions, and the state of the room to generate image data.

[0737] The "server" is a computer system that analyzes image data sent from the terminal, stores the information in a database, and generates notifications and reward messages.

[0738] "Image data" refers to visual information of an object or the state of a room captured by a camera.

[0739] A "database" is an information management system located within a server that stores and manages information on analyzed items and their fixed locations.

[0740] An "alert notification" is a message sent to the user via the terminal to warn them when an item is not in its designated location.

[0741] A "reward message" is a message of praise or encouragement that is generated by the server and sent to the user via the terminal when the user returns the item to the correct location.

[0742] "Analysis" refers to the operation in which the server processes the received image data, identifies the location of the item, and makes appropriate decisions.

[0743] This invention is a system that helps children put things away in the correct place after use, reducing the household chore burden on parents. The system is mainly composed of the user, terminal, and server elements, and functions as follows.

[0744] First, the user uses the device's app to register an item. When registering an item, the user inputs the name of the item (e.g., a toy or household item) and its designated location. Next, the device's camera is used to take a photo of the item and its designated location, and the image data is sent to the server. For example, if the user wants to set the "blue block set" to the "top left of the toy box," the user takes a photo of that location with the device's camera.

[0745] The server receives image data sent from the terminal and analyzes the items and their fixed locations. The analysis results are saved in a database, and the item location information is managed as digital data. The server's analysis makes it possible to determine whether the items have been placed in the correct locations.

[0746] The device is equipped with a timer function that periodically takes photos of the room's condition, and the camera takes photos at specified times. The image data is then sent back to the server, which analyzes the received images. If an item is not in its registered location, the server generates an alert message and notifies the user via the device. For example, a specific message such as "Please return the blue block set to the top left of the toy box" is notified by voice or displayed on the screen.

[0747] When the user puts the items away in the correct place, the device's camera takes another photo of the situation and sends the image data to the server. The server analyzes this data and confirms that the items have been returned to their proper places. If successful, the server generates a reward message and notifies the user via the device. For example, a praising message such as "Good job! Amazing!" can be conveyed by voice or displayed on the screen, increasing the child's motivation to tidy up on their own.

[0748] As a concrete example, consider the case where a user registers a "red toy car" as "the center of the shelf." The user enters the name and designated location of the item into the device's app and takes a photo of the location with the device's camera. The device then sends the image data to the server, which analyzes it and stores it in a database. The device then periodically takes photos of the room, and the server analyzes the images to check whether the item is in its designated location. If the item is not in its designated location, an alert notification is sent to the user, prompting them to return the item to its correct location.

[0749] This system promotes tidiness and efficiency in the home, reduces the burden of housework on parents, and helps children develop the habit of tidying up on their own, which is expected to improve the overall home environment.

[0750] Examples of prompts to input to a generative AI model include:

[0751] "I've registered the red toy car in the center of the shelf, but what's the notification mechanism for when a child forgets to put it back?"

[0752] "Tell me the process that generates the reward message when the child returns the object to its correct location."

[0753] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0754] System program processing flow

[0755] Step 1: Registering and positioning items

[0756] Specific behavior:

[0757] 1. The user launches the app on their device and opens the item registration screen.

[0758] 2. The user selects the "Register Item" option and enters the name of the item (e.g., "Blue Block Set").

[0759] 3. The user inputs the item's fixed location (e.g., "top left of the toy box").

[0760] 4. The user takes a photo of the item and its designated location using the device's camera.

[0761] Input and Output:

[0762] Input: Name and location of the object, camera image.

[0763] Data processing: Associate item names and location information with camera images.

[0764] Output: Send image data to the server.

[0765] Step 2: Receiving and saving image data

[0766] 1. The server receives the image data sent from the terminal.

[0767] 2. The server analyzes the received image data and extracts information about the item and its location.

[0768] 3. The server stores the item and its location information in a database.

[0769] Input and Output:

[0770] Input: Image data sent from the device.

[0771] Data processing: Extraction of information about items and their locations through image analysis.

[0772] Output: Saves the information to the database and sends a message to the terminal confirming the success.

[0773] Step 3: Photograph the room

[0774] 1. The device takes photos of the room at specified intervals.

[0775] Input and Output:

[0776] Enter: Timer settings.

[0777] Data processing: Regular photography.

[0778] Output: Captured image data.

[0779] Step 4: Sending image data

[0780] 1. The device sends the captured image data to the server.

[0781] Input and Output:

[0782] Input: Captured image data.

[0783] Data processing: Preparing image data for transmission.

[0784] Output: Send image data to the server.

[0785] Step 5: Locate your items

[0786] 1. The server analyzes the received image data and checks whether the item is in the registered location.

[0787] 2. If the server determines that the item is not in place, it generates an alert message.

[0788] 3. The server sends an alert message to the device.

[0789] Input and Output:

[0790] Input: Image data sent to the server.

[0791] Data processing: Confirmation of item location through image analysis.

[0792] Output: Generates and sends an alert message.

[0793] Step 6: Alert Notifications

[0794] 1. The terminal notifies the user of the received alert message by voice or on-screen display.

[0795] Specific behavior:

[0796] For example, a message might appear on the device screen saying, "Please return the blue block set to the top left of the toy box."

[0797] Input and Output:

[0798] Input: The alert message sent by the server.

[0799] Data processing: Preparing messages for audio or screen display.

[0800] Output: Alert notification to the user.

[0801] Step 7: Check that the item is in place

[0802] 1. The user returns the item to its correct location.

[0803] 2. The device will take another photo of the situation with its camera.

[0804] 3. The device sends the captured image data to the server.

[0805] Input and Output:

[0806] Input: User actions, captured image data.

[0807] Data processing: Preparing image data for transmission.

[0808] Output: Send image data to the server.

[0809] Step 8: Reward Message Generation and Notification

[0810] 1. The server analyzes the received image data again and confirms that the item has been returned to its designated location.

[0811] 2. The server generates a reward message and sends it to the terminal.

[0812] 3. The terminal notifies the user of the reward message by voice or on-screen display.

[0813] Specific behavior:

[0814] For example, a message like "Great job! Awesome!" will appear on the device screen.

[0815] Input and Output:

[0816] Input: Re-captured image data.

[0817] Data processing: Image analysis and verification, reward message generation.

[0818] Output: Send reward message to device and notify user.

[0819] (Application example 1)

[0820] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0821] Currently, many homes and factories face the problem of items not being returned to their correct locations after use. This increases the household chore burden on parents at home and reduces production efficiency in factories. In particular, in industrial sites, improper management of tools and parts can increase the risk of production line shutdowns and accidents. To solve these problems, a system is needed that automatically manages and checks the location of items and notifies workers as necessary.

[0822] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0823] In this invention, the server includes a means for setting a fixed position for an item used by a user, a means for detecting the position of the item with a camera, a means for notifying when the item has deviated from the fixed position, and a means for detecting when the item has been returned to the fixed position and generating a reward message to notify the user. This automates the management of item positions in homes and factories, reducing the burden of housework on parents, improving factory production efficiency, and ensuring safety.

[0824] "User" refers to a person who uses an item in a home, factory, etc.

[0825] "Article" refers to a concrete object used for a specific purpose, such as a child's toy or a tool or part used in a factory.

[0826] "Home location" refers to a predetermined location to which an item should be returned before or after use.

[0827] The "means for setting" refers to a device or software that allows a user to register the fixed location of an item and save that information.

[0828] "Detection means" refers to technology for recognizing the location of an item using a device such as a camera and acquiring that information.

[0829] "Means for notifying" means an audio, visual, or other method for notifying a user if an item is displaced from its designated location.

[0830] "Means for generating and notifying reward messages" refers to devices or software that confirm that an item has been returned to its proper location and send a message of praise or encouragement to the user.

[0831] "Camera" refers to a photographing device for photographing the position of an item and acquiring image data thereof.

[0832] "Image data" refers to digital data containing image information captured by a camera.

[0833] "Server" refers to a computer system that receives and analyzes image data and generates necessary notifications and messages.

[0834] "Software" refers to programs or applications that run on a computer or other digital device.

[0835] "OpenCV" is a library for image analysis and is used to detect the position of objects.

[0836] "Notification Library" means a program library for generating and sending reward messages and alerts.

[0837] This invention is a system that supports returning used items to their correct positions using a terminal in homes and factories. The system includes means for setting a fixed position for the item, detecting it with a camera, notifying the user if the item has deviated from its fixed position, and detecting that the item has been returned to the correct position and generating and notifying the user of a reward message.

[0838] First, the user uses a terminal to register items and set their fixed locations. The user registers the items in the system and sets the fixed location for each item. This information is sent to the server and saved in the database. For example, in a factory, a "maintenance screwdriver set" is set to the "top right of the tool shelf."

[0839] Next, the camera periodically captures images of the facility and sends the image data to a server. The server analyzes the image data to check whether the items are in their designated locations. This is done using an image analysis library such as OpenCV. If, based on the analysis results, the items have not been returned to their designated locations, the server sends a notification to the terminal. For example, an alert will be displayed saying, "Please return the maintenance screwdriver set to the top right shelf of the tool shelf."

[0840] When the user returns the item to the correct position, the camera takes a picture again and sends the image data to the server. The server analyzes the image again and confirms that the item is in its correct position. Once this is confirmed, the server generates a reward message and sends a message such as "Good job!" to the device. A specific notification library is used to send the reward message.

[0841] As a specific example, if a factory worker leaves a "maintenance screwdriver set" on the floor after use, the camera captures the state, the server analyzes it and issues a notification, and when the worker returns the screwdriver set to its original position, the camera checks again, and the server generates and sends a reward message.

[0842] An example of a prompt to be input to the generative AI model is as follows:

[0843] "Design a support system for managing the location of items in a factory. This system will manage the fixed locations of certain tools and parts and use cameras to check whether the items are in the correct location. If an item deviates from its fixed location, it will send an alert and if it is returned to the correct location, it will generate a reward message to notify. The system will use OpenCV for image analysis and a notification library to send messages. It will also check the room status every minute and manage the items."

[0844] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0845] Step 1: Register items and set their locations on the device

[0846] The user uses a terminal to register items in the system and set the fixed location of each item. The input is the item name and fixed location information, and the output is a process of sending this information to the server to store it in the database.

[0847] Step 2: Capture a fixed position image with the camera

[0848] The camera on the device periodically captures images of the facility. The input is the image capture operation by the camera, and the output is the captured image data. This image data is used in the next processing step.

[0849] Step 3: Send image data to the server

[0850] The terminal sends the captured image data to the server. The input is the captured image data, and the output is the image data sent to the server.

[0851] Step 4: Analyzing the image data

[0852] The server analyzes the received image data and checks whether the item is in the set fixed position. The input is the image data and the fixed position database. The server uses OpenCV to identify the item's position and compares it with the fixed position information. The output is the result of determining whether the item is in the set fixed position.

[0853] Step 5: Detect deviations from home position

[0854] The server detects whether the item is out of place based on the image analysis results. The input is the result of image analysis. The output is to generate an alert notification if the item is out of place.

[0855] Step 6: Sending alert notifications

[0856] The server sends an alert notification to the terminal. The input is information about the item that has moved from its designated location, and the output is an alert notification to the user. For example, a message such as "Please return the maintenance screwdriver set to the top right of the tool shelf" is displayed on the terminal.

[0857] Step 7: Relocate items

[0858] When the user returns the item to the correct position, the camera takes a picture of the item again and sends the image data to the server. The input is the re-taken image data, and the output is the new image data sent to the server.

[0859] Step 8: Reanalysis and location verification

[0860] The server analyzes the received image data and checks whether the item is in its original position. The input is the new image data and the original position database. The output is the result of determining whether the item has been returned to its original position.

[0861] Step 9: Generate Reward Messages

[0862] If the server confirms that the item has been returned to the correct location, it generates a reward message. The input is the data that confirms that the item's location is correct. The output is a reward message (e.g., "Good job!").

[0863] Step 10: Sending Reward Messages

[0864] The server sends a reward message to the terminal. The input is the generated reward message, and the output is a reward notification to the user. The terminal notifies the user of this message by voice or on a screen.

[0865] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0866] The present invention relates to a system that helps parents return items used by children to their correct designated locations, thereby reducing the burden of housework on parents. Furthermore, the present invention incorporates an emotion engine that recognizes the user's emotions and adjusts reward messages based on the emotions. Specifically, the system includes means for setting a designated location for an item, detecting the location of the item with a camera, notifying the user if the item has moved away from its designated location, detecting that the item has been returned to its designated location, and generating and notifying the user of a reward message.

[0867] First, the user registers the child's items in the system through a terminal. The user inputs the name and fixed location of the item, and the fixed location information is input into the system. For example, the user sets the "blue block set" to the "top left of the toy box." The terminal uses a camera to take a photo of the blue block set and its fixed location, and sends the image data to the server. The server analyzes this image data and saves information about the item and its fixed location in a database.

[0868] Next, after the child has finished using an item, the device's camera periodically takes pictures of the room to check whether the item has been returned to its designated location. This image data is sent to the server. The server analyzes the received image data, and if an item has been moved from its designated location, it generates an alert and sends it to the device. The device notifies the child of the alert by voice or by displaying it on the screen. For example, a message such as "Please return the blue block set to the top left of the toy box" may be displayed.

[0869] Furthermore, the system incorporates an emotion engine to recognize the user's emotions. This function records the user's emotions during the operation of locating and returning the item to its original position, and analyzes the emotion data. For example, the system uses a camera or microphone to analyze the user's facial expressions and tone of voice to detect the current emotion.

[0870] If the user returns the item to its correct location, the device takes another photo with the camera and sends the image data to the server. The server analyzes the image data and, once it confirms that the item has been returned to its proper place, adjusts the reward message based on the emotion detected by the emotion engine. For example, if the child is excited, the message displayed might say, "Good job! That's amazing!", whereas if the child is calm, the message displayed might say, "Good job, let's try harder next time."

[0871] Specific examples are as follows:

[0872] Suppose a user registers a "red toy car" in the "center of the shelf." The user opens the app on their device and takes a photo of the item and its designated location to register it. If a child plays with the red toy car and then leaves it on the floor after use, the device's camera takes a photo of the room, which the server analyzes. If the device detects that the item has not been returned to its designated location, a notification appears on the device saying, "Please return the red toy car to the center of the shelf." If the user complies and returns the item to the correct location, the device's camera takes another photo, and the server confirms that it has been returned to the correct location. The emotion engine then analyzes the user's emotions and generates an appropriate reward message, such as "Good job! Amazing!", which the device notifies them by voice or on-screen display.

[0873] This allows the system to develop the habit of returning items to their correct places, reducing the household chore burden on parents and providing effective education by providing feedback based on the user's emotions.

[0874] The processing flow will be explained below.

[0875] Step 1:

[0876] The user opens the app on their device and prepares to register an item in the system. The user selects the "Register Item" menu and enters the item name and its designated location. For example, the user sets the "blue block set" to the "top left of the toy box."

[0877] Step 2:

[0878] The device activates the camera, and the user places the blue block set in front of the camera. The user presses the shutter button, and the device captures the image. The captured image data is temporarily stored in the device.

[0879] Step 3:

[0880] The device transmits the captured image data and fixed position information to the server, which receives the data and analyzes the image data and fixed position information.

[0881] Step 4:

[0882] The server stores the image data and location information in a database, and this storage operation records the item and its location in the system.

[0883] Step 5:

[0884] A child uses an object, for example, takes a blue block set out of a toy box and starts playing with it.

[0885] Step 6:

[0886] The device uses a camera to take pictures of the room at a pre-set frequency, for example, every hour, and sends the image data to a server.

[0887] Step 7:

[0888] The server analyzes the received image data and checks the current location of the item. If the blue block set is not in the "top left of the toy box," the server generates an alert.

[0889] Step 8:

[0890] The server generates an alert and sends it to the device, which then notifies the child of the alert through audio or a display on the screen. For example, it might say, "Put the blue block set back on the top left shelf of the toy box."

[0891] Step 9:

[0892] The emotion engine analyzes the user's facial expressions and tone of voice to recognize their current emotion. The emotion data is processed internally and referenced when generating reward messages.

[0893] Step 10:

[0894] The user instructs the child to return the items to their correct locations, for example, returning the blue block set to the top left of the toy box.

[0895] Step 11:

[0896] The device then starts the camera again and takes a picture of the room. The captured image data is then sent to the server again.

[0897] Step 12:

[0898] The server analyzes the received image data and verifies that the blue block set has been returned to its original position. Based on the verification result, the server adjusts the reward message based on the emotion detected by the emotion engine.

[0899] Step 13:

[0900] The server sends a reward message to the device. The reward message is tailored to the child's current emotions. For example, messages such as "Good job! Amazing!" or "Good job, let's try harder next time" are generated.

[0901] Step 14:

[0902] The device will notify the child with a message via voice or screen display, allowing the child to receive appropriate feedback and form the habit of returning items to their correct place.

[0903] The above are the specific processing steps of the system.

[0904] Example 2

[0905] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0906] Many parents are forced to do additional housework to organize items because children are unable to smoothly return them to their correct locations. Furthermore, reward messages for children are standardized and lack appropriate feedback tailored to the situation. To solve these problems, a system is needed that automatically checks where items are returned and generates reward messages tailored to the user's emotions.

[0907] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0908] In this invention, the server includes means for setting a fixed position for an item used by a user, means for detecting the position of the item with a camera, means for notifying the user when the item has deviated from its fixed position, means for recognizing the user's emotion using an emotion engine and adjusting a reward message based on the emotion, and means for detecting when the item has been returned to its fixed position and generating and notifying the user of the reward message. This allows children to develop the habit of returning items to their correct positions, reduces the housework burden on parents, and enables effective feedback according to the user's emotions.

[0909] "User" refers to a person who uses the system to register and manage items.

[0910] "Item" refers to an object owned by a user that is intended to be registered in the system and returned to a designated location.

[0911] "Home location" refers to a predetermined place where an item should be returned after use.

[0912] "Camera" refers to a device that captures the current location of an item and the state of the room, and acquires the image data.

[0913] "Server" refers to the central processing unit that analyzes image data, operates the emotion engine, locates and notifies items, and generates reward messages.

[0914] "Emotion engine" refers to software or a system for recognizing a user's emotions and tailoring reward messages based on those emotions.

[0915] "Reward message" refers to a message generated in response to returning an item to its proper location that encourages or praises the user's behavior.

[0916] "Notification" refers to a means of communication in the form of audio or screen display to notify the user if an item has not been returned to its proper location or to notify the user of a reward message.

[0917] "Image data" refers to the digital information of images taken by a camera, and is used to analyze the location of an item or the user's emotions.

[0918] The present invention relates to a system that assists parents in returning items used by children to their designated locations, thereby reducing the burden of housework. Furthermore, the present invention incorporates an emotion engine that recognizes the user's emotions and adjusts reward messages based on the emotions. Specific embodiments of the system will be described below.

[0919] This system is primarily composed of three elements: the user, the terminal, and the server. First, the user registers and manages items through the terminal. The terminal provides an interface for inputting the name of the item and its designated location. When the user sets an item and its designated location, for example, by setting a "blue block set" to the "top left of the toy box," the terminal's camera takes a photo of the item and its designated location and sends the image data to the server. The server analyzes the received image data and stores the information about the item and its designated location in a database. An image recognition algorithm is used in this process.

[0920] Next, the process begins to check whether the items are returned to their designated locations after the child has used them. The device's camera periodically takes pictures of the room and sends this image data to the server. The server analyzes the received image data and checks whether the items are in their designated locations. If the items are not returned to their designated locations, the server generates an alert and sends notification data to the device. For example, the server could generate an alert stating, "The blue block set is not in its designated location," and the device will notify the user by voice or by displaying a message on its screen.

[0921] The system also incorporates an emotion engine that recognizes the user's emotions when returning an item to its proper place. When the user returns an item to its correct location, the device's camera and microphone record the user's facial expressions and tone of voice, and send this as emotional data to the server. The server analyzes the received emotional data and classifies the user's current emotional state as "joy," "excitement," "calm," or something similar.

[0922] When the user returns the item to its correct location, the device again takes a photo of the room with its camera and sends the image data to the server. The server analyzes the image data and, once it confirms that the item has been returned to its proper location, adjusts the reward message based on the emotion detected by the emotion engine. For example, if the child is excited, the message "Good job! That's amazing!" is generated, whereas if the child is calm, the message "Good job, let's try harder next time" is generated. Finally, the device notifies the user of the generated reward message by voice or on-screen display.

[0923] As a concrete example, consider the case where a user registers a "red toy car" as "the center of the shelf." The user opens the app on their device and takes a photo of the item and its designated location to register it. If a child plays with the red toy car and then leaves it on the floor after use, the device's camera takes a photo of the room and the server analyzes the image data. If the device detects that the item has not been returned to its designated location, a notification appears on the device saying, "Please return the red toy car to the center of the shelf." If the user complies and returns the item to the correct location, the device's camera takes another photo, and the server confirms that it has been returned to the correct location. The emotion engine then analyzes the user's emotions and generates an appropriate reward message, such as "Good job! Amazing!", which the device notifies them by voice or on-screen display.

[0924] An example of a prompt to be input to a generative AI model is, "Please describe the workflow of a system that helps children return toys to their proper places after playing with them. Please provide a detailed process, including user input, confirmation of the item's proper place, sentiment analysis, and reward message generation."

[0925] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0926] Step 1:

[0927] Product registration

[0928] The user opens the app on their device and inputs the items and their designated locations. For example, the "blue block set" is set to the "top left of the toy box."

[0929] Input: Item name and location information

[0930] The device uses the camera to take a photo of the item and the set fixed position, and sends the captured image data to the server.

[0931] Output: Image data

[0932] Specific operation: The app screen displays the instruction "Take a photo of the blue block set," and the user operates the device's camera to take a photo.

[0933] Step 2:

[0934] Data storage and analysis

[0935] The server analyzes the received image data and stores information about the item and its fixed location in a database.

[0936] Input: Image data

[0937] The server uses image recognition algorithms to identify the items and their locations and stores this information in a database.

[0938] Output: Database entry for item and location

[0939] Specific operation: As the server analyzes the image data, a progress bar will be displayed in the device app, and a confirmation notification will be sent to the user when the analysis is complete.

[0940] Step 3:

[0941] Checking the location of items

[0942] After the child uses the item, the terminal periodically takes pictures of the room and sends the image data to the server.

[0943] Input: Image data of the room condition

[0944] The server analyzes the received image data and checks whether the item is in the set fixed position.

[0945] Output: Item location status (in place or not)

[0946] Specific operation: The device displays a notification saying "Taking a photo" at the time of shooting, and the captured image is sent to the server.

[0947] Step 4:

[0948] Alert Generation

[0949] If the item is not returned to its original location, the server generates an alert and sends notification data to the terminal.

[0950] Input: Item location status

[0951] The server generates an alert message and sends it to the terminal.

[0952] Output: Alert message

[0953] Specific actions: For example, an alert saying "The blue block set is not in its proper place" will be displayed on the device, and a voice message will be played saying "Please return the blue block set to the top left of the toy box."

[0954] Step 5:

[0955] sentiment analysis

[0956] When the user returns the item to its correct location, the device's camera and microphone record the user's facial expressions and tone of voice, and send this data to the server as emotional data.

[0957] Input: User's facial expression data, voice data

[0958] The server analyzes the received emotional data and identifies the current emotional state.

[0959] Output: User's emotional state (e.g., happy, excited, calm)

[0960] What it does: The camera indicator light will turn on to let the user know that sentiment analysis is taking place.

[0961] Step 6:

[0962] Reward message generation and notification

[0963] The terminal then takes another photo with the camera to check if the item has been returned to the correct position, and sends the image data to the server.

[0964] Input: Image data of the returned item

[0965] The server analyzes the image data and, once it determines that the item has been returned to its proper location, adjusts the reward message based on the emotion detected by the emotion engine.

[0966] Output: Reward message

[0967] Specific behavior: For example, if a child is excited, the device will generate a message saying "Good job! That's amazing!" and notify the child by voice or displaying a message on the screen.

[0968] (Application example 2)

[0969] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0970] In traditional factories, parts placement errors and inaccurate storage were common, resulting in problems such as reduced work efficiency and mistakes. Preventing these mistakes required human supervision and alerts, which required a significant amount of effort and time. Furthermore, the lack of a system for properly understanding employees' emotions and motivation and providing feedback accordingly hindered improvements in work efficiency.

[0971] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes a means for setting a fixed position for items used by children, a means for detecting the position of the item with a camera, a means for notifying when the item has deviated from its fixed position, a means for detecting when the item has been returned to its fixed position and for generating and notifying a reward message, and a means for recognizing the emotion of the worker and adjusting the reward message based on the emotion. This makes it possible to prevent mistakes in part placement and a decrease in work efficiency, as well as to appropriately grasp the emotion of the worker and provide feedback accordingly.

[0972] The "means for setting a fixed location for an item used by a child" is a function that allows a user to set a predetermined location within the system where a particular item should be stored or placed.

[0973] The "means for detecting the position of an item using a camera" is a function for capturing the current position of an item using a camera and transmitting the position information to the system.

[0974] The "means for notifying when an article deviates from its fixed position" is a function for sending an alert to the user when an article deviates from its set fixed position.

[0975] The "means for detecting that an item has been returned to its designated location and generating and notifying a reward message" is a function for confirming that an item has been returned to the correct location and sending a reward message to the user based on that.

[0976] "Means for recognizing the emotions of workers and adjusting reward messages based on those emotions" refers to a function that uses sensors such as cameras and microphones to detect the emotional state of workers and adjust the content and tone of reward messages to match those emotions.

[0977] A "camera that monitors the usage status of an item" is a camera that monitors the location and status of an item while it is in use, and sends this information to a server as image data.

[0978] "Means for analyzing image data received by the server and confirming whether the item is in the set fixed position" is a function for analyzing image data sent to the server and confirming whether the item is in the set fixed position.

[0979] This invention relates to a system that aims to accurately manage the location of parts in a factory and improve work efficiency. Specifically, the system supports workers wearing smart glasses, confirms that items are returned to their designated locations, and provides appropriate feedback according to the worker's emotions.

[0980] First, we will explain the process of setting the fixed positions of items. The user uses smart glasses to register the fixed positions of each part in the system. Specifically, the user takes a photo of the part and its fixed position using the smart glasses' camera and sends the image data to the server. At that time, the user enters the part's name and fixed position, and the server stores this information in a database.

[0981] Next, we will explain how to monitor the status of parts when workers use them during work. The camera in the smart glasses periodically captures images of the workspace and sends the image data to a server. The server analyzes the received image data and generates a notification if an item deviates from its designated position. As a specific example, a message such as "Part A has deviated from its designated position. Please return it to the correct position" is displayed on the smart glasses.

[0982] Once the worker returns the item to its correct location, the smart glasses' camera takes another photo and sends the image data to the server. The server analyzes the image data and confirms that the item has been returned to its proper place. At this time, an emotion engine is used to detect the worker's emotions and tailor reward messages based on their emotions. For example, if the worker is focused, a short message like "Good job!" is displayed, whereas if the worker is nervous, a message like "Great job! Try again next time" is displayed.

[0983] For emotion recognition, the system uses a camera and microphone to analyze facial expressions and vocal tone, and employs an emotion recognition model. It processes image data using OpenCV and uses object detection algorithms such as HOGDescriptor to identify the location of objects. It also utilizes an external emotion recognition library.

[0984] Examples of prompts include:

[0985] "Detect the location of part A and notify it to return to its original position."

[0986] "Recognize workers' emotions and provide appropriate reward messages."

[0987] This system can prevent mistakes in part placement and improve work efficiency, as well as promote worker motivation and provide a more effective work environment.

[0988] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0989] Step 1:

[0990] The user registers the fixed location of an item. The user takes a photo of the item and its fixed location through the smart glasses and inputs the item's name and fixed location information. The camera in the smart glasses captures this and the image data is sent to the server. The input data is the item's name and location data, and the output is the registered fixed location information stored on the server.

[0991] Step 2:

[0992] The server analyzes the received image data and stores the fixed location information of the items in a database. Specifically, OpenCV is used to extract the features of the items in the image and store them in the database. The input data is image data and location information, and the output is the fixed location information of the items stored in the database.

[0993] Step 3:

[0994] The camera in the smart glasses periodically captures images of the workspace and sends the captured image data to a server. The input data is the periodically captured image data, and the output is the image data sent to the server.

[0995] Step 4:

[0996] The server analyzes the received image data and checks whether the item is in its designated location. Specifically, it uses OpenCV and HOGDescriptor to detect the item's location and compares it with the database's designated location information. The input data is the captured image data, and the output is the result of determining whether the item is in its designated location.

[0997] Step 5:

[0998] If an item is out of place, the server generates an alert message and notifies the smart glasses. Specifically, it displays a message such as "Part A is out of place. Please return it to the correct position." The input data is the judgment result, and the output is the alert message displayed on the smart glasses.

[0999] Step 6:

[1000] When the worker returns the item to the correct position, the camera in the smart glasses takes a picture again and sends the image data to the server. The input data is the recaptured image data, and the output is the image data sent to the server.

[1001] Step 7:

[1002] The server analyzes the image data again to confirm that the item has been returned to the correct location. If confirmed, it uses an emotion engine to recognize the worker's emotion and generates a reward message based on that emotion. The input data is the received image data and emotion data again, and the output is a reward message based on the emotion.

[1003] Step 8:

[1004] The server sends the generated reward message to the smart glasses and notifies the worker. Specifically, a message such as "Good job! Try your best next time" is displayed. The input data is the generated reward message, and the output is the reward message displayed on the smart glasses.

[1005] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1006] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1007] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1008] [Fourth embodiment]

[1009] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1010] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1011] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1012] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1013] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1014] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1015] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1016] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1017] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1018] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1019] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1020] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1021] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1022] The present invention provides a system for helping children put away items in their correct places after use, thereby reducing the burden of housework on parents. The system includes means for setting fixed positions for items, detecting the positions of the items with a camera, notifying the parent if the item has moved away from its fixed position, detecting that the item has been returned to its fixed position, and generating a reward message to notify the parent.

[1023] Specifically, the user registers items through a terminal. The user registers children's toys and household items in the system and sets fixed locations for the items. For example, the user sets the "blue block set" to the "top left of the toy box." The terminal uses a camera to take a photo of the blue block set and its fixed location, and sends the image data to the server. The server analyzes this image data and stores information about the items and their fixed locations in a database.

[1024] If a child uses an item but does not return it to its registered location, the system detects this and notifies the user. The device's camera periodically takes pictures of the room and sends the image data to the server. The server analyzes the received image data and, if an item is not in its designated location, sends an alert to the device. The device notifies the user by voice or on the screen, saying, "Please return the blue block set to the top left of the toy box."

[1025] When the user returns the item to the correct location, the device takes a photo with the camera again and sends the image data to the server. The server analyzes the image data and, if it confirms that the item has been returned to its designated location, generates a reward message. For example, the server may generate a message such as "Good job! That's great!" and send it to the device. The device then notifies the user of the message by voice or by displaying it on its screen.

[1026] Specific examples are as follows:

[1027] Suppose a user registers a "red toy car" as "the center of the shelf." The user opens the app on their device and takes a photo of the item and its designated location to register it. If a child plays with the red toy car and leaves it on the floor after using it, the system will determine that it is not in its designated location. The device's camera takes a photo of the room, which the server analyzes. If it detects that the item has not been returned to its designated location, a notification will appear on the device saying, "Please return the red toy car to the center of the shelf." If the child complies and returns the item to the correct location, the device's camera will take another photo, and the server will confirm that it has been returned to the correct location. This will cause the server to generate a reward message, and the device will issue an audio notification saying, "Good job!"

[1028] The system of the present invention can improve the efficiency of household chores and reduce the burden on parents.

[1029] The processing flow will be explained below.

[1030] Step 1:

[1031] The user opens the app on their device to register their child's items in the system. The user selects the "Register Item" menu and enters the item name and its designated location. For example, the "blue block set" is set to the "top left of the toy box."

[1032] Step 2:

[1033] The device's camera is activated, and the user places the blue block set in front of the camera. The user presses the shutter button, and the device captures the image. The captured image data is temporarily stored in the device.

[1034] Step 3:

[1035] The device transmits the captured image data and fixed position information to the server, which receives the data and analyzes the image data and fixed position information.

[1036] Step 4:

[1037] The server stores the image data and location information in a database, and this storage operation records the item and its location in the system.

[1038] Step 5:

[1039] A child uses an object, for example, takes a blue block set out of a toy box and starts playing with it.

[1040] Step 6:

[1041] The device uses a camera to take pictures of the room at a pre-set frequency, for example, every hour, and sends the image data to a server.

[1042] Step 7:

[1043] The server analyzes the received image data and checks the current location of the item. If the blue block set is not in the "top left of the toy box," the server generates an alert.

[1044] Step 8:

[1045] The server generates an alert and sends it to the device, which then notifies the child of the alert through audio or a display on the screen. For example, it might say, "Put the blue block set back on the top left shelf of the toy box."

[1046] Step 9:

[1047] The user instructs the child to return the items to their correct locations, for example, returning the blue block set to the top left of the toy box.

[1048] Step 10:

[1049] The device then starts the camera again and takes a picture of the room. The captured image data is then sent to the server again.

[1050] Step 11:

[1051] The server analyzes the received image data and confirms that the blue block set has been returned to its original position. Based on the confirmation result, the server generates a reward message.

[1052] Step 12:

[1053] The server sends a reward message to the device, which then notifies the child of the message by voice or on-screen display. For example, "Great job! Amazing!" is displayed.

[1054] This system helps children develop the habit of returning items to their correct places, reducing the household chore burden on parents.

[1055] Example 1

[1056] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1057] Children often do not put away their belongings or household items in the correct place after playing, which creates a problem for parents, who have to spend a lot of time and effort tidying up for them. Furthermore, children have fewer opportunities to develop the habit of tidying up on their own. This increases the household chore burden and creates the problem of increased stress for parents.

[1058] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1059] In this invention, the server includes: means for a user to set a fixed position for an item; means for taking a picture of the item and its fixed position with a camera on the terminal and sending the image data to the server; means for the server to analyze the image data and store information about the item and its fixed position in a database; means for sending an alert notification to the terminal if the item has deviated from its fixed position; means for taking another picture with the camera when the user returns the item to the correct position and sending the image data to the server; and means for the server to analyze the image data, confirm that the item has been returned to its fixed position, and generate a reward message to notify the user. This automatically monitors the tidying up of items, reducing the housework burden on parents and enabling children to develop the habit of tidying up.

[1060] A "user" is a person, including an adult in the household, who operates the system and registers items and sets their locations.

[1061] "Items" refer to items to be organized, such as toys and household items used by children.

[1062] A "home location" is a specific place or area where an item is to be stored.

[1063] A "terminal" is a device that includes a computer device operated by a user and a camera mounted on that device.

[1064] A "camera" is a device that is installed in a terminal and takes pictures of items, their fixed positions, and the state of the room to generate image data.

[1065] The "server" is a computer system that analyzes image data sent from the terminal, stores the information in a database, and generates notifications and reward messages.

[1066] "Image data" refers to visual information of an object or the state of a room captured by a camera.

[1067] A "database" is an information management system located within a server that stores and manages information on analyzed items and their fixed locations.

[1068] An "alert notification" is a message sent to the user via the terminal to warn them when an item is not in its designated location.

[1069] A "reward message" is a message of praise or encouragement that is generated by the server and sent to the user via the terminal when the user returns the item to the correct location.

[1070] "Analysis" refers to the operation in which the server processes the received image data, identifies the location of the item, and makes appropriate decisions.

[1071] This invention is a system that helps children put things away in the correct place after use, reducing the household chore burden on parents. The system is mainly composed of the user, terminal, and server elements, and functions as follows.

[1072] First, the user uses the device's app to register an item. When registering an item, the user inputs the name of the item (e.g., a toy or household item) and its designated location. Next, the device's camera is used to take a photo of the item and its designated location, and the image data is sent to the server. For example, if the user wants to set the "blue block set" to the "top left of the toy box," the user takes a photo of that location with the device's camera.

[1073] The server receives image data sent from the terminal and analyzes the items and their fixed locations. The analysis results are saved in a database, and the item location information is managed as digital data. The server's analysis makes it possible to determine whether the items have been placed in the correct locations.

[1074] The device is equipped with a timer function that periodically takes photos of the room's condition, and the camera takes photos at specified times. The image data is then sent back to the server, which analyzes the received images. If an item is not in its registered location, the server generates an alert message and notifies the user via the device. For example, a specific message such as "Please return the blue block set to the top left of the toy box" is notified by voice or displayed on the screen.

[1075] When the user puts the items away in the correct place, the device's camera takes another photo of the situation and sends the image data to the server. The server analyzes this data and confirms that the items have been returned to their proper places. If successful, the server generates a reward message and notifies the user via the device. For example, a praising message such as "Good job! Amazing!" can be conveyed by voice or displayed on the screen, increasing the child's motivation to tidy up on their own.

[1076] As a concrete example, consider the case where a user registers a "red toy car" as "the center of the shelf." The user enters the name and designated location of the item into the device's app and takes a photo of the location with the device's camera. The device then sends the image data to the server, which analyzes it and stores it in a database. The device then periodically takes photos of the room, and the server analyzes the images to check whether the item is in its designated location. If the item is not in its designated location, an alert notification is sent to the user, prompting them to return the item to its correct location.

[1077] This system promotes tidiness and efficiency in the home, reduces the burden of housework on parents, and helps children develop the habit of tidying up on their own, which is expected to improve the overall home environment.

[1078] Examples of prompts to input to a generative AI model include:

[1079] "I've registered the red toy car in the center of the shelf, but what's the notification mechanism for when a child forgets to put it back?"

[1080] "Tell me the process that generates the reward message when the child returns the object to its correct location."

[1081] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1082] System program processing flow

[1083] Step 1: Registering and positioning items

[1084] Specific behavior:

[1085] 1. The user launches the app on their device and opens the item registration screen.

[1086] 2. The user selects the "Register Item" option and enters the name of the item (e.g., "Blue Block Set").

[1087] 3. The user inputs the item's fixed location (e.g., "top left of the toy box").

[1088] 4. The user takes a photo of the item and its designated location using the device's camera.

[1089] Input and Output:

[1090] Input: Name and location of the object, camera image.

[1091] Data processing: Associate item names and location information with camera images.

[1092] Output: Send image data to the server.

[1093] Step 2: Receiving and saving image data

[1094] 1. The server receives the image data sent from the terminal.

[1095] 2. The server analyzes the received image data and extracts information about the item and its location.

[1096] 3. The server stores the item and its location information in a database.

[1097] Input and Output:

[1098] Input: Image data sent from the device.

[1099] Data processing: Extraction of information about items and their locations through image analysis.

[1100] Output: Saves the information to the database and sends a message to the terminal confirming the success.

[1101] Step 3: Photograph the room

[1102] 1. The device takes photos of the room at specified intervals.

[1103] Input and Output:

[1104] Enter: Timer settings.

[1105] Data processing: Regular photography.

[1106] Output: Captured image data.

[1107] Step 4: Sending image data

[1108] 1. The device sends the captured image data to the server.

[1109] Input and Output:

[1110] Input: Captured image data.

[1111] Data processing: Preparing image data for transmission.

[1112] Output: Send image data to the server.

[1113] Step 5: Locate your items

[1114] 1. The server analyzes the received image data and checks whether the item is in the registered location.

[1115] 2. If the server determines that the item is not in place, it generates an alert message.

[1116] 3. The server sends an alert message to the device.

[1117] Input and Output:

[1118] Input: Image data sent to the server.

[1119] Data processing: Confirmation of item location through image analysis.

[1120] Output: Generates and sends an alert message.

[1121] Step 6: Alert Notifications

[1122] 1. The terminal notifies the user of the received alert message by voice or on-screen display.

[1123] Specific behavior:

[1124] For example, a message might appear on the device screen saying, "Please return the blue block set to the top left of the toy box."

[1125] Input and Output:

[1126] Input: The alert message sent by the server.

[1127] Data processing: Preparing messages for audio or screen display.

[1128] Output: Alert notification to the user.

[1129] Step 7: Check that the item is in place

[1130] 1. The user returns the item to its correct location.

[1131] 2. The device will take another photo of the situation with its camera.

[1132] 3. The device sends the captured image data to the server.

[1133] Input and Output:

[1134] Input: User actions, captured image data.

[1135] Data processing: Preparing image data for transmission.

[1136] Output: Send image data to the server.

[1137] Step 8: Reward Message Generation and Notification

[1138] 1. The server analyzes the received image data again and confirms that the item has been returned to its designated location.

[1139] 2. The server generates a reward message and sends it to the terminal.

[1140] 3. The terminal notifies the user of the reward message by voice or on-screen display.

[1141] Specific behavior:

[1142] For example, a message like "Great job! Awesome!" will appear on the device screen.

[1143] Input and Output:

[1144] Input: Re-captured image data.

[1145] Data processing: Image analysis and verification, reward message generation.

[1146] Output: Send reward message to device and notify user.

[1147] (Application example 1)

[1148] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1149] Currently, many homes and factories face the problem of items not being returned to their correct locations after use. This increases the household chore burden on parents at home and reduces production efficiency in factories. In particular, in industrial sites, improper management of tools and parts can increase the risk of production line shutdowns and accidents. To solve these problems, a system is needed that automatically manages and checks the location of items and notifies workers as necessary.

[1150] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1151] In this invention, the server includes a means for setting a fixed position for an item used by a user, a means for detecting the position of the item with a camera, a means for notifying when the item has deviated from the fixed position, and a means for detecting when the item has been returned to the fixed position and generating a reward message to notify the user. This automates the management of item positions in homes and factories, reducing the burden of housework on parents, improving factory production efficiency, and ensuring safety.

[1152] "User" refers to a person who uses an item in a home, factory, etc.

[1153] "Article" refers to a concrete object used for a specific purpose, such as a child's toy or a tool or part used in a factory.

[1154] "Home location" refers to a predetermined location to which an item should be returned before or after use.

[1155] The "means for setting" refers to a device or software that allows a user to register the fixed location of an item and save that information.

[1156] "Detection means" refers to technology for recognizing the location of an item using a device such as a camera and acquiring that information.

[1157] "Means for notifying" means an audio, visual, or other method for notifying a user if an item is displaced from its designated location.

[1158] "Means for generating and notifying reward messages" refers to devices or software that confirm that an item has been returned to its proper location and send a message of praise or encouragement to the user.

[1159] "Camera" refers to a photographing device for photographing the position of an item and acquiring image data thereof.

[1160] "Image data" refers to digital data containing image information captured by a camera.

[1161] "Server" refers to a computer system that receives and analyzes image data and generates necessary notifications and messages.

[1162] "Software" refers to programs or applications that run on a computer or other digital device.

[1163] "OpenCV" is a library for image analysis and is used to detect the position of objects.

[1164] "Notification Library" means a program library for generating and sending reward messages and alerts.

[1165] This invention is a system that supports returning used items to their correct positions using a terminal in homes and factories. The system includes means for setting a fixed position for the item, detecting it with a camera, notifying the user if the item has deviated from its fixed position, and detecting that the item has been returned to the correct position and generating and notifying the user of a reward message.

[1166] First, the user uses a terminal to register items and set their fixed locations. The user registers the items in the system and sets the fixed location for each item. This information is sent to the server and saved in the database. For example, in a factory, a "maintenance screwdriver set" is set to the "top right of the tool shelf."

[1167] Next, the camera periodically captures images of the facility and sends the image data to a server. The server analyzes the image data to check whether the items are in their designated locations. This is done using an image analysis library such as OpenCV. If, based on the analysis results, the items have not been returned to their designated locations, the server sends a notification to the terminal. For example, an alert will be displayed saying, "Please return the maintenance screwdriver set to the top right shelf of the tool shelf."

[1168] When the user returns the item to the correct position, the camera takes a picture again and sends the image data to the server. The server analyzes the image again and confirms that the item is in its correct position. Once this is confirmed, the server generates a reward message and sends a message such as "Good job!" to the device. A specific notification library is used to send the reward message.

[1169] As a specific example, if a factory worker leaves a "maintenance screwdriver set" on the floor after use, the camera captures the state, the server analyzes it and issues a notification, and when the worker returns the screwdriver set to its original position, the camera checks again, and the server generates and sends a reward message.

[1170] An example of a prompt to be input to the generative AI model is as follows:

[1171] "Design a support system for managing the location of items in a factory. This system will manage the fixed locations of certain tools and parts and use cameras to check whether the items are in the correct location. If an item deviates from its fixed location, it will send an alert and if it is returned to the correct location, it will generate a reward message to notify. The system will use OpenCV for image analysis and a notification library to send messages. It will also check the room status every minute and manage the items."

[1172] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1173] Step 1: Register items and set their locations on the device

[1174] The user uses a terminal to register items in the system and set the fixed location of each item. The input is the item name and fixed location information, and the output is a process of sending this information to the server to store it in the database.

[1175] Step 2: Capture a fixed position image with the camera

[1176] The camera on the device periodically captures images of the facility. The input is the image capture operation by the camera, and the output is the captured image data. This image data is used in the next processing step.

[1177] Step 3: Send image data to the server

[1178] The terminal sends the captured image data to the server. The input is the captured image data, and the output is the image data sent to the server.

[1179] Step 4: Analyzing the image data

[1180] The server analyzes the received image data and checks whether the item is in the set fixed position. The input is the image data and the fixed position database. The server uses OpenCV to identify the item's position and compares it with the fixed position information. The output is the result of determining whether the item is in the set fixed position.

[1181] Step 5: Detect deviations from home position

[1182] The server detects whether the item is out of place based on the image analysis results. The input is the result of image analysis. The output is to generate an alert notification if the item is out of place.

[1183] Step 6: Sending alert notifications

[1184] The server sends an alert notification to the terminal. The input is information about the item that has moved from its designated location, and the output is an alert notification to the user. For example, a message such as "Please return the maintenance screwdriver set to the top right of the tool shelf" is displayed on the terminal.

[1185] Step 7: Relocate items

[1186] When the user returns the item to the correct position, the camera takes a picture of the item again and sends the image data to the server. The input is the re-taken image data, and the output is the new image data sent to the server.

[1187] Step 8: Reanalysis and location verification

[1188] The server analyzes the received image data and checks whether the item is in its original position. The input is the new image data and the original position database. The output is the result of determining whether the item has been returned to its original position.

[1189] Step 9: Generate Reward Messages

[1190] If the server confirms that the item has been returned to the correct location, it generates a reward message. The input is the data that confirms that the item's location is correct. The output is a reward message (e.g., "Good job!").

[1191] Step 10: Sending Reward Messages

[1192] The server sends a reward message to the terminal. The input is the generated reward message, and the output is a reward notification to the user. The terminal notifies the user of this message by voice or on a screen.

[1193] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1194] The present invention relates to a system that helps parents return items used by children to their correct designated locations, thereby reducing the burden of housework on parents. Furthermore, the present invention incorporates an emotion engine that recognizes the user's emotions and adjusts reward messages based on the emotions. Specifically, the system includes means for setting a designated location for an item, detecting the location of the item with a camera, notifying the user if the item has moved away from its designated location, detecting that the item has been returned to its designated location, and generating and notifying the user of a reward message.

[1195] First, the user registers the child's items in the system through a terminal. The user inputs the name and fixed location of the item, and the fixed location information is input into the system. For example, the user sets the "blue block set" to the "top left of the toy box." The terminal uses a camera to take a photo of the blue block set and its fixed location, and sends the image data to the server. The server analyzes this image data and saves information about the item and its fixed location in a database.

[1196] Next, after the child has finished using an item, the device's camera periodically takes pictures of the room to check whether the item has been returned to its designated location. This image data is sent to the server. The server analyzes the received image data, and if an item has been moved from its designated location, it generates an alert and sends it to the device. The device notifies the child of the alert by voice or by displaying it on the screen. For example, a message such as "Please return the blue block set to the top left of the toy box" may be displayed.

[1197] Furthermore, the system incorporates an emotion engine to recognize the user's emotions. This function records the user's emotions during the operation of locating and returning the item to its original position, and analyzes the emotion data. For example, the system uses a camera or microphone to analyze the user's facial expressions and tone of voice to detect the current emotion.

[1198] If the user returns the item to its correct location, the device takes another photo with the camera and sends the image data to the server. The server analyzes the image data and, once it confirms that the item has been returned to its proper place, adjusts the reward message based on the emotion detected by the emotion engine. For example, if the child is excited, the message displayed might say, "Good job! That's amazing!", whereas if the child is calm, the message displayed might say, "Good job, let's try harder next time."

[1199] Specific examples are as follows:

[1200] Suppose a user registers a "red toy car" in the "center of the shelf." The user opens the app on their device and takes a photo of the item and its designated location to register it. If a child plays with the red toy car and then leaves it on the floor after use, the device's camera takes a photo of the room, which the server analyzes. If the device detects that the item has not been returned to its designated location, a notification appears on the device saying, "Please return the red toy car to the center of the shelf." If the user complies and returns the item to the correct location, the device's camera takes another photo, and the server confirms that it has been returned to the correct location. The emotion engine then analyzes the user's emotions and generates an appropriate reward message, such as "Good job! Amazing!", which the device notifies them by voice or on-screen display.

[1201] This allows the system to develop the habit of returning items to their correct places, reducing the household chore burden on parents and providing effective education by providing feedback based on the user's emotions.

[1202] The processing flow will be explained below.

[1203] Step 1:

[1204] The user opens the app on their device and prepares to register an item in the system. The user selects the "Register Item" menu and enters the item name and its designated location. For example, the user sets the "blue block set" to the "top left of the toy box."

[1205] Step 2:

[1206] The device activates the camera, and the user places the blue block set in front of the camera. The user presses the shutter button, and the device captures the image. The captured image data is temporarily stored in the device.

[1207] Step 3:

[1208] The device transmits the captured image data and fixed position information to the server, which receives the data and analyzes the image data and fixed position information.

[1209] Step 4:

[1210] The server stores the image data and location information in a database, and this storage operation records the item and its location in the system.

[1211] Step 5:

[1212] A child uses an object, for example, takes a blue block set out of a toy box and starts playing with it.

[1213] Step 6:

[1214] The device uses a camera to take pictures of the room at a pre-set frequency, for example, every hour, and sends the image data to a server.

[1215] Step 7:

[1216] The server analyzes the received image data and checks the current location of the item. If the blue block set is not in the "top left of the toy box," the server generates an alert.

[1217] Step 8:

[1218] The server generates an alert and sends it to the device, which then notifies the child of the alert through audio or a display on the screen. For example, it might say, "Put the blue block set back on the top left shelf of the toy box."

[1219] Step 9:

[1220] The emotion engine analyzes the user's facial expressions and tone of voice to recognize their current emotion. The emotion data is processed internally and referenced when generating reward messages.

[1221] Step 10:

[1222] The user instructs the child to return the items to their correct locations, for example, returning the blue block set to the top left of the toy box.

[1223] Step 11:

[1224] The device then starts the camera again and takes a picture of the room. The captured image data is then sent to the server again.

[1225] Step 12:

[1226] The server analyzes the received image data and verifies that the blue block set has been returned to its original position. Based on the verification result, the server adjusts the reward message based on the emotion detected by the emotion engine.

[1227] Step 13:

[1228] The server sends a reward message to the device. The reward message is tailored to the child's current emotions. For example, messages such as "Good job! Amazing!" or "Good job, let's try harder next time" are generated.

[1229] Step 14:

[1230] The device will notify the child with a message via voice or screen display, allowing the child to receive appropriate feedback and form the habit of returning items to their correct place.

[1231] The above are the specific processing steps of the system.

[1232] Example 2

[1233] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1234] Many parents are forced to do additional housework to organize items because children are unable to smoothly return them to their correct locations. Furthermore, reward messages for children are standardized and lack appropriate feedback tailored to the situation. To solve these problems, a system is needed that automatically checks where items are returned and generates reward messages tailored to the user's emotions.

[1235] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1236] In this invention, the server includes means for setting a fixed position for an item used by a user, means for detecting the position of the item with a camera, means for notifying the user when the item has deviated from its fixed position, means for recognizing the user's emotion using an emotion engine and adjusting a reward message based on the emotion, and means for detecting when the item has been returned to its fixed position and generating and notifying the user of the reward message. This allows children to develop the habit of returning items to their correct positions, reduces the housework burden on parents, and enables effective feedback according to the user's emotions.

[1237] "User" refers to a person who uses the system to register and manage items.

[1238] "Item" refers to an object owned by a user that is intended to be registered in the system and returned to a designated location.

[1239] "Home location" refers to a predetermined place where an item should be returned after use.

[1240] "Camera" refers to a device that captures the current location of an item and the state of the room, and acquires the image data.

[1241] "Server" refers to the central processing unit that analyzes image data, operates the emotion engine, locates and notifies items, and generates reward messages.

[1242] "Emotion engine" refers to software or a system for recognizing a user's emotions and tailoring reward messages based on those emotions.

[1243] "Reward message" refers to a message generated in response to returning an item to its proper location that encourages or praises the user's behavior.

[1244] "Notification" refers to a means of communication in the form of audio or screen display to notify the user if an item has not been returned to its proper location or to notify the user of a reward message.

[1245] "Image data" refers to the digital information of images taken by a camera, and is used to analyze the location of an item or the user's emotions.

[1246] The present invention relates to a system that assists parents in returning items used by children to their designated locations, thereby reducing the burden of housework. Furthermore, the present invention incorporates an emotion engine that recognizes the user's emotions and adjusts reward messages based on the emotions. Specific embodiments of the system will be described below.

[1247] This system is primarily composed of three elements: the user, the terminal, and the server. First, the user registers and manages items through the terminal. The terminal provides an interface for inputting the name of the item and its designated location. When the user sets an item and its designated location, for example, by setting a "blue block set" to the "top left of the toy box," the terminal's camera takes a photo of the item and its designated location and sends the image data to the server. The server analyzes the received image data and stores the information about the item and its designated location in a database. An image recognition algorithm is used in this process.

[1248] Next, the process begins to check whether the items are returned to their designated locations after the child has used them. The device's camera periodically takes pictures of the room and sends this image data to the server. The server analyzes the received image data and checks whether the items are in their designated locations. If the items are not returned to their designated locations, the server generates an alert and sends notification data to the device. For example, the server could generate an alert stating, "The blue block set is not in its designated location," and the device will notify the user by voice or by displaying a message on its screen.

[1249] The system also incorporates an emotion engine that recognizes the user's emotions when returning an item to its proper place. When the user returns an item to its correct location, the device's camera and microphone record the user's facial expressions and tone of voice, and send this as emotional data to the server. The server analyzes the received emotional data and classifies the user's current emotional state as "joy," "excitement," "calm," or something similar.

[1250] When the user returns the item to its correct location, the device again takes a photo of the room with its camera and sends the image data to the server. The server analyzes the image data and, once it confirms that the item has been returned to its proper location, adjusts the reward message based on the emotion detected by the emotion engine. For example, if the child is excited, the message "Good job! That's amazing!" is generated, whereas if the child is calm, the message "Good job, let's try harder next time" is generated. Finally, the device notifies the user of the generated reward message by voice or on-screen display.

[1251] As a concrete example, consider the case where a user registers a "red toy car" as "the center of the shelf." The user opens the app on their device and takes a photo of the item and its designated location to register it. If a child plays with the red toy car and then leaves it on the floor after use, the device's camera takes a photo of the room and the server analyzes the image data. If the device detects that the item has not been returned to its designated location, a notification appears on the device saying, "Please return the red toy car to the center of the shelf." If the user complies and returns the item to the correct location, the device's camera takes another photo, and the server confirms that it has been returned to the correct location. The emotion engine then analyzes the user's emotions and generates an appropriate reward message, such as "Good job! Amazing!", which the device notifies them by voice or on-screen display.

[1252] An example of a prompt to be input to a generative AI model is, "Please describe the workflow of a system that helps children return toys to their proper places after playing with them. Please provide a detailed process, including user input, confirmation of the item's proper place, sentiment analysis, and reward message generation."

[1253] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1254] Step 1:

[1255] Product registration

[1256] The user opens the app on their device and inputs the items and their designated locations. For example, the "blue block set" is set to the "top left of the toy box."

[1257] Input: Item name and location information

[1258] The device uses the camera to take a photo of the item and the set fixed position, and sends the captured image data to the server.

[1259] Output: Image data

[1260] Specific operation: The app screen displays the instruction "Take a photo of the blue block set," and the user operates the device's camera to take a photo.

[1261] Step 2:

[1262] Data storage and analysis

[1263] The server analyzes the received image data and stores information about the item and its fixed location in a database.

[1264] Input: Image data

[1265] The server uses image recognition algorithms to identify the items and their locations and stores this information in a database.

[1266] Output: Database entry for item and location

[1267] Specific operation: As the server analyzes the image data, a progress bar will be displayed in the device app, and a confirmation notification will be sent to the user when the analysis is complete.

[1268] Step 3:

[1269] Checking the location of items

[1270] After the child uses the item, the terminal periodically takes pictures of the room and sends the image data to the server.

[1271] Input: Image data of the room condition

[1272] The server analyzes the received image data and checks whether the item is in the set fixed position.

[1273] Output: Item location status (in place or not)

[1274] Specific operation: The device displays a notification saying "Taking a photo" at the time of shooting, and the captured image is sent to the server.

[1275] Step 4:

[1276] Alert Generation

[1277] If the item is not returned to its original location, the server generates an alert and sends notification data to the terminal.

[1278] Input: Item location status

[1279] The server generates an alert message and sends it to the terminal.

[1280] Output: Alert message

[1281] Specific actions: For example, an alert saying "The blue block set is not in its proper place" will be displayed on the device, and a voice message will be played saying "Please return the blue block set to the top left of the toy box."

[1282] Step 5:

[1283] sentiment analysis

[1284] When the user returns the item to its correct location, the device's camera and microphone record the user's facial expressions and tone of voice, and send this data to the server as emotional data.

[1285] Input: User's facial expression data, voice data

[1286] The server analyzes the received emotional data and identifies the current emotional state.

[1287] Output: User's emotional state (e.g., happy, excited, calm)

[1288] What it does: The camera indicator light will turn on to let the user know that sentiment analysis is taking place.

[1289] Step 6:

[1290] Reward message generation and notification

[1291] The terminal then takes another photo with the camera to check if the item has been returned to the correct position, and sends the image data to the server.

[1292] Input: Image data of the returned item

[1293] The server analyzes the image data and, once it determines that the item has been returned to its proper location, adjusts the reward message based on the emotion detected by the emotion engine.

[1294] Output: Reward message

[1295] Specific behavior: For example, if a child is excited, the device will generate a message saying "Good job! That's amazing!" and notify the child by voice or displaying a message on the screen.

[1296] (Application example 2)

[1297] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1298] In traditional factories, parts placement errors and inaccurate storage were common, resulting in problems such as reduced work efficiency and mistakes. Preventing these mistakes required human supervision and alerts, which required a significant amount of effort and time. Furthermore, the lack of a system for properly understanding employees' emotions and motivation and providing feedback accordingly hindered improvements in work efficiency.

[1299] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes a means for setting a fixed position for items used by children, a means for detecting the position of the item with a camera, a means for notifying when the item has deviated from its fixed position, a means for detecting when the item has been returned to its fixed position and for generating and notifying a reward message, and a means for recognizing the emotion of the worker and adjusting the reward message based on the emotion. This makes it possible to prevent mistakes in part placement and a decrease in work efficiency, as well as to appropriately grasp the emotion of the worker and provide feedback accordingly.

[1300] The "means for setting a fixed location for an item used by a child" is a function that allows a user to set a predetermined location within the system where a particular item should be stored or placed.

[1301] The "means for detecting the position of an item using a camera" is a function for capturing the current position of an item using a camera and transmitting the position information to the system.

[1302] The "means for notifying when an article deviates from its fixed position" is a function for sending an alert to the user when an article deviates from its set fixed position.

[1303] The "means for detecting that an item has been returned to its designated location and generating and notifying a reward message" is a function for confirming that an item has been returned to the correct location and sending a reward message to the user based on that.

[1304] "Means for recognizing the emotions of workers and adjusting reward messages based on those emotions" refers to a function that uses sensors such as cameras and microphones to detect the emotional state of workers and adjust the content and tone of reward messages to match those emotions.

[1305] A "camera that monitors the usage status of an item" is a camera that monitors the location and status of an item while it is in use, and sends this information to a server as image data.

[1306] "Means for analyzing image data received by the server and confirming whether the item is in the set fixed position" is a function for analyzing image data sent to the server and confirming whether the item is in the set fixed position.

[1307] This invention relates to a system that aims to accurately manage the location of parts in a factory and improve work efficiency. Specifically, the system supports workers wearing smart glasses, confirms that items are returned to their designated locations, and provides appropriate feedback according to the worker's emotions.

[1308] First, we will explain the process of setting the fixed positions of items. The user uses smart glasses to register the fixed positions of each part in the system. Specifically, the user takes a photo of the part and its fixed position using the smart glasses' camera and sends the image data to the server. At that time, the user enters the part's name and fixed position, and the server stores this information in a database.

[1309] Next, we will explain how to monitor the status of parts when workers use them during work. The camera in the smart glasses periodically captures images of the workspace and sends the image data to a server. The server analyzes the received image data and generates a notification if an item deviates from its designated position. As a specific example, a message such as "Part A has deviated from its designated position. Please return it to the correct position" is displayed on the smart glasses.

[1310] Once the worker returns the item to its correct location, the smart glasses' camera takes another photo and sends the image data to the server. The server analyzes the image data and confirms that the item has been returned to its proper place. At this time, an emotion engine is used to detect the worker's emotions and tailor reward messages based on their emotions. For example, if the worker is focused, a short message like "Good job!" is displayed, whereas if the worker is nervous, a message like "Great job! Try again next time" is displayed.

[1311] For emotion recognition, the system uses a camera and microphone to analyze facial expressions and vocal tone, and employs an emotion recognition model. It processes image data using OpenCV and uses object detection algorithms such as HOGDescriptor to identify the location of objects. It also utilizes an external emotion recognition library.

[1312] Examples of prompts include:

[1313] "Detect the location of part A and notify it to return to its original position."

[1314] "Recognize workers' emotions and provide appropriate reward messages."

[1315] This system can prevent mistakes in part placement and improve work efficiency, as well as promote worker motivation and provide a more effective work environment.

[1316] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1317] Step 1:

[1318] The user registers the fixed location of an item. The user takes a photo of the item and its fixed location through the smart glasses and inputs the item's name and fixed location information. The camera in the smart glasses captures this and the image data is sent to the server. The input data is the item's name and location data, and the output is the registered fixed location information stored on the server.

[1319] Step 2:

[1320] The server analyzes the received image data and stores the fixed location information of the items in a database. Specifically, OpenCV is used to extract the features of the items in the image and store them in the database. The input data is image data and location information, and the output is the fixed location information of the items stored in the database.

[1321] Step 3:

[1322] The camera in the smart glasses periodically captures images of the workspace and sends the captured image data to a server. The input data is the periodically captured image data, and the output is the image data sent to the server.

[1323] Step 4:

[1324] The server analyzes the received image data and checks whether the item is in its designated location. Specifically, it uses OpenCV and HOGDescriptor to detect the item's location and compares it with the database's designated location information. The input data is the captured image data, and the output is the result of determining whether the item is in its designated location.

[1325] Step 5:

[1326] If an item is out of place, the server generates an alert message and notifies the smart glasses. Specifically, it displays a message such as "Part A is out of place. Please return it to the correct position." The input data is the judgment result, and the output is the alert message displayed on the smart glasses.

[1327] Step 6:

[1328] When the worker returns the item to the correct position, the camera in the smart glasses takes a picture again and sends the image data to the server. The input data is the recaptured image data, and the output is the image data sent to the server.

[1329] Step 7:

[1330] The server analyzes the image data again to confirm that the item has been returned to the correct location. If confirmed, it uses an emotion engine to recognize the worker's emotion and generates a reward message based on that emotion. The input data is the received image data and emotion data again, and the output is a reward message based on the emotion.

[1331] Step 8:

[1332] The server sends the generated reward message to the smart glasses and notifies the worker. Specifically, a message such as "Good job! Try your best next time" is displayed. The input data is the generated reward message, and the output is the reward message displayed on the smart glasses.

[1333] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1334] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1335] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1336] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1337] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1338] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1339] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1340] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1341] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1342] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1343] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1344] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1345] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1346] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1347] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1348] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1349] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1350] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1351] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1352] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1353] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1354] The following is further disclosed regarding the above embodiment.

[1355] (Claim 1)

[1356] a means for establishing a fixed location for items used by children;

[1357] means for detecting the position of the article by a camera;

[1358] means for notifying if an item is out of position;

[1359] means for detecting when the item has been returned to its original location and generating and notifying a reward message;

[1360] A system including:

[1361] (Claim 2)

[1362] 2. The system according to claim 1, wherein the camera that monitors the usage status of the items includes means for periodically photographing the state of the room and transmitting the image data to the server.

[1363] (Claim 3)

[1364] 2. The system according to claim 1, further comprising means for analyzing the image data received by the server and confirming whether the article is present in the set fixed position.

[1365] "Example 1"

[1366] (Claim 1)

[1367] A means for a user to set a fixed location for an item;

[1368] a means for taking an image of an item and its fixed position using a camera of the terminal and transmitting the image data to a server;

[1369] A means for the server to analyze the image data and store information about the items and their fixed locations in a database;

[1370] means for sending an alert notification to a terminal if the article is displaced from its home position;

[1371] a means for taking another photograph with the camera when the user returns the item to the correct position and transmitting the image data to the server;

[1372] A server analyzes the image data, confirms that the item has been returned to its designated location, and generates a reward message to notify the user.

[1373] A system including:

[1374] (Claim 2)

[1375] 2. The system according to claim 1, further comprising means for periodically photographing the state of the room with a camera of the terminal in order to monitor the usage of the items, and transmitting the image data to the server.

[1376] (Claim 3)

[1377] 2. The system according to claim 1, further comprising means for analyzing the image data received by the server and confirming whether the article is present in the set fixed position.

[1378] "Application Example 1"

[1379] (Claim 1)

[1380] a means for establishing fixed locations for items used by children and workers;

[1381] means for detecting the position of the article by a camera;

[1382] means for notifying if an item is out of position;

[1383] means for detecting when the item has been returned to its original location and generating and notifying a reward message;

[1384] A system including:

[1385] (Claim 2)

[1386] 2. The system according to claim 1, wherein the camera that monitors the usage status of the items includes means for periodically photographing the status of the facility and transmitting the image data to the server.

[1387] (Claim 3)

[1388] 2. The system according to claim 1, further comprising means for analyzing the image data received by the server and confirming whether the article is present in the set fixed position.

[1389] (Claim 4)

[1390] The system according to claim 1, further comprising: means for managing the positions of tools and parts in an industrial site, detecting tools and parts that have deviated from their designated positions and notifying production workers thereof; and means for generating and notifying a reward message when the tools and parts are returned to their correct positions.

[1391] (Claim 5)

[1392] 10. The system of claim 1, further comprising means for using OpenCV or an equivalent image analysis library as software for performing image analysis.

[1393] (Claim 6)

[1394] 10. The system of claim 1, further comprising a notification library for sending reward messages.

[1395] "Example 2: Combining Emotion Engines"

[1396] (Claim 1)

[1397] A means for setting a fixed position for an item used by a user;

[1398] means for detecting the position of the article by a camera;

[1399] means for notifying if an item is out of position;

[1400] means for recognizing a user's emotion using an emotion engine and tailoring a reward message based on the emotion;

[1401] means for detecting when the item has been returned to its original location and generating and notifying a reward message;

[1402] A system including:

[1403] (Claim 2)

[1404] 2. The system according to claim 1, wherein the camera that monitors the usage status of the items includes means for periodically photographing the state of the room and transmitting the image data to the server.

[1405] (Claim 3)

[1406] 2. The system according to claim 1, further comprising means for analyzing the image data received by the server and confirming whether the article is present in the set fixed position.

[1407] "Application example 2 when combining emotion engines"

[1408] (Claim 1)

[1409] a means for establishing a fixed location for items used by children;

[1410] means for detecting the position of the article by a camera;

[1411] means for notifying if an item is out of position;

[1412] means for detecting when the item has been returned to its original location and generating and notifying a reward message;

[1413] a means for recognizing worker emotions and tailoring reward messages based on those emotions;

[1414] A system including:

[1415] (Claim 2)

[1416] 2. The system according to claim 1, wherein the camera that monitors the usage status of the items includes means for periodically photographing the status of the workspace and transmitting the image data to the server.

[1417] (Claim 3)

[1418] 2. The system according to claim 1, further comprising means for analyzing the image data received by the server and confirming whether the article is present in the set fixed position. [Explanation of symbols]

[1419] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. a means for establishing a fixed location for items used by children; means for detecting the position of the article by a camera; means for notifying if an item is out of position; means for detecting when the item has been returned to its original location and generating and notifying a reward message; A system including:

2. 2. The system according to claim 1, wherein the camera for monitoring the usage of the items includes means for periodically photographing the state of the room and transmitting the image data to the server.

3. 2. The system according to claim 1, further comprising means for analyzing the image data received by the server and confirming whether the article is present in the set fixed position.

Images