System

Abstract

A system is provided.SOLUTION: A system including means for capturing, by a guest, an image of a specific place in a room of an accommodation facility, means for transmitting captured image data to a server, means for analyzing the image data received by the server and detecting an article in the image, means for collecting, by the server, data from a sensor in the room, means for determining, by the server, presence or absence of a lost article by integrating an image analysis result and the data from the sensor, and means for notifying a front desk and a terminal of the guest of a determination result.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] At hotels, items are frequently left behind when guests check out, resulting in significant costs for storage and handling. Another contributing factor is that the front desk is extremely busy at checkout, making it difficult to adequately check for lost items. Furthermore, there is a need for a method to effectively prevent items from being left behind while ensuring privacy. A system that can solve these issues and improve operational efficiency at hotels is desirable. [Means for solving the problem]

[0005] The present invention provides a means for guests to take images of specific locations in their rooms (such as their feet, under the bed, the refrigerator, or the closet) before checking out. It also includes a means for sending the captured image data to a server, which then analyzes the image data to detect whether any items (lost items) have been left behind. The server also collects data from sensors in the room (such as weight sensors and power connection sensors) and combines this data with the image analysis results to determine whether any items have been left behind. The results of this determination are notified in real time to the front desk and the guest's terminal. Furthermore, it also includes a means for displaying specific instructions when guests take images of their rooms, thereby preventing them from leaving items behind.

[0006] "Guest" means a guest staying at an accommodation facility.

[0007] "Accommodation facilities" refers to facilities that provide lodging services, such as hotels and guesthouses.

[0008] "Inside the room" refers to the interior of the guest room where the guest is staying.

[0009] "Images" refers to photographic data taken by guests using their smartphones or cameras.

[0010] "Image data" refers to data that represents a captured image in digital format.

[0011] "Server" refers to a central computer that processes, stores, and transmits data over a network.

[0012] "Image analysis" refers to the process of analyzing image data using AI or other technologies to extract specific information (for example, whether or not an item has been left behind).

[0013] A "sensor" refers to a device that detects a physical quantity (e.g., weight or power connection) and converts it into digital data.

[0014] "Weight sensor" refers to a sensor for measuring the weight of an object.

[0015] "Power connection sensor" refers to a sensor for detecting whether or not an electrical device is connected to a power source.

[0016] "Items" refers to all personal possessions brought by the Guest.

[0017] "Lost property" refers to items left behind by guests in their accommodation rooms.

[0018] "Determination" refers to the process in which the server determines whether an item has been left behind based on image analysis and sensor information.

[0019] "Front desk" refers to the department that manages the accommodation facility and handles reception duties.

[0020] "Terminal" refers to an external device such as a smartphone or tablet operated by a guest.

[0021] "Notification" refers to a message sent to communicate the results of the assessment to the guest or front desk.

[0022] "Instructions" refers to the specific location and procedure for taking photos that the application provides to guests when they take photos of their rooms. [Brief explanation of the drawings]

[0023] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0024] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0025] First, the terms used in the following description will be explained.

[0026] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0027] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0028] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0029] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0030] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0031] [First embodiment]

[0032] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0033] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0034] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0035] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0036] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0037] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0038] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0039] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0040] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0041] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0042] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0043] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0044] The present invention provides a system and method for effectively preventing items left behind at check-out in hotels. This system detects items left behind in real time by combining image analysis and sensor technology, improving the operational efficiency of hotels. Specific embodiments of the present invention are described below.

[0045] System configuration

[0046] The system consists of a guest terminal, a server, and sensors in the room.

[0047] 1. User (guest) terminal

[0048] Before checking out, guests use a smartphone app to take pictures of specific areas in their room (at their feet, under the bed, inside the refrigerator, inside the closet).

[0049] The terminal transmits the captured image data to the server via a secure protocol (for example, HTTPS).

[0050] 2. Server

[0051] The server receives the image data sent from the terminal and stores it in an internal database.

[0052] It is equipped with an AI module for analyzing the received image data, which uses image analysis technology to detect objects in the image.

[0053] Data is collected from sensors installed in the room (e.g., weight sensors and power connection sensors), and this data is integrated with the results of image analysis.

[0054] The system determines whether any items have been left behind and notifies the guest's terminal and the front desk of the results.

[0055] 3. In-room sensors

[0056] A weight sensor inside the refrigerator detects whether there is anything left inside the refrigerator.

[0057] The power connection sensor detects chargers and other electronic devices that remain plugged in.

[0058] Program processing

[0059] A way for guests to take photos

[0060] Before checking out, users launch the app on their smartphone and follow the app's instructions to take pictures of specific points in the room (at their feet, under the bed, inside the refrigerator, inside the closet).The images are temporarily saved on the device.

[0061] Image data transmission means

[0062] The terminal transmits the captured image data to the server using a secure protocol (for example, HTTPS).

[0063] Image analysis and sensor information integration

[0064] The server stores the received image data in a database and sends it to the AI ​​module for analysis. The AI ​​module detects specific items in the image (e.g., chargers, clothing, household items). The server also collects data from weight sensors and power connection sensors installed in the room and integrates it with the image analysis results.

[0065] How to determine whether an item has been left behind and how to notify the person

[0066] The server determines whether any items have been left behind based on image analysis and sensor information. The results are sent in real time to the front desk management system and the guest's device. For example, a message such as "There is a charger in the closet. Please check." is displayed.

[0067] Specific examples

[0068] The user launches the room confirmation app and follows the instructions to take images of their feet, under the bed, inside the refrigerator, and inside the closet. The device sends this image data to a server, which analyzes it using an AI module. If the analysis detects a charger in the closet, the server notifies the front desk and the guest's device of this information, prompting them to check if they have left anything behind. The user confirms the notification and retrieves the forgotten item, completing the check-out process without any problems.

[0069] Through the above process, the present invention can improve the efficiency of lost property handling at accommodation facilities, reduce operational costs, and improve service quality.

[0070] The processing flow will be explained below.

[0071] Step 1:

[0072] Before checking out, the user launches the smartphone app in their accommodation room and selects the "Check Room" function.

[0073] Step 2:

[0074] The app prompts the user to take pictures of each point in turn: at their feet, under the bed, inside the refrigerator, and inside the closet.

[0075] Step 3:

[0076] The user follows the app's instructions and takes pictures of each point using their smartphone camera.

[0077] Step 4:

[0078] The device temporarily stores the captured image in its internal storage and prepares it for transmission to a server using a secure protocol (e.g., HTTPS).

[0079] Step 5:

[0080] The terminal transmits the captured image data to the server.

[0081] Step 6:

[0082] The server stores the received image data in a database.

[0083] Step 7:

[0084] The server calls the AI ​​module and begins image analysis using the image data as input.

[0085] Step 8:

[0086] The AI ​​module extracts features within the image and detects specific items (e.g., chargers, clothing, everyday items).

[0087] Step 9:

[0088] At the same time, the server collects data from weight sensors and power connection sensors installed in the room.

[0089] Step 10:

[0090] The server combines the image analysis results with data from the sensors to determine whether any items have been left behind.

[0091] Step 11:

[0092] The server notifies the judgment results in real time to the front management system and the user's terminal.

[0093] Step 12:

[0094] The device receives the notification from the server and displays a warning or confirmation message to the user about whether they have left something behind. For example, it displays "There is a charger in the closet. Please check."

[0095] Step 13:

[0096] The user checks the message and retrieves any items left in the room as necessary.

[0097] Step 14:

[0098] Once the user has confirmed that all of their lost items have been collected, they report this to the server via the app.

[0099] Step 15:

[0100] The server receives the user's report and notifies the front desk that the user has not left anything behind.

[0101] This allows users to check out with peace of mind and allows accommodation facilities to efficiently deal with lost items.

[0102] Example 1

[0103] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0104] Leaving something behind when checking out of a hotel is inconvenient for guests and incurs additional costs for the hotel. The present invention aims to provide a method and system for preventing guests from leaving something behind when checking out, efficiently and accurately detecting whether an item has been left behind, and quickly notifying guests and hotel staff. Another objective is to simplify the procedure for preventing guests from leaving something behind, thereby reducing the hassle for guests.

[0105] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0106] In this invention, the server includes means for a guest to take an image of a specific location in a room at an accommodation facility, means for transmitting the captured image data to the server, means for the server to analyze the received image data and detect items in the image, means for the server to collect data from sensors in the room, means for the server to integrate the image analysis results and the data from the sensors to determine whether an item has been left behind, and means for notifying the management system and the guest's terminal of the determination result. This allows guests to prevent leaving items in their rooms with a simple procedure when checking out, improves the operational efficiency of accommodation facilities, and reduces the cost of dealing with lost items.

[0107] "Guest" means a user staying at an accommodation facility.

[0108] "Accommodation facilities" are facilities that provide short-term stays, such as hotels and inns.

[0109] "Room" means a separate space within the accommodation facility where the guest stays.

[0110] "Means for taking images" refers to a method for obtaining images of a specified location using the camera function of a smartphone, tablet, etc.

[0111] "Image data" is digital data that includes information about a captured image.

[0112] A "server" is a computer system that receives, analyzes, and stores data sent from the guest's terminal.

[0113] "Transmission means" means a method for transferring data using the Internet and a secure communication protocol (e.g., HTTPS).

[0114] "Means for analyzing" refers to a method of processing received image data using analytical technology such as an AI module to detect specific items.

[0115] "Items" are individual belongings brought by guests, such as chargers, clothing, and daily necessities.

[0116] A "sensor" is a device that is installed in a room and measures weight and power connection status.

[0117] A "weight sensor" is a sensor that detects changes in weight when an object is placed on it.

[0118] A "power connection sensor" is a sensor that detects whether a charger or electronic device is connected to a power source.

[0119] "Means of collecting data" refers to the method of transferring information obtained from the sensor to the server.

[0120] "Means of integration" refers to a method of combining analysis results with sensor data to make a comprehensive judgment.

[0121] The "means for determining whether or not an item has been left behind" is a method for determining whether or not any items have been left behind in a room based on the image analysis results and sensor data.

[0122] The "means for notifying the determination result" is a method for reporting the presence or absence of a lost item to the guest and the front desk management system.

[0123] The "management system" is a computer system for managing the front desk operations of accommodation facilities.

[0124] The present invention provides a system and method for effectively preventing items from being left behind at check-out in accommodation facilities. This system combines image analysis technology and sensor technology to detect whether or not items have been left behind in real time, improving the operational efficiency of accommodation facilities.

[0125] System configuration

[0126] The system consists of the following elements:

[0127] 1. User's device

[0128] Before checking out, guests use a smartphone app to take pictures of specific areas in their room (at their feet, under the bed, inside the refrigerator, inside the closet).

[0129] The terminal transmits the captured image data to the server via a secure protocol (for example, HTTPS).

[0130] 2. Server

[0131] The server receives the image data sent from the terminal and stores it in an internal database.

[0132] It includes an AI module for analyzing the received image data, which uses image analysis technology to detect items (e.g., chargers or clothing) in the image.

[0133] The server collects data from sensors installed in the room (e.g., weight sensors and power connection sensors) and integrates this data with the results of image analysis.

[0134] The system determines whether any items have been left behind and notifies the guest's terminal and management system of the results.

[0135] 3. In-room sensors

[0136] A weight sensor inside the refrigerator detects whether there is anything left inside the refrigerator.

[0137] The power connection sensor detects chargers and other electronic devices that remain plugged in.

[0138] Specific processing of the program

[0139] Before checking out, users launch the app on their smartphone and follow the app's instructions to take pictures of specific points in the room (at their feet, under the bed, inside the refrigerator, inside the closet).The images are temporarily saved on the device.

[0140] The terminal transmits the captured image data to the server using a secure protocol (for example, HTTPS).

[0141] The server stores the received image data in a database and sends it to the AI ​​module for analysis. The AI ​​module detects specific items in the image (e.g., chargers, clothing, household items). The server also collects data from weight sensors and power connection sensors installed in the room and integrates the image analysis results with this sensor data.

[0142] The server determines whether any items have been left behind based on image analysis and sensor information. The results are sent to the management system and the guest's device in real time. For example, a message such as "There is a charger in the closet. Please check." is displayed.

[0143] Specific examples

[0144] The user launches the room inspection app and follows the instructions to take images of the floor, under the bed, inside the refrigerator, and inside the closet. The device sends this image data to a server, which analyzes it using an AI module. If the analysis detects a charger in the closet, the server notifies the management system and the guest's device of this information, prompting the guest to check if they have left anything behind. The user confirms the notification and retrieves the forgotten item, completing the check-out process without any problems.

[0145] Example prompt sentence:

[0146] "Please explain how a hotel can capture specific images of a room upon check-out to detect lost items. Please include the specific steps, technology used, and examples."

[0147] Through the above process, the present invention can improve the efficiency of lost property handling at accommodation facilities, reduce operational costs, and improve service quality.

[0148] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0149] Step 1:

[0150] Before checking out, users launch the app on their smartphone and follow the app's instructions to take images of specific locations in the room (at their feet, under the bed, inside the refrigerator, inside the closet), which allows visual information about specific locations in the room to be collected.

[0151] Input: App instructions

[0152] Output: Captured image data

[0153] Specific operation: The user presses the camera button in the app to take a picture of a specific location, and the image is saved on the device.

[0154] Step 2:

[0155] The device sends the captured image data to the server using a secure protocol (e.g., HTTPS). At this stage, the image data is encrypted and sent securely.

[0156] Input: Captured image data

[0157] Output: Transfer of image data to the server

[0158] Specific operation: The device encrypts the image data using the HTTPS protocol and sends it to the server over the network.

[0159] Step 3:

[0160] The server stores the received image data in a database, then sends it to the AI ​​module to begin analysis, which uses image analysis technology to detect specific items.

[0161] Input: Received image data

[0162] Output: Analysis results (detection results of objects in the image)

[0163] Data processing: Image data is input into the AI ​​module for image segmentation and object detection.

[0164] How it works: The server reads the image data, and the AI ​​module identifies specific items in the image (e.g., chargers or clothing).

[0165] Step 4:

[0166] The server collects real-time data from sensors installed in the room (e.g., weight sensors and power connection sensors), and stores the sensor data in a database.

[0167] Input: Data from sensors

[0168] Output: Sensor data stored in a database

[0169] Data processing: Determine whether or not there is an item from the weight sensor data, and whether or not there is a device connected to a power source from the power connection sensor data.

[0170] Specific operation: A weight sensor inside the refrigerator detects whether something is left inside, and a power connection sensor detects whether a device such as a charger is connected.

[0171] Step 5:

[0172] The server combines the image analysis results with sensor data to determine whether an item has been left behind, and the AI ​​module's analysis results with sensor information to confirm whether an item has been left behind.

[0173] Input: Image analysis results, sensor data

[0174] Output: Result of lost item detection

[0175] Data calculation: Image analysis results are combined with sensor data to calculate the probability of a lost item being found.

[0176] Specific operation: The server compares the image analysis results with the sensor data and makes a determination such as "There is a charger in the closet."

[0177] Step 6:

[0178] The server notifies the management system and the user's terminal of the determination result in real time. The notification content is a specific message (for example, "There is a charger in the closet. Please check it.").

[0179] Input: Lost item detection result

[0180] Output: Information message

[0181] Specific operation: The server generates push notifications or emails based on the judgment results and sends them to the management system and the user's device.

[0182] (Application example 1)

[0183] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0184] There is a need to solve the problem of workers leaving parts or tools behind in their work area when they finish work in a factory. In large factories in particular, the effort required to check and find forgotten items increases, so a system that can efficiently detect lost items and notify workers is needed.

[0185] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0186] In this invention, the server includes: means for a guest to take an image of a specific location in a hotel room; means for transmitting the captured image data to the server; means for the server to analyze the received image data and detect items in the image; means for the server to collect data from sensors in the room; means for the server to combine the image analysis results and the sensor data to determine whether any items have been left behind; means for notifying the front desk and the guest's terminal of the determination result; means for a factory worker to take an image of a specific point in a specific work area; means for a robot to analyze and combine the sensor data and image data in the work area to detect left-behind parts or tools; and means for notifying the worker's terminal of the detection result. This makes it possible to efficiently and quickly detect items left behind in a factory and notify the worker.

[0187] "Guest" refers to a person using an accommodation facility.

[0188] "Means of capturing images of a specific location" refers to the process by which a user uses a smartphone or device to take a photo of a specific area.

[0189] "Server" refers to a central computing device that analyzes and stores received data and manages various processes.

[0190] A "sensor" refers to a device that detects physical data (such as weight or position) and outputs it as electronic data.

[0191] "Means for analyzing image data" refers to technology that identifies and detects specific items or structures based on received images.

[0192] "Means for determining whether an item has been left behind by integrating image analysis results and data from sensors" refers to the process of combining analyzed image information with sensor data to determine whether an item has been left behind.

[0193] "Means for notifying the front desk and guest terminals of the results of the determination" refers to a system that transmits information regarding whether or not an item has been left behind to the accommodation staff and guest terminals in real time.

[0194] "Factory workers" refers to employees who perform production or management work within a factory.

[0195] "Work area" refers to a specific area or section within a factory where specific work is carried out.

[0196] "Robot" refers to a programmable mechanical device that operates automatically to perform specific tasks.

[0197] "Means for detecting left-behind parts and tools" refers to technology that identifies parts and tools left behind in the work area after work is completed.

[0198] "Means of notification" refers to the communication methods and protocols used to convey specific information to users and related systems.

[0199] The present invention provides a system for efficiently detecting parts and tools left behind in work areas within a factory and notifying workers. This system combines image analysis technology and sensor technology to detect left-behind items in real time and improve work efficiency. Specific embodiments of the present invention are described below.

[0200] System configuration

[0201] The system consists of terminals for factory workers, a server, and robots and sensors within the work area.

[0202] 1. Factory worker terminals

[0203] Before finishing work, workers use their smartphones or tablets to take images of specific locations within their work area (on their desks, inside shelves, under the floor).

[0204] The device sends the captured image data to the server via a secure protocol (e.g., HTTPS).

[0205] 2. Server

[0206] The server stores the received image data in an internal database.

[0207] The server is equipped with an image analysis module based on a generative AI model, which analyzes the captured image data and detects parts and tools within the image.

[0208] Meanwhile, the server also collects data from sensors (weight sensors, position sensors) installed within the work area, and integrates this data with the image analysis results to determine whether any items have been left behind.

[0209] The results of the assessment are sent to the factory workers' terminals in real time.

[0210] 3. Robots and sensors in the work area

[0211] The robot is equipped with a camera that scans specific locations to capture real-time images, allowing it to automatically detect parts or tools left behind after a task is completed.

[0212] Weight sensors and position sensors are used as sensors, and this data is sent to a server.

[0213] Hardware and Software Use

[0214] Hardware: Factory robots (with cameras), sensors (weight sensors, position sensors)

[0215] Software: Image analysis software (TensorFlow, OpenCV), notification system (Firebase, Push notifications)

[0216] Process Overview

[0217] The server receives, stores, and analyzes image data sent from the terminal. A generative AI model using TensorFlow and OpenCV is used for the analysis to detect parts and tools in the image. The detection results are integrated with data from weight and position sensors to ultimately determine whether any items have been left behind. The results are then sent to the worker's terminal in real time via Firebase. This allows for quick and efficient confirmation of lost items in the work area.

[0218] Specific examples

[0219] After completing their work, factory workers use their smartphones to take pictures of their designated work areas. The images are sent to a server where they are analyzed by a generative AI model. At the same time, a robot scans the work area and captures real-time images. Based on this data, any lost items are identified and a notification is sent to the worker's smartphone.

[0220] Prompt Sentence Examples

[0221] "Implement a system to detect items left behind after work is completed. The system uses image analysis and sensor technology to automatically detect parts or tools left behind in the work area and notify the worker of the results in real time."

[0222] Through the above process, the present invention can improve the efficiency of dealing with lost items in factories and significantly improve work efficiency and safety.

[0223] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0224] Step 1:

[0225] A factory worker takes an image of a specific location in the work area (a work desk, inside a shelf, under the floor) using a smartphone or tablet. The input is the image data captured using the smartphone or tablet, and the output is the image data temporarily stored on the device.

[0226] Step 2:

[0227] The device sends the captured image data to the server using a secure protocol (e.g., HTTPS). The input in this step is the image data, and the output is the image data uploaded to the server.

[0228] Step 3:

[0229] The server stores the received image data in an internal database. The input in this step is the image data sent from the terminal, and the output is the image data stored in the database.

[0230] Step 4:

[0231] The server performs image analysis. It uses its generative AI model (using TensorFlow and OpenCV) to analyze the image data and detect specific parts and tools. The input is the image data stored in the database, and the output is the analysis results (a list of detected parts and tools).

[0232] Step 5:

[0233] The server collects data from sensors (weight sensors, position sensors) installed in the work area. The input in this step is the data measured by the sensors, and the output is the sensor data sent to the server.

[0234] Step 6:

[0235] The server combines the image analysis results and sensor data to determine whether an item has been left behind. The input is the analysis results and sensor data, and based on this, it calculates whether an item has been left behind. The output is the result of the lost item determination (whether an item has been left behind or not, and the name of the specific part or tool).

[0236] Step 7:

[0237] The server notifies the factory worker of the result via their smartphone or tablet. The server sends the result in real time using a notification system such as Firebase. The input is the result of the lost item detection, and the output is a notification message that is displayed on the worker's device.

[0238] Step 8:

[0239] The factory worker checks the notification and returns to the site to retrieve any items that have been left behind. The input is the notification message displayed on the terminal, and the output is the parts or tools that have actually been retrieved.

[0240] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0241] The present invention combines a system for preventing items left behind at check-out of accommodation facilities with an emotion engine that recognizes the user's emotions, providing notifications and instructions according to the user's state. This system can improve the efficiency of handling lost items and the user experience. Specific embodiments of the present invention are described below.

[0242] System configuration

[0243] This system consists of a guest terminal, a server, sensors in the room, and an emotion engine.

[0244] 1. User (guest) terminal

[0245] Before checking out, guests use a smartphone app to take pictures of specific areas in their room (at their feet, under the bed, inside the refrigerator, inside the closet).

[0246] The terminal transmits the captured image data to the server via a secure protocol (for example, HTTPS).

[0247] The device is equipped with an emotion engine that analyzes the user's voice and facial expressions to recognize their emotions.

[0248] 2. Server

[0249] The server receives the image data sent from the terminal and stores it in an internal database.

[0250] It is equipped with an AI module for analyzing the received image data, which uses image analysis technology to detect items (lost items) in the image.

[0251] Data is collected from sensors installed in the room (such as weight sensors and power connection sensors), and this data is integrated with the results of image analysis.

[0252] The system determines whether any items have been left behind and notifies the guest's terminal and the front desk of the results.

[0253] 3. In-room sensors

[0254] A weight sensor inside the refrigerator detects whether there is anything left inside the refrigerator.

[0255] The power connection sensor detects chargers and other electronic devices that remain plugged in.

[0256] 4. Emotion Engine

[0257] The emotion engine recognizes the user's emotions by analyzing their voice and facial expressions, and if the user is feeling stressed, a warning message is displayed urging them to check for forgotten items.

[0258] Program processing

[0259] A way for guests to take photos

[0260] Before checking out, users launch the app on their smartphone and follow the app's instructions to take pictures of specific points in the room (at their feet, under the bed, inside the refrigerator, inside the closet).The images are temporarily saved on the device.

[0261] Image data transmission means

[0262] The terminal transmits the captured image data to the server using a secure protocol (for example, HTTPS).

[0263] Image analysis and sensor information integration

[0264] The server stores the received image data in a database and sends it to the AI ​​module for analysis. The AI ​​module detects specific items in the image (e.g., chargers, clothing, household items). The server also collects data from weight sensors and power connection sensors installed in the room and integrates it with the image analysis results.

[0265] How to determine whether an item has been left behind and how to notify the person

[0266] The server determines whether any items have been left behind based on image analysis and sensor information. The results are sent in real time to the front desk management system and the guest's device. For example, a message will be displayed saying, "There is a charger in the closet. Please check."

[0267] Emotion recognition and notification using an emotion engine

[0268] The emotion engine installed on the device analyzes the user's voice and facial expressions to recognize their emotions. If the user is feeling stressed, the emotion engine displays a warning message to encourage them to check for forgotten items. Conversely, if the user is relaxed, the engine displays a reminder to help ensure a smooth checkout.

[0269] Specific examples

[0270] The user launches the room inspection app and follows the instructions to take pictures of their feet, under the bed, inside the refrigerator, and inside the closet. The device sends this image data to a server, which analyzes it using an AI module. If the analysis results indicate that a charger is found in the closet, the server notifies the front desk and the guest's device of this information. The user checks the notification and retrieves the forgotten item. During this time, the emotion engine analyzes the user's emotions, and if the user is feeling stressed, it displays a warning saying, "You may have left something behind. Please check," and if the user is relaxed, it displays a reminder saying, "You've checked everything. Good job."

[0271] In this way, by combining an emotion engine, the present invention enables flexible responses according to the user's condition, improving the efficiency of lost property responses at accommodation facilities while also improving the user experience.

[0272] The processing flow will be explained below.

[0273] Step 1:

[0274] Before checking out, the user launches the smartphone app in their accommodation room and selects the "Check Room" function.

[0275] Step 2:

[0276] The app prompts the user to take pictures of each point in turn: at their feet, under the bed, inside the refrigerator, and inside the closet.

[0277] Step 3:

[0278] The user follows the app's instructions and takes pictures of each point using their smartphone camera.

[0279] Step 4:

[0280] The device temporarily stores the captured image in its internal storage and prepares it for transmission to a server using a secure protocol (e.g., HTTPS).

[0281] Step 5:

[0282] The terminal transmits the captured image data to the server.

[0283] Step 6:

[0284] The server stores the received image data in a database.

[0285] Step 7:

[0286] The server calls the AI ​​module and begins image analysis using the image data as input.

[0287] Step 8:

[0288] The AI ​​module extracts features within the image and detects specific items (e.g., chargers, clothing, everyday items).

[0289] Step 9:

[0290] At the same time, the server collects data from weight sensors and power connection sensors installed in the room.

[0291] Step 10:

[0292] The server combines the image analysis results with data from the sensors to determine whether any items have been left behind.

[0293] Step 11:

[0294] The server notifies the judgment results in real time to the front management system and the user's terminal.

[0295] Step 12:

[0296] The device receives the notification from the server and displays a warning or confirmation message to the user about whether they have left something behind. For example, it displays "There is a charger in the closet. Please check."

[0297] Step 13:

[0298] The emotion engine analyzes the user's emotions from their voice and facial expressions, and if the user is feeling stressed, a warning message is displayed urging them to check for forgotten items. For example, it displays "You may have left something behind. Please check."

[0299] Step 14:

[0300] Conversely, if the emotion engine detects a relaxed state in the user, it will display a reminder to make sure they have not forgotten anything, for example, "You've checked everything, good job."

[0301] Step 15:

[0302] The user checks the message and retrieves any items left in the room as necessary.

[0303] Step 16:

[0304] Once the user has confirmed that all of their lost items have been collected, they report this to the server via the app.

[0305] Step 17:

[0306] The server receives the user's report and notifies the front desk that the user has not left anything behind.

[0307] This allows users to check out with peace of mind and allows accommodation facilities to efficiently deal with lost items.

[0308] Example 2

[0309] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0310] Discovering and dealing with items left behind at check-out has traditionally been a major challenge at hotels. Users often neglect to check their belongings when checking out, which can lead to items being left behind. Furthermore, depending on the guest's condition, stress or fatigue can lead to neglecting to check their belongings, creating a need for appropriate notifications and support tailored to the user's emotional state. To solve these issues, it is necessary to efficiently and accurately determine whether an item has been left behind, as well as to respond flexibly by taking the user's emotional state into account.

[0311] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0312] In this invention, the server includes means for allowing guests to take images of specific locations in their rooms, means for transmitting the captured image data to the server, means for the server to analyze the received image data and detect items in the images, means for the server to collect data from sensors in the room, means for the server to integrate the image analysis results with the sensor data to determine whether any items have been left behind, means for notifying the front desk and the guest's terminal of the determination result, a terminal including an emotion engine that recognizes emotions by analyzing the guest's voice and facial expressions, and means for providing notifications and warning messages according to the guest's emotional state. This allows guests to efficiently and accurately check their belongings when checking out and receive appropriate notifications and support according to their emotional state.

[0313] "Guest" refers to a user staying at an accommodation facility.

[0314] "Inside the room" refers to the interior of the room in which the guest stays within the accommodation facility.

[0315] "Specific locations" refers to areas within a hotel room where items are likely to be left behind upon check-out (such as at the feet, under the bed, in the refrigerator, in the closet, etc.).

[0316] "Means of taking images" refers to the process by which guests take photos of specific locations in their rooms using a camera-equipped device such as a smartphone or tablet.

[0317] "Means of transmission" refers to the method of transferring the captured image data to a server via a communication means such as the Internet. Specifically, this is done using a secure protocol (e.g., HTTPS).

[0318] "Server" refers to a computer system for receiving, storing, and analyzing image data.

[0319] "Means of analyzing and detecting items in images" refers to the process of recognizing and identifying items (lost items) from received image data using AI technology.

[0320] "Sensor" refers to a device installed in a room of a lodging facility, including a weight sensor and a power connection sensor, for detecting various information within the room.

[0321] "Means of collecting data" refers to the process of obtaining information from sensors installed in the room.

[0322] "Method of integrating to determine whether or not an item has been left behind" refers to the process of combining the results of image analysis with data obtained from sensors to determine whether or not an item has been left behind in the room.

[0323] "Means of notification" refers to the method of transmitting the results of the determination of whether or not an item has been left behind to the front desk and the guest's terminal.

[0324] An "emotion engine" refers to software or hardware that analyzes a user's voice and facial expressions to recognize their emotions.

[0325] "Emotional state" refers to the psychological state of the guest, such as stress or relaxation.

[0326] "Means for providing notification or warning messages" refers to the process of displaying appropriate messages depending on the emotional state of the guest.

[0327] MODE FOR CARRYING OUT THE INVENTION

[0328] The present invention is a system for preventing users from leaving items behind when checking out of accommodation facilities, and provides notifications and instructions according to the user's state by combining it with an emotion engine that recognizes the user's emotions. Specific embodiments of the present invention are described below.

[0329] System configuration

[0330] This system consists of a guest terminal, a server, sensors in the room, and an emotion engine.

[0331] 1. User (guest) terminal

[0332] Before checking out, guests use a smartphone app to take images of specific locations in their room (at their feet, under the bed, inside the refrigerator, inside the closet). The captured image data is temporarily stored on the device and sent to a server via a secure protocol (e.g., HTTPS). An emotion engine is installed on the device, which analyzes the user's voice and facial expressions to recognize their emotions.

[0333] 2. Server

[0334] The server receives the image data sent from the device and stores it in an internal database. The received image data is sent to an AI module, which uses image analysis technology to detect items (lost items) in the image. The server also collects data from sensors in the room (weight sensors, power connection sensors, etc.) and combines this data with the results of the image analysis. Finally, it determines whether any items have been left behind and notifies the guest's device and the front desk of the results.

[0335] 3. In-room sensors

[0336] Weight sensors inside the refrigerator detect whether there are any items left inside, and power connection sensors detect chargers and other electronic devices that are still plugged in.

[0337] 4. Emotion Engine

[0338] The emotion engine recognizes the user's emotions by analyzing their voice and facial expressions. If the user is stressed, it displays a warning message urging them to check for forgotten items. If the user is relaxed, it displays a reminder to help ensure a smooth checkout.

[0339] Specific examples

[0340] For example, a user launches a room inspection app and takes a photo following the instruction, "First, check under the bed." Next, they are instructed to "check inside the refrigerator," and take a photo of the inside of the refrigerator. At this time, the device sends this image data to the server. The server receives the image data and analyzes it using an AI module, while also collecting data from sensors in the room for integrated analysis. For example, if image analysis reveals a charger in the closet, the server notifies this information to the front desk and the user's device. The user receives this notification and actually checks the closet, finds the charger, and retrieves it. During this time, the emotion engine observes the user, and if it determines that the user is stressed, it displays a warning message saying, "You may have left something behind. Please check." If the user is relaxed, it displays a reminder such as, "You've checked everything, good job."

[0341] Prompt Sentence Examples

[0342] "Please tell me more about the app that helps you check for forgotten items when checking out of a room. I'd especially like to know about the notification feature that takes user emotions into consideration."

[0343] "Please tell me how the system works to prevent guests from leaving their belongings when checking out of a hotel. I'm interested in how image analysis and emotion recognition technology are used."

[0344] "Please explain with specific examples the operating procedures of a lost property prevention system used in lodging facilities and how the emotion engine works."

[0345] In this way, the present invention combines an emotion engine with AI image analysis to provide flexible responses according to the user's condition, thereby improving the efficiency of lost property responses in accommodation facilities and improving the user experience.

[0346] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0347] Step 1:

[0348] Before checking out, the user launches the app and takes images of specific locations in the room (at their feet, under the bed, inside the refrigerator, inside the closet).

[0349] Input: User's smartphone device

[0350] Output: Image data for each location

[0351] How it works: The user follows the app's instructions to take pictures of each location with the camera. During this process, the device analyzes the user's voice and facial expressions with its emotion engine to recognize their current emotional state.

[0352] Step 2:

[0353] The device sends the captured image data to the server using HTTPS.

[0354] Input: photographed image data, user emotion data

[0355] Output: Image data and emotion data sent to the server

[0356] Specific operation: Image data and emotion data are temporarily stored on the device and then transferred to the server via the appropriate protocol. During the transfer, a progress bar and a completion message are displayed to inform the user.

[0357] Step 3:

[0358] The server stores the received image data in a database and simultaneously sends it to the AI ​​module.

[0359] Input: Image data sent from the device

[0360] Output: Image data sent to the AI ​​module

[0361] Specific operation: The server receives the image data and automatically stores it in a database. It then passes the data to the AI ​​module to begin analysis.

[0362] Step 4:

[0363] The server's AI module analyzes the image data and detects objects within the image.

[0364] Input: Image data passed to the AI ​​module

[0365] Output: Information about the detected item

[0366] How it works: The AI ​​module analyzes patterns and features in the image to identify potentially lost items (e.g., chargers, clothing, household items).

[0367] Step 5:

[0368] The server collects data from sensors in the room and integrates it with the image analysis results.

[0369] Input: Data from sensors, analysis results of AI modules

[0370] Output: Consolidated data

[0371] How it works: The server collects data from weight sensors and power connection sensors inside the refrigerator and combines it with image analysis results to create a single integrated data set.

[0372] Step 6:

[0373] The server determines whether any items have been left behind and notifies the terminal and front desk of the results.

[0374] Input: Integrated data

[0375] Output: Notification of judgment results (terminal and front-end management system)

[0376] Specific operation: Based on the integrated data, it determines whether an item has been left behind. Once the determination result is generated, a notification is sent in real time to the front desk and the user's device. For example, a message such as "There is a charger in the closet. Please check it" is displayed.

[0377] Step 7:

[0378] The emotion engine analyzes the user's emotional state and provides notification and warning messages accordingly.

[0379] Input: User's voice and facial expression data

[0380] Output: Sentiment-based notification message

[0381] Specific operation: The emotion engine analyzes the user's current emotional state, and if the user is stressed, it displays a warning message such as "You may have forgotten something, please check." Conversely, if the user is relaxed, it displays a reminder such as "You've checked everything, good job."

[0382] (Application example 2)

[0383] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0384] Preventing customers from forgetting items in physical stores and improving the customer experience are important issues. Conventional loss prevention systems have difficulty in providing real-time notifications and responding flexibly to the emotional state of customers, so there was a need to solve these issues.

[0385] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes a means for a customer to take an image of a specific location in the store, a means for transmitting the captured image data to the server, a means for the server to analyze the received image data and detect items in the image, a means for the server to collect data from sensors in the store, a means for the server to integrate the image analysis results with the data from the sensors and determine whether an item has been left behind, a means for notifying the customer of the determination result to their terminal, and a means for analyzing the customer's emotions and adjusting the content of the notification. This prevents customers from leaving things behind in physical stores and enables real-time notifications and flexible responses based on the customer's emotional state.

[0386] A "customer" is a person who purchases or browses products in a physical store.

[0387] "In-store" refers to the interior space of a physical store, the area where customers can move around.

[0388] "Means for taking images" refers to the method by which a customer takes an image of a particular location using a smartphone or similar device.

[0389] The "means for transmitting image data to a server" refers to a set of protocols and technologies for transmitting captured image data to a server via the Internet.

[0390] "Server" refers to a computer system that receives, stores, and analyzes image data and sensor data.

[0391] "Means for detecting items in images" refers to technologies or algorithms that use an AI module to recognize and detect specific items from received image data.

[0392] "Sensors" are devices such as weight sensors and camera sensors that are installed in stores to detect the presence or movement of items.

[0393] "Means of collecting data" refers to the methods and technologies used to input measurement data from sensors installed in the store into a server.

[0394] The "means for determining whether or not an item has been left behind" is a technology in which the server integrates the image analysis results and sensor data to determine whether or not the customer has left any items behind.

[0395] "Means for notifying the customer's device of the judgment result" refers to a technology or method by which the server notifies the customer's smartphone or similar device in real time based on the judgment result.

[0396] "Means for analyzing emotions and adjusting notification content" refers to methods and technologies that analyze the customer's voice and facial expressions and appropriately change the notification content based on the results.

[0397] The present invention provides a system for preventing customers from forgetting items in a physical store and improving the customer experience. Specific embodiments for implementing this system will be described below.

[0398] System configuration

[0399] This system consists of a customer's device (e.g., a smartphone), a server, sensors installed in the store, and an emotion engine.

[0400] Customer's device

[0401] Customers use a smartphone app to take pictures of specific locations in the store. The captured image data is sent to a server via a secure protocol (e.g., HTTPS). An emotion engine is installed on the device, which analyzes the customer's voice and facial expressions to recognize their emotions.

[0402] server

[0403] The server receives image data sent from the terminal and stores it in an internal database. It also has an AI module for analyzing the received image data. The AI ​​module uses image analysis technology to detect items (lost items) in the image. It also collects data from sensors installed in the store (weight sensors, camera sensors, etc.) and integrates this data with the results of image analysis. It determines whether an item has been left behind and notifies the customer's terminal of the results.

[0404] In-store sensors

[0405] Weight sensors detect whether an item is left in a specific location in the store, while camera sensors record customer behavior and movements relative to products.

[0406] Emotion Engine

[0407] The emotion engine recognizes the customer's emotions by analyzing their voice and facial expressions. If the customer is stressed, it displays a warning message urging them to check for forgotten items. If the customer is relaxed, it displays a reminder to check for forgotten items.

[0408] Program processing

[0409] Taking and sending images

[0410] The user launches the point app in the store and follows the instructions to take a picture of a specific location. The image is temporarily stored on the device and then sent to the server using HTTPS.

[0411] Image data analysis and sensor information integration

[0412] The server stores the received image data in a database and sends it to the AI ​​module for analysis. The AI ​​module detects specific items in the image (e.g., forgotten purchases or personal items left behind). The server also collects data from weight sensors and camera sensors installed in the store and integrates it with the image analysis results.

[0413] Determine whether or not there is anything lost and notify you

[0414] The server determines whether an item has been left behind based on image analysis and sensor information. The result of the determination is sent to the customer's device in real time. For example, a message will be displayed saying, "An item has been left behind in the showcase. Please check."

[0415] Emotion recognition and notification content adjustment using an emotion engine

[0416] The emotion engine installed on the device analyzes the customer's voice and facial expressions to recognize their emotions. If the customer is stressed, the emotion engine displays a warning message to encourage them to check for forgotten items. If the customer is relaxed, the emotion engine responds by displaying a reminder to check.

[0417] Specific examples

[0418] A customer launches the smartphone app while in the store and follows the instructions to take a picture of a specific location. The device then sends this image data to a server, which analyzes it using an AI module. If the analysis results indicate that an item has been left behind in the display case, the server notifies the customer's device of this information. The user checks the notification and retrieves the item. During this time, the emotion engine analyzes the user's emotions, and if the user is feeling stressed, it displays a warning saying, "Did you forget to pick up an item?", or if the user is relaxed, it displays a reminder saying, "You've checked everything, good job."

[0419] Prompt Sentence Examples

[0420] "The weight sensor at the entrance reacted. Have you left anything behind?"

[0421] "Based on your recent purchasing behavior, you've forgotten a big-ticket item. Please check."

[0422] This will enable improved customer experience and more efficient operations in physical stores.

[0423] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0424] Step 1:

[0425] The user launches the smartphone app while in the store and follows the instructions to take a picture of a specific location.

[0426] Specific operation: When the app starts, it displays a guide to the user, instructing the location of the shooting point (e.g., shelf, showcase). The user then takes a picture of the specified location.

[0427] Input: An image of a specific location in the store.

[0428] Output: Image data temporarily stored on the smartphone.

[0429] Step 2:

[0430] The device sends the captured image data to the server using HTTPS.

[0431] Specific operation: After the user finishes taking a photo, the app automatically encrypts the image data and sends it to the server using a secure protocol (HTTPS).

[0432] Input: Image data stored on a smartphone.

[0433] Output: Image data sent to the server.

[0434] Step 3:

[0435] The server stores the received image data in a database and sends it to the AI ​​module to begin analysis.

[0436] Specific operation: When the server receives the image data, it stores it in a database and then passes the image data to an AI module (e.g., TensorFlow, PyTorch), which then begins image analysis.

[0437] Input: Image data sent to the server.

[0438] Output: Information on specific items detected through image analysis.

[0439] Step 4:

[0440] The server also collects data from weight sensors and camera sensors installed in the store and integrates it with the image analysis results.

[0441] Specific operation: The server periodically collects data from weight sensors and camera sensors, and combines it with analytical results to improve the accuracy of lost item detection.

[0442] Input: Data from sensors (weight sensor readings, camera sensor footage).

[0443] Output: The integrated result of determining whether or not an item has been lost.

[0444] Step 5:

[0445] The server determines whether any items have been left behind and notifies the customer's terminal of the result.

[0446] Specific operation: Based on the analysis results and sensor information, the server determines whether an item has been left behind and sends a real-time notification to the customer's device. The notification content may be something like, "An item has been left behind in the showcase. Please check it."

[0447] Input: Integrated judgement data.

[0448] Output: Notification message sent to customer's device.

[0449] Step 6:

[0450] An emotion engine installed on the device analyzes the customer's voice and facial expressions to recognize their emotions.

[0451] How it works: The device uses audio and camera sensors to collect the customer's facial expressions and voice, and then passes the data to an emotion engine (e.g., Microsoft Azure Cognitive Services), which then analyzes the data.

[0452] Input: Customer voice and facial expression data.

[0453] Output: Emotion recognition result (stress, relaxed, etc.).

[0454] Step 7:

[0455] The emotion engine adjusts notifications based on the customer's emotions, prompting them to check for lost items and displaying reminders.

[0456] Specific behavior: Based on the emotion recognition results, the device will display an appropriate notification. For example, if you are feeling stressed, it will display a warning saying, "Did you forget to pick up an item?", and if you are feeling relaxed, it will display a reminder saying, "You've checked everything, good job."

[0457] Input: Emotion recognition results.

[0458] Output: The adjusted notification message.

[0459] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0460] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0461] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0462] [Second embodiment]

[0463] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0464] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0465] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0466] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0467] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0468] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0469] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0470] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0471] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0472] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0473] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0474] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0475] The present invention provides a system and method for effectively preventing items left behind at check-out in hotels. This system detects items left behind in real time by combining image analysis and sensor technology, improving the operational efficiency of hotels. Specific embodiments of the present invention are described below.

[0476] System configuration

[0477] The system consists of a guest terminal, a server, and sensors in the room.

[0478] 1. User (guest) terminal

[0479] Before checking out, guests use a smartphone app to take pictures of specific areas in their room (at their feet, under the bed, inside the refrigerator, inside the closet).

[0480] The terminal transmits the captured image data to the server via a secure protocol (for example, HTTPS).

[0481] 2. Server

[0482] The server receives the image data sent from the terminal and stores it in an internal database.

[0483] It is equipped with an AI module for analyzing the received image data, which uses image analysis technology to detect objects in the image.

[0484] Data is collected from sensors installed in the room (e.g., weight sensors and power connection sensors), and this data is integrated with the results of image analysis.

[0485] The system determines whether any items have been left behind and notifies the guest's terminal and the front desk of the results.

[0486] 3. In-room sensors

[0487] A weight sensor inside the refrigerator detects whether there is anything left inside the refrigerator.

[0488] The power connection sensor detects chargers and other electronic devices that remain plugged in.

[0489] Program processing

[0490] A way for guests to take photos

[0491] Before checking out, users launch the app on their smartphone and follow the app's instructions to take pictures of specific points in the room (at their feet, under the bed, inside the refrigerator, inside the closet).The images are temporarily saved on the device.

[0492] Image data transmission means

[0493] The terminal transmits the captured image data to the server using a secure protocol (for example, HTTPS).

[0494] Image analysis and sensor information integration

[0495] The server stores the received image data in a database and sends it to the AI ​​module for analysis. The AI ​​module detects specific items in the image (e.g., chargers, clothing, household items). The server also collects data from weight sensors and power connection sensors installed in the room and integrates it with the image analysis results.

[0496] How to determine whether an item has been left behind and how to notify the person

[0497] The server determines whether any items have been left behind based on image analysis and sensor information. The results are sent in real time to the front desk management system and the guest's device. For example, a message such as "There is a charger in the closet. Please check." is displayed.

[0498] Specific examples

[0499] The user launches the room confirmation app and follows the instructions to take images of their feet, under the bed, inside the refrigerator, and inside the closet. The device sends this image data to a server, which analyzes it using an AI module. If the analysis detects a charger in the closet, the server notifies the front desk and the guest's device of this information, prompting them to check if they have left anything behind. The user confirms the notification and retrieves the forgotten item, completing the check-out process without any problems.

[0500] Through the above process, the present invention can improve the efficiency of lost property handling at accommodation facilities, reduce operational costs, and improve service quality.

[0501] The processing flow will be explained below.

[0502] Step 1:

[0503] Before checking out, the user launches the smartphone app in their accommodation room and selects the "Check Room" function.

[0504] Step 2:

[0505] The app prompts the user to take pictures of each point in turn: at their feet, under the bed, inside the refrigerator, and inside the closet.

[0506] Step 3:

[0507] The user follows the app's instructions and takes pictures of each point using their smartphone camera.

[0508] Step 4:

[0509] The device temporarily stores the captured image in its internal storage and prepares it for transmission to a server using a secure protocol (e.g., HTTPS).

[0510] Step 5:

[0511] The terminal transmits the captured image data to the server.

[0512] Step 6:

[0513] The server stores the received image data in a database.

[0514] Step 7:

[0515] The server calls the AI ​​module and begins image analysis using the image data as input.

[0516] Step 8:

[0517] The AI ​​module extracts features within the image and detects specific items (e.g., chargers, clothing, everyday items).

[0518] Step 9:

[0519] At the same time, the server collects data from weight sensors and power connection sensors installed in the room.

[0520] Step 10:

[0521] The server combines the image analysis results with data from the sensors to determine whether any items have been left behind.

[0522] Step 11:

[0523] The server notifies the judgment results in real time to the front management system and the user's terminal.

[0524] Step 12:

[0525] The device receives the notification from the server and displays a warning or confirmation message to the user about whether they have left something behind. For example, it displays "There is a charger in the closet. Please check."

[0526] Step 13:

[0527] The user checks the message and retrieves any items left in the room as necessary.

[0528] Step 14:

[0529] Once the user has confirmed that all of their lost items have been collected, they report this to the server via the app.

[0530] Step 15:

[0531] The server receives the user's report and notifies the front desk that the user has not left anything behind.

[0532] This allows users to check out with peace of mind and allows accommodation facilities to efficiently deal with lost items.

[0533] Example 1

[0534] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0535] Leaving something behind when checking out of a hotel is inconvenient for guests and incurs additional costs for the hotel. The present invention aims to provide a method and system for preventing guests from leaving something behind when checking out, efficiently and accurately detecting whether an item has been left behind, and quickly notifying guests and hotel staff. Another objective is to simplify the procedure for preventing guests from leaving something behind, thereby reducing the hassle for guests.

[0536] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0537] In this invention, the server includes means for a guest to take an image of a specific location in a room at an accommodation facility, means for transmitting the captured image data to the server, means for the server to analyze the received image data and detect items in the image, means for the server to collect data from sensors in the room, means for the server to integrate the image analysis results and the data from the sensors to determine whether an item has been left behind, and means for notifying the management system and the guest's terminal of the determination result. This allows guests to prevent leaving items in their rooms with a simple procedure when checking out, improves the operational efficiency of accommodation facilities, and reduces the cost of dealing with lost items.

[0538] "Guest" means a user staying at an accommodation facility.

[0539] "Accommodation facilities" are facilities that provide short-term stays, such as hotels and inns.

[0540] "Room" means a separate space within the accommodation facility where the guest stays.

[0541] "Means for taking images" refers to a method for obtaining images of a specified location using the camera function of a smartphone, tablet, etc.

[0542] "Image data" is digital data that includes information about a captured image.

[0543] A "server" is a computer system that receives, analyzes, and stores data sent from the guest's terminal.

[0544] "Transmission means" means a method for transferring data using the Internet and a secure communication protocol (e.g., HTTPS).

[0545] "Means for analyzing" refers to a method of processing received image data using analytical technology such as an AI module to detect specific items.

[0546] "Items" are individual belongings brought by guests, such as chargers, clothing, and daily necessities.

[0547] A "sensor" is a device that is installed in a room and measures weight and power connection status.

[0548] A "weight sensor" is a sensor that detects changes in weight when an object is placed on it.

[0549] A "power connection sensor" is a sensor that detects whether a charger or electronic device is connected to a power source.

[0550] "Means of collecting data" refers to the method of transferring information obtained from the sensor to the server.

[0551] "Means of integration" refers to a method of combining analysis results with sensor data to make a comprehensive judgment.

[0552] The "means for determining whether or not an item has been left behind" is a method for determining whether or not any items have been left behind in a room based on the image analysis results and sensor data.

[0553] The "means for notifying the determination result" is a method for reporting the presence or absence of a lost item to the guest and the front desk management system.

[0554] The "management system" is a computer system for managing the front desk operations of accommodation facilities.

[0555] The present invention provides a system and method for effectively preventing items from being left behind at check-out in accommodation facilities. This system combines image analysis technology and sensor technology to detect whether or not items have been left behind in real time, improving the operational efficiency of accommodation facilities.

[0556] System configuration

[0557] The system consists of the following elements:

[0558] 1. User's device

[0559] Before checking out, guests use a smartphone app to take pictures of specific areas in their room (at their feet, under the bed, inside the refrigerator, inside the closet).

[0560] The terminal transmits the captured image data to the server via a secure protocol (for example, HTTPS).

[0561] 2. Server

[0562] The server receives the image data sent from the terminal and stores it in an internal database.

[0563] It includes an AI module for analyzing the received image data, which uses image analysis technology to detect items (e.g., chargers or clothing) in the image.

[0564] The server collects data from sensors installed in the room (e.g., weight sensors and power connection sensors) and integrates this data with the results of image analysis.

[0565] The system determines whether any items have been left behind and notifies the guest's terminal and management system of the results.

[0566] 3. In-room sensors

[0567] A weight sensor inside the refrigerator detects whether there is anything left inside the refrigerator.

[0568] The power connection sensor detects chargers and other electronic devices that remain plugged in.

[0569] Specific processing of the program

[0570] Before checking out, users launch the app on their smartphone and follow the app's instructions to take pictures of specific points in the room (at their feet, under the bed, inside the refrigerator, inside the closet).The images are temporarily saved on the device.

[0571] The terminal transmits the captured image data to the server using a secure protocol (for example, HTTPS).

[0572] The server stores the received image data in a database and sends it to the AI ​​module for analysis. The AI ​​module detects specific items in the image (e.g., chargers, clothing, household items). The server also collects data from weight sensors and power connection sensors installed in the room and integrates the image analysis results with this sensor data.

[0573] The server determines whether any items have been left behind based on image analysis and sensor information. The results are sent to the management system and the guest's device in real time. For example, a message such as "There is a charger in the closet. Please check." is displayed.

[0574] Specific examples

[0575] The user launches the room inspection app and follows the instructions to take images of the floor, under the bed, inside the refrigerator, and inside the closet. The device sends this image data to a server, which analyzes it using an AI module. If the analysis detects a charger in the closet, the server notifies the management system and the guest's device of this information, prompting the guest to check if they have left anything behind. The user confirms the notification and retrieves the forgotten item, completing the check-out process without any problems.

[0576] Example prompt sentence:

[0577] "Please explain how a hotel can capture specific images of a room upon check-out to detect lost items. Please include the specific steps, technology used, and examples."

[0578] Through the above process, the present invention can improve the efficiency of lost property handling at accommodation facilities, reduce operational costs, and improve service quality.

[0579] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0580] Step 1:

[0581] Before checking out, users launch the app on their smartphone and follow the app's instructions to take images of specific locations in the room (at their feet, under the bed, inside the refrigerator, inside the closet), which allows visual information about specific locations in the room to be collected.

[0582] Input: App instructions

[0583] Output: Captured image data

[0584] Specific operation: The user presses the camera button in the app to take a picture of a specific location, and the image is saved on the device.

[0585] Step 2:

[0586] The device sends the captured image data to the server using a secure protocol (e.g., HTTPS). At this stage, the image data is encrypted and sent securely.

[0587] Input: Captured image data

[0588] Output: Transfer of image data to the server

[0589] Specific operation: The device encrypts the image data using the HTTPS protocol and sends it to the server over the network.

[0590] Step 3:

[0591] The server stores the received image data in a database, then sends it to the AI ​​module to begin analysis, which uses image analysis technology to detect specific items.

[0592] Input: Received image data

[0593] Output: Analysis results (detection results of objects in the image)

[0594] Data processing: Image data is input into the AI ​​module for image segmentation and object detection.

[0595] How it works: The server reads the image data, and the AI ​​module identifies specific items in the image (e.g., chargers or clothing).

[0596] Step 4:

[0597] The server collects real-time data from sensors installed in the room (e.g., weight sensors and power connection sensors), and stores the sensor data in a database.

[0598] Input: Data from sensors

[0599] Output: Sensor data stored in a database

[0600] Data processing: Determine whether or not there is an item from the weight sensor data, and whether or not there is a device connected to a power source from the power connection sensor data.

[0601] Specific operation: A weight sensor inside the refrigerator detects whether something is left inside, and a power connection sensor detects whether a device such as a charger is connected.

[0602] Step 5:

[0603] The server combines the image analysis results with sensor data to determine whether an item has been left behind, and the AI ​​module's analysis results with sensor information to confirm whether an item has been left behind.

[0604] Input: Image analysis results, sensor data

[0605] Output: Result of lost item detection

[0606] Data calculation: Image analysis results are combined with sensor data to calculate the probability of a lost item being found.

[0607] Specific operation: The server compares the image analysis results with the sensor data and makes a determination such as "There is a charger in the closet."

[0608] Step 6:

[0609] The server notifies the management system and the user's terminal of the determination result in real time. The notification content is a specific message (for example, "There is a charger in the closet. Please check it.").

[0610] Input: Lost item detection result

[0611] Output: Information message

[0612] Specific operation: The server generates push notifications or emails based on the judgment results and sends them to the management system and the user's device.

[0613] (Application example 1)

[0614] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0615] There is a need to solve the problem of workers leaving parts or tools behind in their work area when they finish work in a factory. In large factories in particular, the effort required to check and find forgotten items increases, so a system that can efficiently detect lost items and notify workers is needed.

[0616] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0617] In this invention, the server includes: means for a guest to take an image of a specific location in a hotel room; means for transmitting the captured image data to the server; means for the server to analyze the received image data and detect items in the image; means for the server to collect data from sensors in the room; means for the server to combine the image analysis results and the sensor data to determine whether any items have been left behind; means for notifying the front desk and the guest's terminal of the determination result; means for a factory worker to take an image of a specific point in a specific work area; means for a robot to analyze and combine the sensor data and image data in the work area to detect left-behind parts or tools; and means for notifying the worker's terminal of the detection result. This makes it possible to efficiently and quickly detect items left behind in a factory and notify the worker.

[0618] "Guest" refers to a person using an accommodation facility.

[0619] "Means of capturing images of a specific location" refers to the process by which a user uses a smartphone or device to take a photo of a specific area.

[0620] "Server" refers to a central computing device that analyzes and stores received data and manages various processes.

[0621] A "sensor" refers to a device that detects physical data (such as weight or position) and outputs it as electronic data.

[0622] "Means for analyzing image data" refers to technology that identifies and detects specific items or structures based on received images.

[0623] "Means for determining whether an item has been left behind by integrating image analysis results and data from sensors" refers to the process of combining analyzed image information with sensor data to determine whether an item has been left behind.

[0624] "Means for notifying the front desk and guest terminals of the results of the determination" refers to a system that transmits information regarding whether or not an item has been left behind to the accommodation staff and guest terminals in real time.

[0625] "Factory workers" refers to employees who perform production or management work within a factory.

[0626] "Work area" refers to a specific area or section within a factory where specific work is carried out.

[0627] "Robot" refers to a programmable mechanical device that operates automatically to perform specific tasks.

[0628] "Means for detecting left-behind parts and tools" refers to technology that identifies parts and tools left behind in the work area after work is completed.

[0629] "Means of notification" refers to the communication methods and protocols used to convey specific information to users and related systems.

[0630] The present invention provides a system for efficiently detecting parts and tools left behind in work areas within a factory and notifying workers. This system combines image analysis technology and sensor technology to detect left-behind items in real time and improve work efficiency. Specific embodiments of the present invention are described below.

[0631] System configuration

[0632] The system consists of terminals for factory workers, a server, and robots and sensors within the work area.

[0633] 1. Factory worker terminals

[0634] Before finishing work, workers use their smartphones or tablets to take images of specific locations within their work area (on their desks, inside shelves, under the floor).

[0635] The device sends the captured image data to the server via a secure protocol (e.g., HTTPS).

[0636] 2. Server

[0637] The server stores the received image data in an internal database.

[0638] The server is equipped with an image analysis module based on a generative AI model, which analyzes the captured image data and detects parts and tools within the image.

[0639] Meanwhile, the server also collects data from sensors (weight sensors, position sensors) installed within the work area, and integrates this data with the image analysis results to determine whether any items have been left behind.

[0640] The results of the assessment are sent to the factory workers' terminals in real time.

[0641] 3. Robots and sensors in the work area

[0642] The robot is equipped with a camera that scans specific locations to capture real-time images, allowing it to automatically detect parts or tools left behind after a task is completed.

[0643] Weight sensors and position sensors are used as sensors, and this data is sent to a server.

[0644] Hardware and Software Use

[0645] Hardware: Factory robots (with cameras), sensors (weight sensors, position sensors)

[0646] Software: Image analysis software (TensorFlow, OpenCV), notification system (Firebase, Push notifications)

[0647] Process Overview

[0648] The server receives, stores, and analyzes image data sent from the terminal. A generative AI model using TensorFlow and OpenCV is used for the analysis to detect parts and tools in the image. The detection results are integrated with data from weight and position sensors to ultimately determine whether any items have been left behind. The results are then sent to the worker's terminal in real time via Firebase. This allows for quick and efficient confirmation of lost items in the work area.

[0649] Specific examples

[0650] After completing their work, factory workers use their smartphones to take pictures of their designated work areas. The images are sent to a server where they are analyzed by a generative AI model. At the same time, a robot scans the work area and captures real-time images. Based on this data, any lost items are identified and a notification is sent to the worker's smartphone.

[0651] Prompt Sentence Examples

[0652] "Implement a system to detect items left behind after work is completed. The system uses image analysis and sensor technology to automatically detect parts or tools left behind in the work area and notify the worker of the results in real time."

[0653] Through the above process, the present invention can improve the efficiency of dealing with lost items in factories and significantly improve work efficiency and safety.

[0654] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0655] Step 1:

[0656] A factory worker takes an image of a specific location in the work area (a work desk, inside a shelf, under the floor) using a smartphone or tablet. The input is the image data captured using the smartphone or tablet, and the output is the image data temporarily stored on the device.

[0657] Step 2:

[0658] The device sends the captured image data to the server using a secure protocol (e.g., HTTPS). The input in this step is the image data, and the output is the image data uploaded to the server.

[0659] Step 3:

[0660] The server stores the received image data in an internal database. The input in this step is the image data sent from the terminal, and the output is the image data stored in the database.

[0661] Step 4:

[0662] The server performs image analysis. It uses its generative AI model (using TensorFlow and OpenCV) to analyze the image data and detect specific parts and tools. The input is the image data stored in the database, and the output is the analysis results (a list of detected parts and tools).

[0663] Step 5:

[0664] The server collects data from sensors (weight sensors, position sensors) installed in the work area. The input in this step is the data measured by the sensors, and the output is the sensor data sent to the server.

[0665] Step 6:

[0666] The server combines the image analysis results and sensor data to determine whether an item has been left behind. The input is the analysis results and sensor data, and based on this, it calculates whether an item has been left behind. The output is the result of the lost item determination (whether an item has been left behind or not, and the name of the specific part or tool).

[0667] Step 7:

[0668] The server notifies the factory worker of the result via their smartphone or tablet. The server sends the result in real time using a notification system such as Firebase. The input is the result of the lost item detection, and the output is a notification message that is displayed on the worker's device.

[0669] Step 8:

[0670] The factory worker checks the notification and returns to the site to retrieve any items that have been left behind. The input is the notification message displayed on the terminal, and the output is the parts or tools that have actually been retrieved.

[0671] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0672] The present invention combines a system for preventing items left behind at check-out of accommodation facilities with an emotion engine that recognizes the user's emotions, providing notifications and instructions according to the user's state. This system can improve the efficiency of handling lost items and the user experience. Specific embodiments of the present invention are described below.

[0673] System configuration

[0674] This system consists of a guest terminal, a server, sensors in the room, and an emotion engine.

[0675] 1. User (guest) terminal

[0676] Before checking out, guests use a smartphone app to take pictures of specific areas in their room (at their feet, under the bed, inside the refrigerator, inside the closet).

[0677] The terminal transmits the captured image data to the server via a secure protocol (for example, HTTPS).

[0678] The device is equipped with an emotion engine that analyzes the user's voice and facial expressions to recognize their emotions.

[0679] 2. Server

[0680] The server receives the image data sent from the terminal and stores it in an internal database.

[0681] It is equipped with an AI module for analyzing the received image data, which uses image analysis technology to detect items (lost items) in the image.

[0682] Data is collected from sensors installed in the room (such as weight sensors and power connection sensors), and this data is integrated with the results of image analysis.

[0683] The system determines whether any items have been left behind and notifies the guest's terminal and the front desk of the results.

[0684] 3. In-room sensors

[0685] A weight sensor inside the refrigerator detects whether there is anything left inside the refrigerator.

[0686] The power connection sensor detects chargers and other electronic devices that remain plugged in.

[0687] 4. Emotion Engine

[0688] The emotion engine recognizes the user's emotions by analyzing their voice and facial expressions, and if the user is feeling stressed, a warning message is displayed urging them to check for forgotten items.

[0689] Program processing

[0690] A way for guests to take photos

[0691] Before checking out, users launch the app on their smartphone and follow the app's instructions to take pictures of specific points in the room (at their feet, under the bed, inside the refrigerator, inside the closet).The images are temporarily saved on the device.

[0692] Image data transmission means

[0693] The terminal transmits the captured image data to the server using a secure protocol (for example, HTTPS).

[0694] Image analysis and sensor information integration

[0695] The server stores the received image data in a database and sends it to the AI ​​module for analysis. The AI ​​module detects specific items in the image (e.g., chargers, clothing, household items). The server also collects data from weight sensors and power connection sensors installed in the room and integrates it with the image analysis results.

[0696] How to determine whether an item has been left behind and how to notify the person

[0697] The server determines whether any items have been left behind based on image analysis and sensor information. The results are sent in real time to the front desk management system and the guest's device. For example, a message will be displayed saying, "There is a charger in the closet. Please check."

[0698] Emotion recognition and notification using an emotion engine

[0699] The emotion engine installed on the device analyzes the user's voice and facial expressions to recognize their emotions. If the user is feeling stressed, the emotion engine displays a warning message to encourage them to check for forgotten items. Conversely, if the user is relaxed, the engine displays a reminder to help ensure a smooth checkout.

[0700] Specific examples

[0701] The user launches the room inspection app and follows the instructions to take pictures of their feet, under the bed, inside the refrigerator, and inside the closet. The device sends this image data to a server, which analyzes it using an AI module. If the analysis results indicate that a charger is found in the closet, the server notifies the front desk and the guest's device of this information. The user checks the notification and retrieves the forgotten item. During this time, the emotion engine analyzes the user's emotions, and if the user is feeling stressed, it displays a warning saying, "You may have left something behind. Please check," and if the user is relaxed, it displays a reminder saying, "You've checked everything. Good job."

[0702] In this way, by combining an emotion engine, the present invention enables flexible responses according to the user's condition, improving the efficiency of lost property responses at accommodation facilities while also improving the user experience.

[0703] The processing flow will be explained below.

[0704] Step 1:

[0705] Before checking out, the user launches the smartphone app in their accommodation room and selects the "Check Room" function.

[0706] Step 2:

[0707] The app prompts the user to take pictures of each point in turn: at their feet, under the bed, inside the refrigerator, and inside the closet.

[0708] Step 3:

[0709] The user follows the app's instructions and takes pictures of each point using their smartphone camera.

[0710] Step 4:

[0711] The device temporarily stores the captured image in its internal storage and prepares it for transmission to a server using a secure protocol (e.g., HTTPS).

[0712] Step 5:

[0713] The terminal transmits the captured image data to the server.

[0714] Step 6:

[0715] The server stores the received image data in a database.

[0716] Step 7:

[0717] The server calls the AI ​​module and begins image analysis using the image data as input.

[0718] Step 8:

[0719] The AI ​​module extracts features within the image and detects specific items (e.g., chargers, clothing, everyday items).

[0720] Step 9:

[0721] At the same time, the server collects data from weight sensors and power connection sensors installed in the room.

[0722] Step 10:

[0723] The server combines the image analysis results with data from the sensors to determine whether any items have been left behind.

[0724] Step 11:

[0725] The server notifies the judgment results in real time to the front management system and the user's terminal.

[0726] Step 12:

[0727] The device receives the notification from the server and displays a warning or confirmation message to the user about whether they have left something behind. For example, it displays "There is a charger in the closet. Please check."

[0728] Step 13:

[0729] The emotion engine analyzes the user's emotions from their voice and facial expressions, and if the user is feeling stressed, a warning message is displayed urging them to check for forgotten items. For example, it displays "You may have left something behind. Please check."

[0730] Step 14:

[0731] Conversely, if the emotion engine detects a relaxed state in the user, it will display a reminder to make sure they have not forgotten anything, for example, "You've checked everything, good job."

[0732] Step 15:

[0733] The user checks the message and retrieves any items left in the room as necessary.

[0734] Step 16:

[0735] Once the user has confirmed that all of their lost items have been collected, they report this to the server via the app.

[0736] Step 17:

[0737] The server receives the user's report and notifies the front desk that the user has not left anything behind.

[0738] This allows users to check out with peace of mind and allows accommodation facilities to efficiently deal with lost items.

[0739] Example 2

[0740] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0741] Discovering and dealing with items left behind at check-out has traditionally been a major challenge at hotels. Users often neglect to check their belongings when checking out, which can lead to items being left behind. Furthermore, depending on the guest's condition, stress or fatigue can lead to neglecting to check their belongings, creating a need for appropriate notifications and support tailored to the user's emotional state. To solve these issues, it is necessary to efficiently and accurately determine whether an item has been left behind, as well as to respond flexibly by taking the user's emotional state into account.

[0742] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0743] In this invention, the server includes means for allowing guests to take images of specific locations in their rooms, means for transmitting the captured image data to the server, means for the server to analyze the received image data and detect items in the images, means for the server to collect data from sensors in the room, means for the server to integrate the image analysis results with the sensor data to determine whether any items have been left behind, means for notifying the front desk and the guest's terminal of the determination result, a terminal including an emotion engine that recognizes emotions by analyzing the guest's voice and facial expressions, and means for providing notifications and warning messages according to the guest's emotional state. This allows guests to efficiently and accurately check their belongings when checking out and receive appropriate notifications and support according to their emotional state.

[0744] "Guest" refers to a user staying at an accommodation facility.

[0745] "Inside the room" refers to the interior of the room in which the guest stays within the accommodation facility.

[0746] "Specific locations" refers to areas within a hotel room where items are likely to be left behind upon check-out (such as at the feet, under the bed, in the refrigerator, in the closet, etc.).

[0747] "Means of taking images" refers to the process by which guests take photos of specific locations in their rooms using a camera-equipped device such as a smartphone or tablet.

[0748] "Means of transmission" refers to the method of transferring the captured image data to a server via a communication means such as the Internet. Specifically, this is done using a secure protocol (e.g., HTTPS).

[0749] "Server" refers to a computer system for receiving, storing, and analyzing image data.

[0750] "Means of analyzing and detecting items in images" refers to the process of recognizing and identifying items (lost items) from received image data using AI technology.

[0751] "Sensor" refers to a device installed in a room of a lodging facility, including a weight sensor and a power connection sensor, for detecting various information within the room.

[0752] "Means of collecting data" refers to the process of obtaining information from sensors installed in the room.

[0753] "Method of integrating to determine whether or not an item has been left behind" refers to the process of combining the results of image analysis with data obtained from sensors to determine whether or not an item has been left behind in the room.

[0754] "Means of notification" refers to the method of transmitting the results of the determination of whether or not an item has been left behind to the front desk and the guest's terminal.

[0755] An "emotion engine" refers to software or hardware that analyzes a user's voice and facial expressions to recognize their emotions.

[0756] "Emotional state" refers to the psychological state of the guest, such as stress or relaxation.

[0757] "Means for providing notification or warning messages" refers to the process of displaying appropriate messages depending on the emotional state of the guest.

[0758] MODE FOR CARRYING OUT THE INVENTION

[0759] The present invention is a system for preventing users from leaving items behind when checking out of accommodation facilities, and provides notifications and instructions according to the user's state by combining it with an emotion engine that recognizes the user's emotions. Specific embodiments of the present invention are described below.

[0760] System configuration

[0761] This system consists of a guest terminal, a server, sensors in the room, and an emotion engine.

[0762] 1. User (guest) terminal

[0763] Before checking out, guests use a smartphone app to take images of specific locations in their room (at their feet, under the bed, inside the refrigerator, inside the closet). The captured image data is temporarily stored on the device and sent to a server via a secure protocol (e.g., HTTPS). An emotion engine is installed on the device, which analyzes the user's voice and facial expressions to recognize their emotions.

[0764] 2. Server

[0765] The server receives the image data sent from the device and stores it in an internal database. The received image data is sent to an AI module, which uses image analysis technology to detect items (lost items) in the image. The server also collects data from sensors in the room (weight sensors, power connection sensors, etc.) and combines this data with the results of the image analysis. Finally, it determines whether any items have been left behind and notifies the guest's device and the front desk of the results.

[0766] 3. In-room sensors

[0767] Weight sensors inside the refrigerator detect whether there are any items left inside, and power connection sensors detect chargers and other electronic devices that are still plugged in.

[0768] 4. Emotion Engine

[0769] The emotion engine recognizes the user's emotions by analyzing their voice and facial expressions. If the user is stressed, it displays a warning message urging them to check for forgotten items. If the user is relaxed, it displays a reminder to help ensure a smooth checkout.

[0770] Specific examples

[0771] For example, a user launches a room inspection app and takes a photo following the instruction, "First, check under the bed." Next, they are instructed to "check inside the refrigerator," and take a photo of the inside of the refrigerator. At this time, the device sends this image data to the server. The server receives the image data and analyzes it using an AI module, while also collecting data from sensors in the room for integrated analysis. For example, if image analysis reveals a charger in the closet, the server notifies this information to the front desk and the user's device. The user receives this notification and actually checks the closet, finds the charger, and retrieves it. During this time, the emotion engine observes the user, and if it determines that the user is stressed, it displays a warning message saying, "You may have left something behind. Please check." If the user is relaxed, it displays a reminder such as, "You've checked everything, good job."

[0772] Prompt Sentence Examples

[0773] "Please tell me more about the app that helps you check for forgotten items when checking out of a room. I'd especially like to know about the notification feature that takes user emotions into consideration."

[0774] "Please tell me how the system works to prevent guests from leaving their belongings when checking out of a hotel. I'm interested in how image analysis and emotion recognition technology are used."

[0775] "Please explain with specific examples the operating procedures of a lost property prevention system used in lodging facilities and how the emotion engine works."

[0776] In this way, the present invention combines an emotion engine with AI image analysis to provide flexible responses according to the user's condition, thereby improving the efficiency of lost property responses in accommodation facilities and improving the user experience.

[0777] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0778] Step 1:

[0779] Before checking out, the user launches the app and takes images of specific locations in the room (at their feet, under the bed, inside the refrigerator, inside the closet).

[0780] Input: User's smartphone device

[0781] Output: Image data for each location

[0782] How it works: The user follows the app's instructions to take pictures of each location with the camera. During this process, the device analyzes the user's voice and facial expressions with its emotion engine to recognize their current emotional state.

[0783] Step 2:

[0784] The device sends the captured image data to the server using HTTPS.

[0785] Input: photographed image data, user emotion data

[0786] Output: Image data and emotion data sent to the server

[0787] Specific operation: Image data and emotion data are temporarily stored on the device and then transferred to the server via the appropriate protocol. During the transfer, a progress bar and a completion message are displayed to inform the user.

[0788] Step 3:

[0789] The server stores the received image data in a database and simultaneously sends it to the AI ​​module.

[0790] Input: Image data sent from the device

[0791] Output: Image data sent to the AI ​​module

[0792] Specific operation: The server receives the image data and automatically stores it in a database. It then passes the data to the AI ​​module to begin analysis.

[0793] Step 4:

[0794] The server's AI module analyzes the image data and detects objects within the image.

[0795] Input: Image data passed to the AI ​​module

[0796] Output: Information about the detected item

[0797] How it works: The AI ​​module analyzes patterns and features in the image to identify potentially lost items (e.g., chargers, clothing, household items).

[0798] Step 5:

[0799] The server collects data from sensors in the room and integrates it with the image analysis results.

[0800] Input: Data from sensors, analysis results of AI modules

[0801] Output: Consolidated data

[0802] How it works: The server collects data from weight sensors and power connection sensors inside the refrigerator and combines it with image analysis results to create a single integrated data set.

[0803] Step 6:

[0804] The server determines whether any items have been left behind and notifies the terminal and front desk of the results.

[0805] Input: Integrated data

[0806] Output: Notification of judgment results (terminal and front-end management system)

[0807] Specific operation: Based on the integrated data, it determines whether an item has been left behind. Once the determination result is generated, a notification is sent in real time to the front desk and the user's device. For example, a message such as "There is a charger in the closet. Please check it" is displayed.

[0808] Step 7:

[0809] The emotion engine analyzes the user's emotional state and provides notification and warning messages accordingly.

[0810] Input: User's voice and facial expression data

[0811] Output: Sentiment-based notification message

[0812] Specific operation: The emotion engine analyzes the user's current emotional state, and if the user is stressed, it displays a warning message such as "You may have forgotten something, please check." Conversely, if the user is relaxed, it displays a reminder such as "You've checked everything, good job."

[0813] (Application example 2)

[0814] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0815] Preventing customers from forgetting items in physical stores and improving the customer experience are important issues. Conventional loss prevention systems have difficulty in providing real-time notifications and responding flexibly to the emotional state of customers, so there was a need to solve these issues.

[0816] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes a means for a customer to take an image of a specific location in the store, a means for transmitting the captured image data to the server, a means for the server to analyze the received image data and detect items in the image, a means for the server to collect data from sensors in the store, a means for the server to integrate the image analysis results with the data from the sensors and determine whether an item has been left behind, a means for notifying the customer of the determination result to their terminal, and a means for analyzing the customer's emotions and adjusting the content of the notification. This prevents customers from leaving things behind in physical stores and enables real-time notifications and flexible responses based on the customer's emotional state.

[0817] A "customer" is a person who purchases or browses products in a physical store.

[0818] "In-store" refers to the interior space of a physical store, the area where customers can move around.

[0819] "Means for taking images" refers to the method by which a customer takes an image of a particular location using a smartphone or similar device.

[0820] The "means for transmitting image data to a server" refers to a set of protocols and technologies for transmitting captured image data to a server via the Internet.

[0821] "Server" refers to a computer system that receives, stores, and analyzes image data and sensor data.

[0822] "Means for detecting items in images" refers to technologies or algorithms that use an AI module to recognize and detect specific items from received image data.

[0823] "Sensors" are devices such as weight sensors and camera sensors that are installed in stores to detect the presence or movement of items.

[0824] "Means of collecting data" refers to the methods and technologies used to input measurement data from sensors installed in the store into a server.

[0825] The "means for determining whether or not an item has been left behind" is a technology in which the server integrates the image analysis results and sensor data to determine whether or not the customer has left any items behind.

[0826] "Means for notifying the customer's device of the judgment result" refers to a technology or method by which the server notifies the customer's smartphone or similar device in real time based on the judgment result.

[0827] "Means for analyzing emotions and adjusting notification content" refers to methods and technologies that analyze the customer's voice and facial expressions and appropriately change the notification content based on the results.

[0828] The present invention provides a system for preventing customers from forgetting items in a physical store and improving the customer experience. Specific embodiments for implementing this system will be described below.

[0829] System configuration

[0830] This system consists of a customer's device (e.g., a smartphone), a server, sensors installed in the store, and an emotion engine.

[0831] Customer's device

[0832] Customers use a smartphone app to take pictures of specific locations in the store. The captured image data is sent to a server via a secure protocol (e.g., HTTPS). An emotion engine is installed on the device, which analyzes the customer's voice and facial expressions to recognize their emotions.

[0833] server

[0834] The server receives image data sent from the terminal and stores it in an internal database. It also has an AI module for analyzing the received image data. The AI ​​module uses image analysis technology to detect items (lost items) in the image. It also collects data from sensors installed in the store (weight sensors, camera sensors, etc.) and integrates this data with the results of image analysis. It determines whether an item has been left behind and notifies the customer's terminal of the results.

[0835] In-store sensors

[0836] Weight sensors detect whether an item is left in a specific location in the store, while camera sensors record customer behavior and movements relative to products.

[0837] Emotion Engine

[0838] The emotion engine recognizes the customer's emotions by analyzing their voice and facial expressions. If the customer is stressed, it displays a warning message urging them to check for forgotten items. If the customer is relaxed, it displays a reminder to check for forgotten items.

[0839] Program processing

[0840] Taking and sending images

[0841] The user launches the point app in the store and follows the instructions to take a picture of a specific location. The image is temporarily stored on the device and then sent to the server using HTTPS.

[0842] Image data analysis and sensor information integration

[0843] The server stores the received image data in a database and sends it to the AI ​​module for analysis. The AI ​​module detects specific items in the image (e.g., forgotten purchases or personal items left behind). The server also collects data from weight sensors and camera sensors installed in the store and integrates it with the image analysis results.

[0844] Determine whether or not there is anything lost and notify you

[0845] The server determines whether an item has been left behind based on image analysis and sensor information. The result of the determination is sent to the customer's device in real time. For example, a message will be displayed saying, "An item has been left behind in the showcase. Please check."

[0846] Emotion recognition and notification content adjustment using an emotion engine

[0847] The emotion engine installed on the device analyzes the customer's voice and facial expressions to recognize their emotions. If the customer is stressed, the emotion engine displays a warning message to encourage them to check for forgotten items. If the customer is relaxed, the emotion engine responds by displaying a reminder to check.

[0848] Specific examples

[0849] A customer launches the smartphone app while in the store and follows the instructions to take a picture of a specific location. The device then sends this image data to a server, which analyzes it using an AI module. If the analysis results indicate that an item has been left behind in the display case, the server notifies the customer's device of this information. The user checks the notification and retrieves the item. During this time, the emotion engine analyzes the user's emotions, and if the user is feeling stressed, it displays a warning saying, "Did you forget to pick up an item?", or if the user is relaxed, it displays a reminder saying, "You've checked everything, good job."

[0850] Prompt Sentence Examples

[0851] "The weight sensor at the entrance reacted. Have you left anything behind?"

[0852] "Based on your recent purchasing behavior, you've forgotten a big-ticket item. Please check."

[0853] This will enable improved customer experience and more efficient operations in physical stores.

[0854] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0855] Step 1:

[0856] The user launches the smartphone app while in the store and follows the instructions to take a picture of a specific location.

[0857] Specific operation: When the app starts, it displays a guide to the user, instructing the location of the shooting point (e.g., shelf, showcase). The user then takes a picture of the specified location.

[0858] Input: An image of a specific location in the store.

[0859] Output: Image data temporarily stored on the smartphone.

[0860] Step 2:

[0861] The device sends the captured image data to the server using HTTPS.

[0862] Specific operation: After the user finishes taking a photo, the app automatically encrypts the image data and sends it to the server using a secure protocol (HTTPS).

[0863] Input: Image data stored on a smartphone.

[0864] Output: Image data sent to the server.

[0865] Step 3:

[0866] The server stores the received image data in a database and sends it to the AI ​​module to begin analysis.

[0867] Specific operation: When the server receives the image data, it stores it in a database and then passes the image data to an AI module (e.g., TensorFlow, PyTorch), which then begins image analysis.

[0868] Input: Image data sent to the server.

[0869] Output: Information on specific items detected through image analysis.

[0870] Step 4:

[0871] The server also collects data from weight sensors and camera sensors installed in the store and integrates it with the image analysis results.

[0872] Specific operation: The server periodically collects data from weight sensors and camera sensors, and combines it with analytical results to improve the accuracy of lost item detection.

[0873] Input: Data from sensors (weight sensor readings, camera sensor footage).

[0874] Output: The integrated result of determining whether or not an item has been lost.

[0875] Step 5:

[0876] The server determines whether any items have been left behind and notifies the customer's terminal of the result.

[0877] Specific operation: Based on the analysis results and sensor information, the server determines whether an item has been left behind and sends a real-time notification to the customer's device. The notification content may be something like, "An item has been left behind in the showcase. Please check it."

[0878] Input: Integrated judgement data.

[0879] Output: Notification message sent to customer's device.

[0880] Step 6:

[0881] An emotion engine installed on the device analyzes the customer's voice and facial expressions to recognize their emotions.

[0882] How it works: The device uses audio and camera sensors to collect the customer's facial expressions and voice, and then passes the data to an emotion engine (e.g., Microsoft Azure Cognitive Services), which then analyzes the data.

[0883] Input: Customer voice and facial expression data.

[0884] Output: Emotion recognition result (stress, relaxed, etc.).

[0885] Step 7:

[0886] The emotion engine adjusts notifications based on the customer's emotions, prompting them to check for lost items and displaying reminders.

[0887] Specific behavior: Based on the emotion recognition results, the device will display an appropriate notification. For example, if you are feeling stressed, it will display a warning saying, "Did you forget to pick up an item?", and if you are feeling relaxed, it will display a reminder saying, "You've checked everything, good job."

[0888] Input: Emotion recognition results.

[0889] Output: The adjusted notification message.

[0890] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0891] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0892] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0893] [Third embodiment]

[0894] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0895] 5, the data processing system 310 includes the data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0896] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0897] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0898] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0899] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0900] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0901] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0902] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0903] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0904] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0905] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0906] The present invention provides a system and method for effectively preventing items left behind at check-out in hotels. This system detects items left behind in real time by combining image analysis and sensor technology, improving the operational efficiency of hotels. Specific embodiments of the present invention are described below.

[0907] System configuration

[0908] The system consists of a guest terminal, a server, and sensors in the room.

[0909] 1. User (guest) terminal

[0910] Before checking out, guests use a smartphone app to take pictures of specific areas in their room (at their feet, under the bed, inside the refrigerator, inside the closet).

[0911] The terminal transmits the captured image data to the server via a secure protocol (for example, HTTPS).

[0912] 2. Server

[0913] The server receives the image data sent from the terminal and stores it in an internal database.

[0914] It is equipped with an AI module for analyzing the received image data, which uses image analysis technology to detect objects in the image.

[0915] Data is collected from sensors installed in the room (e.g., weight sensors and power connection sensors), and this data is integrated with the results of image analysis.

[0916] The system determines whether any items have been left behind and notifies the guest's terminal and the front desk of the results.

[0917] 3. In-room sensors

[0918] A weight sensor inside the refrigerator detects whether there is anything left inside the refrigerator.

[0919] The power connection sensor detects chargers and other electronic devices that remain plugged in.

[0920] Program processing

[0921] A way for guests to take photos

[0922] Before checking out, users launch the app on their smartphone and follow the app's instructions to take pictures of specific points in the room (at their feet, under the bed, inside the refrigerator, inside the closet).The images are temporarily saved on the device.

[0923] Image data transmission means

[0924] The terminal transmits the captured image data to the server using a secure protocol (for example, HTTPS).

[0925] Image analysis and sensor information integration

[0926] The server stores the received image data in a database and sends it to the AI ​​module for analysis. The AI ​​module detects specific items in the image (e.g., chargers, clothing, household items). The server also collects data from weight sensors and power connection sensors installed in the room and integrates it with the image analysis results.

[0927] How to determine whether an item has been left behind and how to notify the person

[0928] The server determines whether any items have been left behind based on image analysis and sensor information. The results are sent in real time to the front desk management system and the guest's device. For example, a message such as "There is a charger in the closet. Please check." is displayed.

[0929] Specific examples

[0930] The user launches the room confirmation app and follows the instructions to take images of their feet, under the bed, inside the refrigerator, and inside the closet. The device sends this image data to a server, which analyzes it using an AI module. If the analysis detects a charger in the closet, the server notifies the front desk and the guest's device of this information, prompting them to check if they have left anything behind. The user confirms the notification and retrieves the forgotten item, completing the check-out process without any problems.

[0931] Through the above process, the present invention can improve the efficiency of lost property handling at accommodation facilities, reduce operational costs, and improve service quality.

[0932] The processing flow will be explained below.

[0933] Step 1:

[0934] Before checking out, the user launches the smartphone app in their accommodation room and selects the "Check Room" function.

[0935] Step 2:

[0936] The app prompts the user to take pictures of each point in turn: at their feet, under the bed, inside the refrigerator, and inside the closet.

[0937] Step 3:

[0938] The user follows the app's instructions and takes pictures of each point using their smartphone camera.

[0939] Step 4:

[0940] The device temporarily stores the captured image in its internal storage and prepares it for transmission to a server using a secure protocol (e.g., HTTPS).

[0941] Step 5:

[0942] The terminal transmits the captured image data to the server.

[0943] Step 6:

[0944] The server stores the received image data in a database.

[0945] Step 7:

[0946] The server calls the AI ​​module and begins image analysis using the image data as input.

[0947] Step 8:

[0948] The AI ​​module extracts features within the image and detects specific items (e.g., chargers, clothing, everyday items).

[0949] Step 9:

[0950] At the same time, the server collects data from weight sensors and power connection sensors installed in the room.

[0951] Step 10:

[0952] The server combines the image analysis results with data from the sensors to determine whether any items have been left behind.

[0953] Step 11:

[0954] The server notifies the judgment results in real time to the front management system and the user's terminal.

[0955] Step 12:

[0956] The device receives the notification from the server and displays a warning or confirmation message to the user about whether they have left something behind. For example, it displays "There is a charger in the closet. Please check."

[0957] Step 13:

[0958] The user checks the message and retrieves any items left in the room as necessary.

[0959] Step 14:

[0960] Once the user has confirmed that all of their lost items have been collected, they report this to the server via the app.

[0961] Step 15:

[0962] The server receives the user's report and notifies the front desk that the user has not left anything behind.

[0963] This allows users to check out with peace of mind and allows accommodation facilities to efficiently deal with lost items.

[0964] Example 1

[0965] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0966] Leaving something behind when checking out of a hotel is inconvenient for guests and incurs additional costs for the hotel. The present invention aims to provide a method and system for preventing guests from leaving something behind when checking out, efficiently and accurately detecting whether an item has been left behind, and quickly notifying guests and hotel staff. Another objective is to simplify the procedure for preventing guests from leaving something behind, thereby reducing the hassle for guests.

[0967] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0968] In this invention, the server includes means for a guest to take an image of a specific location in a room at an accommodation facility, means for transmitting the captured image data to the server, means for the server to analyze the received image data and detect items in the image, means for the server to collect data from sensors in the room, means for the server to integrate the image analysis results and the data from the sensors to determine whether an item has been left behind, and means for notifying the management system and the guest's terminal of the determination result. This allows guests to prevent leaving items in their rooms with a simple procedure when checking out, improves the operational efficiency of accommodation facilities, and reduces the cost of dealing with lost items.

[0969] "Guest" means a user staying at an accommodation facility.

[0970] "Accommodation facilities" are facilities that provide short-term stays, such as hotels and inns.

[0971] "Room" means a separate space within the accommodation facility where the guest stays.

[0972] "Means for taking images" refers to a method for obtaining images of a specified location using the camera function of a smartphone, tablet, etc.

[0973] "Image data" is digital data that includes information about a captured image.

[0974] A "server" is a computer system that receives, analyzes, and stores data sent from the guest's terminal.

[0975] "Transmission means" means a method for transferring data using the Internet and a secure communication protocol (e.g., HTTPS).

[0976] "Means for analyzing" refers to a method of processing received image data using analytical technology such as an AI module to detect specific items.

[0977] "Items" are individual belongings brought by guests, such as chargers, clothing, and daily necessities.

[0978] A "sensor" is a device that is installed in a room and measures weight and power connection status.

[0979] A "weight sensor" is a sensor that detects changes in weight when an object is placed on it.

[0980] A "power connection sensor" is a sensor that detects whether a charger or electronic device is connected to a power source.

[0981] "Means of collecting data" refers to the method of transferring information obtained from the sensor to the server.

[0982] "Means of integration" refers to a method of combining analysis results with sensor data to make a comprehensive judgment.

[0983] The "means for determining whether or not an item has been left behind" is a method for determining whether or not any items have been left behind in a room based on the image analysis results and sensor data.

[0984] The "means for notifying the determination result" is a method for reporting the presence or absence of a lost item to the guest and the front desk management system.

[0985] The "management system" is a computer system for managing the front desk operations of accommodation facilities.

[0986] The present invention provides a system and method for effectively preventing items from being left behind at check-out in accommodation facilities. This system combines image analysis technology and sensor technology to detect whether or not items have been left behind in real time, improving the operational efficiency of accommodation facilities.

[0987] System configuration

[0988] The system consists of the following elements:

[0989] 1. User's device

[0990] Before checking out, guests use a smartphone app to take pictures of specific areas in their room (at their feet, under the bed, inside the refrigerator, inside the closet).

[0991] The terminal transmits the captured image data to the server via a secure protocol (for example, HTTPS).

[0992] 2. Server

[0993] The server receives the image data sent from the terminal and stores it in an internal database.

[0994] It includes an AI module for analyzing the received image data, which uses image analysis technology to detect items (e.g., chargers or clothing) in the image.

[0995] The server collects data from sensors installed in the room (e.g., weight sensors and power connection sensors) and integrates this data with the results of image analysis.

[0996] The system determines whether any items have been left behind and notifies the guest's terminal and management system of the results.

[0997] 3. In-room sensors

[0998] A weight sensor inside the refrigerator detects whether there is anything left inside the refrigerator.

[0999] The power connection sensor detects chargers and other electronic devices that remain plugged in.

[1000] Specific processing of the program

[1001] Before checking out, users launch the app on their smartphone and follow the app's instructions to take pictures of specific points in the room (at their feet, under the bed, inside the refrigerator, inside the closet).The images are temporarily saved on the device.

[1002] The terminal transmits the captured image data to the server using a secure protocol (for example, HTTPS).

[1003] The server stores the received image data in a database and sends it to the AI ​​module for analysis. The AI ​​module detects specific items in the image (e.g., chargers, clothing, household items). The server also collects data from weight sensors and power connection sensors installed in the room and integrates the image analysis results with this sensor data.

[1004] The server determines whether any items have been left behind based on image analysis and sensor information. The results are sent to the management system and the guest's device in real time. For example, a message such as "There is a charger in the closet. Please check." is displayed.

[1005] Specific examples

[1006] The user launches the room inspection app and follows the instructions to take images of the floor, under the bed, inside the refrigerator, and inside the closet. The device sends this image data to a server, which analyzes it using an AI module. If the analysis detects a charger in the closet, the server notifies the management system and the guest's device of this information, prompting the guest to check if they have left anything behind. The user confirms the notification and retrieves the forgotten item, completing the check-out process without any problems.

[1007] Example prompt sentence:

[1008] "Please explain how a hotel can capture specific images of a room upon check-out to detect lost items. Please include the specific steps, technology used, and examples."

[1009] Through the above process, the present invention can improve the efficiency of lost property handling at accommodation facilities, reduce operational costs, and improve service quality.

[1010] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1011] Step 1:

[1012] Before checking out, users launch the app on their smartphone and follow the app's instructions to take images of specific locations in the room (at their feet, under the bed, inside the refrigerator, inside the closet), which allows visual information about specific locations in the room to be collected.

[1013] Input: App instructions

[1014] Output: Captured image data

[1015] Specific operation: The user presses the camera button in the app to take a picture of a specific location, and the image is saved on the device.

[1016] Step 2:

[1017] The device sends the captured image data to the server using a secure protocol (e.g., HTTPS). At this stage, the image data is encrypted and sent securely.

[1018] Input: Captured image data

[1019] Output: Transfer of image data to the server

[1020] Specific operation: The device encrypts the image data using the HTTPS protocol and sends it to the server over the network.

[1021] Step 3:

[1022] The server stores the received image data in a database, then sends it to the AI ​​module to begin analysis, which uses image analysis technology to detect specific items.

[1023] Input: Received image data

[1024] Output: Analysis results (detection results of objects in the image)

[1025] Data processing: Image data is input into the AI ​​module for image segmentation and object detection.

[1026] How it works: The server reads the image data, and the AI ​​module identifies specific items in the image (e.g., chargers or clothing).

[1027] Step 4:

[1028] The server collects real-time data from sensors installed in the room (e.g., weight sensors and power connection sensors), and stores the sensor data in a database.

[1029] Input: Data from sensors

[1030] Output: Sensor data stored in a database

[1031] Data processing: Determine whether or not there is an item from the weight sensor data, and whether or not there is a device connected to a power source from the power connection sensor data.

[1032] Specific operation: A weight sensor inside the refrigerator detects whether something is left inside, and a power connection sensor detects whether a device such as a charger is connected.

[1033] Step 5:

[1034] The server combines the image analysis results with sensor data to determine whether an item has been left behind, and the AI ​​module's analysis results with sensor information to confirm whether an item has been left behind.

[1035] Input: Image analysis results, sensor data

[1036] Output: Result of lost item detection

[1037] Data calculation: Image analysis results are combined with sensor data to calculate the probability of a lost item being found.

[1038] Specific operation: The server compares the image analysis results with the sensor data and makes a determination such as "There is a charger in the closet."

[1039] Step 6:

[1040] The server notifies the management system and the user's terminal of the determination result in real time. The notification content is a specific message (for example, "There is a charger in the closet. Please check it.").

[1041] Input: Lost item detection result

[1042] Output: Information message

[1043] Specific operation: The server generates push notifications or emails based on the judgment results and sends them to the management system and the user's device.

[1044] (Application example 1)

[1045] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1046] There is a need to solve the problem of workers leaving parts or tools behind in their work area when they finish work in a factory. In large factories in particular, the effort required to check and find forgotten items increases, so a system that can efficiently detect lost items and notify workers is needed.

[1047] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1048] In this invention, the server includes: means for a guest to take an image of a specific location in a hotel room; means for transmitting the captured image data to the server; means for the server to analyze the received image data and detect items in the image; means for the server to collect data from sensors in the room; means for the server to combine the image analysis results and the sensor data to determine whether any items have been left behind; means for notifying the front desk and the guest's terminal of the determination result; means for a factory worker to take an image of a specific point in a specific work area; means for a robot to analyze and combine the sensor data and image data in the work area to detect left-behind parts or tools; and means for notifying the worker's terminal of the detection result. This makes it possible to efficiently and quickly detect items left behind in a factory and notify the worker.

[1049] "Guest" refers to a person using an accommodation facility.

[1050] "Means of capturing images of a specific location" refers to the process by which a user uses a smartphone or device to take a photo of a specific area.

[1051] "Server" refers to a central computing device that analyzes and stores received data and manages various processes.

[1052] A "sensor" refers to a device that detects physical data (such as weight or position) and outputs it as electronic data.

[1053] "Means for analyzing image data" refers to technology that identifies and detects specific items or structures based on received images.

[1054] "Means for determining whether an item has been left behind by integrating image analysis results and data from sensors" refers to the process of combining analyzed image information with sensor data to determine whether an item has been left behind.

[1055] "Means for notifying the front desk and guest terminals of the results of the determination" refers to a system that transmits information regarding whether or not an item has been left behind to the accommodation staff and guest terminals in real time.

[1056] "Factory workers" refers to employees who perform production or management work within a factory.

[1057] "Work area" refers to a specific area or section within a factory where specific work is carried out.

[1058] "Robot" refers to a programmable mechanical device that operates automatically to perform specific tasks.

[1059] "Means for detecting left-behind parts and tools" refers to technology that identifies parts and tools left behind in the work area after work is completed.

[1060] "Means of notification" refers to the communication methods and protocols used to convey specific information to users and related systems.

[1061] The present invention provides a system for efficiently detecting parts and tools left behind in work areas within a factory and notifying workers. This system combines image analysis technology and sensor technology to detect left-behind items in real time and improve work efficiency. Specific embodiments of the present invention are described below.

[1062] System configuration

[1063] The system consists of terminals for factory workers, a server, and robots and sensors within the work area.

[1064] 1. Factory worker terminals

[1065] Before finishing work, workers use their smartphones or tablets to take images of specific locations within their work area (on their desks, inside shelves, under the floor).

[1066] The device sends the captured image data to the server via a secure protocol (e.g., HTTPS).

[1067] 2. Server

[1068] The server stores the received image data in an internal database.

[1069] The server is equipped with an image analysis module based on a generative AI model, which analyzes the captured image data and detects parts and tools within the image.

[1070] Meanwhile, the server also collects data from sensors (weight sensors, position sensors) installed within the work area, and integrates this data with the image analysis results to determine whether any items have been left behind.

[1071] The results of the assessment are sent to the factory workers' terminals in real time.

[1072] 3. Robots and sensors in the work area

[1073] The robot is equipped with a camera that scans specific locations to capture real-time images, allowing it to automatically detect parts or tools left behind after a task is completed.

[1074] Weight sensors and position sensors are used as sensors, and this data is sent to a server.

[1075] Hardware and Software Use

[1076] Hardware: Factory robots (with cameras), sensors (weight sensors, position sensors)

[1077] Software: Image analysis software (TensorFlow, OpenCV), notification system (Firebase, Push notifications)

[1078] Process Overview

[1079] The server receives, stores, and analyzes image data sent from the terminal. A generative AI model using TensorFlow and OpenCV is used for the analysis to detect parts and tools in the image. The detection results are integrated with data from weight and position sensors to ultimately determine whether any items have been left behind. The results are then sent to the worker's terminal in real time via Firebase. This allows for quick and efficient confirmation of lost items in the work area.

[1080] Specific examples

[1081] After completing their work, factory workers use their smartphones to take pictures of their designated work areas. The images are sent to a server where they are analyzed by a generative AI model. At the same time, a robot scans the work area and captures real-time images. Based on this data, any lost items are identified and a notification is sent to the worker's smartphone.

[1082] Prompt Sentence Examples

[1083] "Implement a system to detect items left behind after work is completed. The system uses image analysis and sensor technology to automatically detect parts or tools left behind in the work area and notify the worker of the results in real time."

[1084] Through the above process, the present invention can improve the efficiency of dealing with lost items in factories and significantly improve work efficiency and safety.

[1085] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1086] Step 1:

[1087] A factory worker takes an image of a specific location in the work area (a work desk, inside a shelf, under the floor) using a smartphone or tablet. The input is the image data captured using the smartphone or tablet, and the output is the image data temporarily stored on the device.

[1088] Step 2:

[1089] The device sends the captured image data to the server using a secure protocol (e.g., HTTPS). The input in this step is the image data, and the output is the image data uploaded to the server.

[1090] Step 3:

[1091] The server stores the received image data in an internal database. The input in this step is the image data sent from the terminal, and the output is the image data stored in the database.

[1092] Step 4:

[1093] The server performs image analysis. It uses its generative AI model (using TensorFlow and OpenCV) to analyze the image data and detect specific parts and tools. The input is the image data stored in the database, and the output is the analysis results (a list of detected parts and tools).

[1094] Step 5:

[1095] The server collects data from sensors (weight sensors, position sensors) installed in the work area. The input in this step is the data measured by the sensors, and the output is the sensor data sent to the server.

[1096] Step 6:

[1097] The server combines the image analysis results and sensor data to determine whether an item has been left behind. The input is the analysis results and sensor data, and based on this, it calculates whether an item has been left behind. The output is the result of the lost item determination (whether an item has been left behind or not, and the name of the specific part or tool).

[1098] Step 7:

[1099] The server notifies the factory worker of the result via their smartphone or tablet. The server sends the result in real time using a notification system such as Firebase. The input is the result of the lost item detection, and the output is a notification message that is displayed on the worker's device.

[1100] Step 8:

[1101] The factory worker checks the notification and returns to the site to retrieve any items that have been left behind. The input is the notification message displayed on the terminal, and the output is the parts or tools that have actually been retrieved.

[1102] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1103] The present invention combines a system for preventing items left behind at check-out of accommodation facilities with an emotion engine that recognizes the user's emotions, providing notifications and instructions according to the user's state. This system can improve the efficiency of handling lost items and the user experience. Specific embodiments of the present invention are described below.

[1104] System configuration

[1105] This system consists of a guest terminal, a server, sensors in the room, and an emotion engine.

[1106] 1. User (guest) terminal

[1107] Before checking out, guests use a smartphone app to take pictures of specific areas in their room (at their feet, under the bed, inside the refrigerator, inside the closet).

[1108] The terminal transmits the captured image data to the server via a secure protocol (for example, HTTPS).

[1109] The device is equipped with an emotion engine that analyzes the user's voice and facial expressions to recognize their emotions.

[1110] 2. Server

[1111] The server receives the image data sent from the terminal and stores it in an internal database.

[1112] It is equipped with an AI module for analyzing the received image data, which uses image analysis technology to detect items (lost items) in the image.

[1113] Data is collected from sensors installed in the room (such as weight sensors and power connection sensors), and this data is integrated with the results of image analysis.

[1114] The system determines whether any items have been left behind and notifies the guest's terminal and the front desk of the results.

[1115] 3. In-room sensors

[1116] A weight sensor inside the refrigerator detects whether there is anything left inside the refrigerator.

[1117] The power connection sensor detects chargers and other electronic devices that remain plugged in.

[1118] 4. Emotion Engine

[1119] The emotion engine recognizes the user's emotions by analyzing their voice and facial expressions, and if the user is feeling stressed, a warning message is displayed urging them to check for forgotten items.

[1120] Program processing

[1121] A way for guests to take photos

[1122] Before checking out, users launch the app on their smartphone and follow the app's instructions to take pictures of specific points in the room (at their feet, under the bed, inside the refrigerator, inside the closet).The images are temporarily saved on the device.

[1123] Image data transmission means

[1124] The terminal transmits the captured image data to the server using a secure protocol (for example, HTTPS).

[1125] Image analysis and sensor information integration

[1126] The server stores the received image data in a database and sends it to the AI ​​module for analysis. The AI ​​module detects specific items in the image (e.g., chargers, clothing, household items). The server also collects data from weight sensors and power connection sensors installed in the room and integrates it with the image analysis results.

[1127] How to determine whether an item has been left behind and how to notify the person

[1128] The server determines whether any items have been left behind based on image analysis and sensor information. The results are sent in real time to the front desk management system and the guest's device. For example, a message will be displayed saying, "There is a charger in the closet. Please check."

[1129] Emotion recognition and notification using an emotion engine

[1130] The emotion engine installed on the device analyzes the user's voice and facial expressions to recognize their emotions. If the user is feeling stressed, the emotion engine displays a warning message to encourage them to check for forgotten items. Conversely, if the user is relaxed, the engine displays a reminder to help ensure a smooth checkout.

[1131] Specific examples

[1132] The user launches the room inspection app and follows the instructions to take pictures of their feet, under the bed, inside the refrigerator, and inside the closet. The device sends this image data to a server, which analyzes it using an AI module. If the analysis results indicate that a charger is found in the closet, the server notifies the front desk and the guest's device of this information. The user checks the notification and retrieves the forgotten item. During this time, the emotion engine analyzes the user's emotions, and if the user is feeling stressed, it displays a warning saying, "You may have left something behind. Please check," and if the user is relaxed, it displays a reminder saying, "You've checked everything. Good job."

[1133] In this way, by combining an emotion engine, the present invention enables flexible responses according to the user's condition, improving the efficiency of lost property responses at accommodation facilities while also improving the user experience.

[1134] The processing flow will be explained below.

[1135] Step 1:

[1136] Before checking out, the user launches the smartphone app in their accommodation room and selects the "Check Room" function.

[1137] Step 2:

[1138] The app prompts the user to take pictures of each point in turn: at their feet, under the bed, inside the refrigerator, and inside the closet.

[1139] Step 3:

[1140] The user follows the app's instructions and takes pictures of each point using their smartphone camera.

[1141] Step 4:

[1142] The device temporarily stores the captured image in its internal storage and prepares it for transmission to a server using a secure protocol (e.g., HTTPS).

[1143] Step 5:

[1144] The terminal transmits the captured image data to the server.

[1145] Step 6:

[1146] The server stores the received image data in a database.

[1147] Step 7:

[1148] The server calls the AI ​​module and begins image analysis using the image data as input.

[1149] Step 8:

[1150] The AI ​​module extracts features within the image and detects specific items (e.g., chargers, clothing, everyday items).

[1151] Step 9:

[1152] At the same time, the server collects data from weight sensors and power connection sensors installed in the room.

[1153] Step 10:

[1154] The server combines the image analysis results with data from the sensors to determine whether any items have been left behind.

[1155] Step 11:

[1156] The server notifies the judgment results in real time to the front management system and the user's terminal.

[1157] Step 12:

[1158] The device receives the notification from the server and displays a warning or confirmation message to the user about whether they have left something behind. For example, it displays "There is a charger in the closet. Please check."

[1159] Step 13:

[1160] The emotion engine analyzes the user's emotions from their voice and facial expressions, and if the user is feeling stressed, a warning message is displayed urging them to check for forgotten items. For example, it displays "You may have left something behind. Please check."

[1161] Step 14:

[1162] Conversely, if the emotion engine detects a relaxed state in the user, it will display a reminder to make sure they have not forgotten anything, for example, "You've checked everything, good job."

[1163] Step 15:

[1164] The user checks the message and retrieves any items left in the room as necessary.

[1165] Step 16:

[1166] Once the user has confirmed that all of their lost items have been collected, they report this to the server via the app.

[1167] Step 17:

[1168] The server receives the user's report and notifies the front desk that the user has not left anything behind.

[1169] This allows users to check out with peace of mind and allows accommodation facilities to efficiently deal with lost items.

[1170] Example 2

[1171] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1172] Discovering and dealing with items left behind at check-out has traditionally been a major challenge at hotels. Users often neglect to check their belongings when checking out, which can lead to items being left behind. Furthermore, depending on the guest's condition, stress or fatigue can lead to neglecting to check their belongings, creating a need for appropriate notifications and support tailored to the user's emotional state. To solve these issues, it is necessary to efficiently and accurately determine whether an item has been left behind, as well as to respond flexibly by taking the user's emotional state into account.

[1173] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1174] In this invention, the server includes means for allowing guests to take images of specific locations in their rooms, means for transmitting the captured image data to the server, means for the server to analyze the received image data and detect items in the images, means for the server to collect data from sensors in the room, means for the server to integrate the image analysis results with the sensor data to determine whether any items have been left behind, means for notifying the front desk and the guest's terminal of the determination result, a terminal including an emotion engine that recognizes emotions by analyzing the guest's voice and facial expressions, and means for providing notifications and warning messages according to the guest's emotional state. This allows guests to efficiently and accurately check their belongings when checking out and receive appropriate notifications and support according to their emotional state.

[1175] "Guest" refers to a user staying at an accommodation facility.

[1176] "Inside the room" refers to the interior of the room in which the guest stays within the accommodation facility.

[1177] "Specific locations" refers to areas within a hotel room where items are likely to be left behind upon check-out (such as at the feet, under the bed, in the refrigerator, in the closet, etc.).

[1178] "Means of taking images" refers to the process by which guests take photos of specific locations in their rooms using a camera-equipped device such as a smartphone or tablet.

[1179] "Means of transmission" refers to the method of transferring the captured image data to a server via a communication means such as the Internet. Specifically, this is done using a secure protocol (e.g., HTTPS).

[1180] "Server" refers to a computer system for receiving, storing, and analyzing image data.

[1181] "Means of analyzing and detecting items in images" refers to the process of recognizing and identifying items (lost items) from received image data using AI technology.

[1182] "Sensor" refers to a device installed in a room of a lodging facility, including a weight sensor and a power connection sensor, for detecting various information within the room.

[1183] "Means of collecting data" refers to the process of obtaining information from sensors installed in the room.

[1184] "Method of integrating to determine whether or not an item has been left behind" refers to the process of combining the results of image analysis with data obtained from sensors to determine whether or not an item has been left behind in the room.

[1185] "Means of notification" refers to the method of transmitting the results of the determination of whether or not an item has been left behind to the front desk and the guest's terminal.

[1186] An "emotion engine" refers to software or hardware that analyzes a user's voice and facial expressions to recognize their emotions.

[1187] "Emotional state" refers to the psychological state of the guest, such as stress or relaxation.

[1188] "Means for providing notification or warning messages" refers to the process of displaying appropriate messages depending on the emotional state of the guest.

[1189] MODE FOR CARRYING OUT THE INVENTION

[1190] The present invention is a system for preventing users from leaving items behind when checking out of accommodation facilities, and provides notifications and instructions according to the user's state by combining it with an emotion engine that recognizes the user's emotions. Specific embodiments of the present invention are described below.

[1191] System configuration

[1192] This system consists of a guest terminal, a server, sensors in the room, and an emotion engine.

[1193] 1. User (guest) terminal

[1194] Before checking out, guests use a smartphone app to take images of specific locations in their room (at their feet, under the bed, inside the refrigerator, inside the closet). The captured image data is temporarily stored on the device and sent to a server via a secure protocol (e.g., HTTPS). An emotion engine is installed on the device, which analyzes the user's voice and facial expressions to recognize their emotions.

[1195] 2. Server

[1196] The server receives the image data sent from the device and stores it in an internal database. The received image data is sent to an AI module, which uses image analysis technology to detect items (lost items) in the image. The server also collects data from sensors in the room (weight sensors, power connection sensors, etc.) and combines this data with the results of the image analysis. Finally, it determines whether any items have been left behind and notifies the guest's device and the front desk of the results.

[1197] 3. In-room sensors

[1198] Weight sensors inside the refrigerator detect whether there are any items left inside, and power connection sensors detect chargers and other electronic devices that are still plugged in.

[1199] 4. Emotion Engine

[1200] The emotion engine recognizes the user's emotions by analyzing their voice and facial expressions. If the user is stressed, it displays a warning message urging them to check for forgotten items. If the user is relaxed, it displays a reminder to help ensure a smooth checkout.

[1201] Specific examples

[1202] For example, a user launches a room inspection app and takes a photo following the instruction, "First, check under the bed." Next, they are instructed to "check inside the refrigerator," and take a photo of the inside of the refrigerator. At this time, the device sends this image data to the server. The server receives the image data and analyzes it using an AI module, while also collecting data from sensors in the room for integrated analysis. For example, if image analysis reveals a charger in the closet, the server notifies this information to the front desk and the user's device. The user receives this notification and actually checks the closet, finds the charger, and retrieves it. During this time, the emotion engine observes the user, and if it determines that the user is stressed, it displays a warning message saying, "You may have left something behind. Please check." If the user is relaxed, it displays a reminder such as, "You've checked everything, good job."

[1203] Prompt Sentence Examples

[1204] "Please tell me more about the app that helps you check for forgotten items when checking out of a room. I'd especially like to know about the notification feature that takes user emotions into consideration."

[1205] "Please tell me how the system works to prevent guests from leaving their belongings when checking out of a hotel. I'm interested in how image analysis and emotion recognition technology are used."

[1206] "Please explain with specific examples the operating procedures of a lost property prevention system used in lodging facilities and how the emotion engine works."

[1207] In this way, the present invention combines an emotion engine with AI image analysis to provide flexible responses according to the user's condition, thereby improving the efficiency of lost property responses in accommodation facilities and improving the user experience.

[1208] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1209] Step 1:

[1210] Before checking out, the user launches the app and takes images of specific locations in the room (at their feet, under the bed, inside the refrigerator, inside the closet).

[1211] Input: User's smartphone device

[1212] Output: Image data for each location

[1213] How it works: The user follows the app's instructions to take pictures of each location with the camera. During this process, the device analyzes the user's voice and facial expressions with its emotion engine to recognize their current emotional state.

[1214] Step 2:

[1215] The device sends the captured image data to the server using HTTPS.

[1216] Input: photographed image data, user emotion data

[1217] Output: Image data and emotion data sent to the server

[1218] Specific operation: Image data and emotion data are temporarily stored on the device and then transferred to the server via the appropriate protocol. During the transfer, a progress bar and a completion message are displayed to inform the user.

[1219] Step 3:

[1220] The server stores the received image data in a database and simultaneously sends it to the AI ​​module.

[1221] Input: Image data sent from the device

[1222] Output: Image data sent to the AI ​​module

[1223] Specific operation: The server receives the image data and automatically stores it in a database. It then passes the data to the AI ​​module to begin analysis.

[1224] Step 4:

[1225] The server's AI module analyzes the image data and detects objects within the image.

[1226] Input: Image data passed to the AI ​​module

[1227] Output: Information about the detected item

[1228] How it works: The AI ​​module analyzes patterns and features in the image to identify potentially lost items (e.g., chargers, clothing, household items).

[1229] Step 5:

[1230] The server collects data from sensors in the room and integrates it with the image analysis results.

[1231] Input: Data from sensors, analysis results of AI modules

[1232] Output: Consolidated data

[1233] How it works: The server collects data from weight sensors and power connection sensors inside the refrigerator and combines it with image analysis results to create a single integrated data set.

[1234] Step 6:

[1235] The server determines whether any items have been left behind and notifies the terminal and front desk of the results.

[1236] Input: Integrated data

[1237] Output: Notification of judgment results (terminal and front-end management system)

[1238] Specific operation: Based on the integrated data, it determines whether an item has been left behind. Once the determination result is generated, a notification is sent in real time to the front desk and the user's device. For example, a message such as "There is a charger in the closet. Please check it" is displayed.

[1239] Step 7:

[1240] The emotion engine analyzes the user's emotional state and provides notification and warning messages accordingly.

[1241] Input: User's voice and facial expression data

[1242] Output: Sentiment-based notification message

[1243] Specific operation: The emotion engine analyzes the user's current emotional state, and if the user is stressed, it displays a warning message such as "You may have forgotten something, please check." Conversely, if the user is relaxed, it displays a reminder such as "You've checked everything, good job."

[1244] (Application example 2)

[1245] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1246] Preventing customers from forgetting items in physical stores and improving the customer experience are important issues. Conventional loss prevention systems have difficulty in providing real-time notifications and responding flexibly to the emotional state of customers, so there was a need to solve these issues.

[1247] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes a means for a customer to take an image of a specific location in the store, a means for transmitting the captured image data to the server, a means for the server to analyze the received image data and detect items in the image, a means for the server to collect data from sensors in the store, a means for the server to integrate the image analysis results with the data from the sensors and determine whether an item has been left behind, a means for notifying the customer of the determination result to their terminal, and a means for analyzing the customer's emotions and adjusting the content of the notification. This prevents customers from leaving things behind in physical stores and enables real-time notifications and flexible responses based on the customer's emotional state.

[1248] A "customer" is a person who purchases or browses products in a physical store.

[1249] "In-store" refers to the interior space of a physical store, the area where customers can move around.

[1250] "Means for taking images" refers to the method by which a customer takes an image of a particular location using a smartphone or similar device.

[1251] The "means for transmitting image data to a server" refers to a set of protocols and technologies for transmitting captured image data to a server via the Internet.

[1252] "Server" refers to a computer system that receives, stores, and analyzes image data and sensor data.

[1253] "Means for detecting items in images" refers to technologies or algorithms that use an AI module to recognize and detect specific items from received image data.

[1254] "Sensors" are devices such as weight sensors and camera sensors that are installed in stores to detect the presence or movement of items.

[1255] "Means of collecting data" refers to the methods and technologies used to input measurement data from sensors installed in the store into a server.

[1256] The "means for determining whether or not an item has been left behind" is a technology in which the server integrates the image analysis results and sensor data to determine whether or not the customer has left any items behind.

[1257] "Means for notifying the customer's device of the judgment result" refers to a technology or method by which the server notifies the customer's smartphone or similar device in real time based on the judgment result.

[1258] "Means for analyzing emotions and adjusting notification content" refers to methods and technologies that analyze the customer's voice and facial expressions and appropriately change the notification content based on the results.

[1259] The present invention provides a system for preventing customers from forgetting items in a physical store and improving the customer experience. Specific embodiments for implementing this system will be described below.

[1260] System configuration

[1261] This system consists of a customer's device (e.g., a smartphone), a server, sensors installed in the store, and an emotion engine.

[1262] Customer's device

[1263] Customers use a smartphone app to take pictures of specific locations in the store. The captured image data is sent to a server via a secure protocol (e.g., HTTPS). An emotion engine is installed on the device, which analyzes the customer's voice and facial expressions to recognize their emotions.

[1264] server

[1265] The server receives image data sent from the terminal and stores it in an internal database. It also has an AI module for analyzing the received image data. The AI ​​module uses image analysis technology to detect items (lost items) in the image. It also collects data from sensors installed in the store (weight sensors, camera sensors, etc.) and integrates this data with the results of image analysis. It determines whether an item has been left behind and notifies the customer's terminal of the results.

[1266] In-store sensors

[1267] Weight sensors detect whether an item is left in a specific location in the store, while camera sensors record customer behavior and movements relative to products.

[1268] Emotion Engine

[1269] The emotion engine recognizes the customer's emotions by analyzing their voice and facial expressions. If the customer is stressed, it displays a warning message urging them to check for forgotten items. If the customer is relaxed, it displays a reminder to check for forgotten items.

[1270] Program processing

[1271] Taking and sending images

[1272] The user launches the point app in the store and follows the instructions to take a picture of a specific location. The image is temporarily stored on the device and then sent to the server using HTTPS.

[1273] Image data analysis and sensor information integration

[1274] The server stores the received image data in a database and sends it to the AI ​​module for analysis. The AI ​​module detects specific items in the image (e.g., forgotten purchases or personal items left behind). The server also collects data from weight sensors and camera sensors installed in the store and integrates it with the image analysis results.

[1275] Determine whether or not there is anything lost and notify you

[1276] The server determines whether an item has been left behind based on image analysis and sensor information. The result of the determination is sent to the customer's device in real time. For example, a message will be displayed saying, "An item has been left behind in the showcase. Please check."

[1277] Emotion recognition and notification content adjustment using an emotion engine

[1278] The emotion engine installed on the device analyzes the customer's voice and facial expressions to recognize their emotions. If the customer is stressed, the emotion engine displays a warning message to encourage them to check for forgotten items. If the customer is relaxed, the emotion engine responds by displaying a reminder to check.

[1279] Specific examples

[1280] A customer launches the smartphone app while in the store and follows the instructions to take a picture of a specific location. The device then sends this image data to a server, which analyzes it using an AI module. If the analysis results indicate that an item has been left behind in the display case, the server notifies the customer's device of this information. The user checks the notification and retrieves the item. During this time, the emotion engine analyzes the user's emotions, and if the user is feeling stressed, it displays a warning saying, "Did you forget to pick up an item?", or if the user is relaxed, it displays a reminder saying, "You've checked everything, good job."

[1281] Prompt Sentence Examples

[1282] "The weight sensor at the entrance reacted. Have you left anything behind?"

[1283] "Based on your recent purchasing behavior, you've forgotten a big-ticket item. Please check."

[1284] This will enable improved customer experience and more efficient operations in physical stores.

[1285] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1286] Step 1:

[1287] The user launches the smartphone app while in the store and follows the instructions to take a picture of a specific location.

[1288] Specific operation: When the app starts, it displays a guide to the user, instructing the location of the shooting point (e.g., shelf, showcase). The user then takes a picture of the specified location.

[1289] Input: An image of a specific location in the store.

[1290] Output: Image data temporarily stored on the smartphone.

[1291] Step 2:

[1292] The device sends the captured image data to the server using HTTPS.

[1293] Specific operation: After the user finishes taking a photo, the app automatically encrypts the image data and sends it to the server using a secure protocol (HTTPS).

[1294] Input: Image data stored on a smartphone.

[1295] Output: Image data sent to the server.

[1296] Step 3:

[1297] The server stores the received image data in a database and sends it to the AI ​​module to begin analysis.

[1298] Specific operation: When the server receives the image data, it stores it in a database and then passes the image data to an AI module (e.g., TensorFlow, PyTorch), which then begins image analysis.

[1299] Input: Image data sent to the server.

[1300] Output: Information on specific items detected through image analysis.

[1301] Step 4:

[1302] The server also collects data from weight sensors and camera sensors installed in the store and integrates it with the image analysis results.

[1303] Specific operation: The server periodically collects data from weight sensors and camera sensors, and combines it with analytical results to improve the accuracy of lost item detection.

[1304] Input: Data from sensors (weight sensor readings, camera sensor footage).

[1305] Output: The integrated result of determining whether or not an item has been lost.

[1306] Step 5:

[1307] The server determines whether any items have been left behind and notifies the customer's terminal of the result.

[1308] Specific operation: Based on the analysis results and sensor information, the server determines whether an item has been left behind and sends a real-time notification to the customer's device. The notification content may be something like, "An item has been left behind in the showcase. Please check it."

[1309] Input: Integrated judgement data.

[1310] Output: Notification message sent to customer's device.

[1311] Step 6:

[1312] An emotion engine installed on the device analyzes the customer's voice and facial expressions to recognize their emotions.

[1313] How it works: The device uses audio and camera sensors to collect the customer's facial expressions and voice, and then passes the data to an emotion engine (e.g., Microsoft Azure Cognitive Services), which then analyzes the data.

[1314] Input: Customer voice and facial expression data.

[1315] Output: Emotion recognition result (stress, relaxed, etc.).

[1316] Step 7:

[1317] The emotion engine adjusts notifications based on the customer's emotions, prompting them to check for lost items and displaying reminders.

[1318] Specific behavior: Based on the emotion recognition results, the device will display an appropriate notification. For example, if you are feeling stressed, it will display a warning saying, "Did you forget to pick up an item?", and if you are feeling relaxed, it will display a reminder saying, "You've checked everything, good job."

[1319] Input: Emotion recognition results.

[1320] Output: The adjusted notification message.

[1321] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1322] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1323] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1324] [Fourth embodiment]

[1325] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1326] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1327] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1328] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1329] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1330] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1331] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1332] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1333] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1334] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1335] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1336] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1337] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1338] The present invention provides a system and method for effectively preventing items left behind at check-out in hotels. This system detects items left behind in real time by combining image analysis and sensor technology, improving the operational efficiency of hotels. Specific embodiments of the present invention are described below.

[1339] System configuration

[1340] The system consists of a guest terminal, a server, and sensors in the room.

[1341] 1. User (guest) terminal

[1342] Before checking out, guests use a smartphone app to take pictures of specific areas in their room (at their feet, under the bed, inside the refrigerator, inside the closet).

[1343] The terminal transmits the captured image data to the server via a secure protocol (for example, HTTPS).

[1344] 2. Server

[1345] The server receives the image data sent from the terminal and stores it in an internal database.

[1346] It is equipped with an AI module for analyzing the received image data, which uses image analysis technology to detect objects in the image.

[1347] Data is collected from sensors installed in the room (e.g., weight sensors and power connection sensors), and this data is integrated with the results of image analysis.

[1348] The system determines whether any items have been left behind and notifies the guest's terminal and the front desk of the results.

[1349] 3. In-room sensors

[1350] A weight sensor inside the refrigerator detects whether there is anything left inside the refrigerator.

[1351] The power connection sensor detects chargers and other electronic devices that remain plugged in.

[1352] Program processing

[1353] A way for guests to take photos

[1354] Before checking out, users launch the app on their smartphone and follow the app's instructions to take pictures of specific points in the room (at their feet, under the bed, inside the refrigerator, inside the closet).The images are temporarily saved on the device.

[1355] Image data transmission means

[1356] The terminal transmits the captured image data to the server using a secure protocol (for example, HTTPS).

[1357] Image analysis and sensor information integration

[1358] The server stores the received image data in a database and sends it to the AI ​​module for analysis. The AI ​​module detects specific items in the image (e.g., chargers, clothing, household items). The server also collects data from weight sensors and power connection sensors installed in the room and integrates it with the image analysis results.

[1359] How to determine whether an item has been left behind and how to notify the person

[1360] The server determines whether any items have been left behind based on image analysis and sensor information. The results are sent in real time to the front desk management system and the guest's device. For example, a message such as "There is a charger in the closet. Please check." is displayed.

[1361] Specific examples

[1362] The user launches the room confirmation app and follows the instructions to take images of their feet, under the bed, inside the refrigerator, and inside the closet. The device sends this image data to a server, which analyzes it using an AI module. If the analysis detects a charger in the closet, the server notifies the front desk and the guest's device of this information, prompting them to check if they have left anything behind. The user confirms the notification and retrieves the forgotten item, completing the check-out process without any problems.

[1363] Through the above process, the present invention can improve the efficiency of lost property handling at accommodation facilities, reduce operational costs, and improve service quality.

[1364] The processing flow will be explained below.

[1365] Step 1:

[1366] Before checking out, the user launches the smartphone app in their accommodation room and selects the "Check Room" function.

[1367] Step 2:

[1368] The app prompts the user to take pictures of each point in turn: at their feet, under the bed, inside the refrigerator, and inside the closet.

[1369] Step 3:

[1370] The user follows the app's instructions and takes pictures of each point using their smartphone camera.

[1371] Step 4:

[1372] The device temporarily stores the captured image in its internal storage and prepares it for transmission to a server using a secure protocol (e.g., HTTPS).

[1373] Step 5:

[1374] The terminal transmits the captured image data to the server.

[1375] Step 6:

[1376] The server stores the received image data in a database.

[1377] Step 7:

[1378] The server calls the AI ​​module and begins image analysis using the image data as input.

[1379] Step 8:

[1380] The AI ​​module extracts features within the image and detects specific items (e.g., chargers, clothing, everyday items).

[1381] Step 9:

[1382] At the same time, the server collects data from weight sensors and power connection sensors installed in the room.

[1383] Step 10:

[1384] The server combines the image analysis results with data from the sensors to determine whether any items have been left behind.

[1385] Step 11:

[1386] The server notifies the judgment results in real time to the front management system and the user's terminal.

[1387] Step 12:

[1388] The device receives the notification from the server and displays a warning or confirmation message to the user about whether they have left something behind. For example, it displays "There is a charger in the closet. Please check."

[1389] Step 13:

[1390] The user checks the message and retrieves any items left in the room as necessary.

[1391] Step 14:

[1392] Once the user has confirmed that all of their lost items have been collected, they report this to the server via the app.

[1393] Step 15:

[1394] The server receives the user's report and notifies the front desk that the user has not left anything behind.

[1395] This allows users to check out with peace of mind and allows accommodation facilities to efficiently deal with lost items.

[1396] Example 1

[1397] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1398] Leaving something behind when checking out of a hotel is inconvenient for guests and incurs additional costs for the hotel. The present invention aims to provide a method and system for preventing guests from leaving something behind when checking out, efficiently and accurately detecting whether an item has been left behind, and quickly notifying guests and hotel staff. Another objective is to simplify the procedure for preventing guests from leaving something behind, thereby reducing the hassle for guests.

[1399] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1400] In this invention, the server includes means for a guest to take an image of a specific location in a room at an accommodation facility, means for transmitting the captured image data to the server, means for the server to analyze the received image data and detect items in the image, means for the server to collect data from sensors in the room, means for the server to integrate the image analysis results and the data from the sensors to determine whether an item has been left behind, and means for notifying the management system and the guest's terminal of the determination result. This allows guests to prevent leaving items in their rooms with a simple procedure when checking out, improves the operational efficiency of accommodation facilities, and reduces the cost of dealing with lost items.

[1401] "Guest" means a user staying at an accommodation facility.

[1402] "Accommodation facilities" are facilities that provide short-term stays, such as hotels and inns.

[1403] "Room" means a separate space within the accommodation facility where the guest stays.

[1404] "Means for taking images" refers to a method for obtaining images of a specified location using the camera function of a smartphone, tablet, etc.

[1405] "Image data" is digital data that includes information about a captured image.

[1406] A "server" is a computer system that receives, analyzes, and stores data sent from the guest's terminal.

[1407] "Transmission means" means a method for transferring data using the Internet and a secure communication protocol (e.g., HTTPS).

[1408] "Means for analyzing" refers to a method of processing received image data using analytical technology such as an AI module to detect specific items.

[1409] "Items" are individual belongings brought by guests, such as chargers, clothing, and daily necessities.

[1410] A "sensor" is a device that is installed in a room and measures weight and power connection status.

[1411] A "weight sensor" is a sensor that detects changes in weight when an object is placed on it.

[1412] A "power connection sensor" is a sensor that detects whether a charger or electronic device is connected to a power source.

[1413] "Means of collecting data" refers to the method of transferring information obtained from the sensor to the server.

[1414] "Means of integration" refers to a method of combining analysis results with sensor data to make a comprehensive judgment.

[1415] The "means for determining whether or not an item has been left behind" is a method for determining whether or not any items have been left behind in a room based on the image analysis results and sensor data.

[1416] The "means for notifying the determination result" is a method for reporting the presence or absence of a lost item to the guest and the front desk management system.

[1417] The "management system" is a computer system for managing the front desk operations of accommodation facilities.

[1418] The present invention provides a system and method for effectively preventing items from being left behind at check-out in accommodation facilities. This system combines image analysis technology and sensor technology to detect whether or not items have been left behind in real time, improving the operational efficiency of accommodation facilities.

[1419] System configuration

[1420] The system consists of the following elements:

[1421] 1. User's device

[1422] Before checking out, guests use a smartphone app to take pictures of specific areas in their room (at their feet, under the bed, inside the refrigerator, inside the closet).

[1423] The terminal transmits the captured image data to the server via a secure protocol (for example, HTTPS).

[1424] 2. Server

[1425] The server receives the image data sent from the terminal and stores it in an internal database.

[1426] It includes an AI module for analyzing the received image data, which uses image analysis technology to detect items (e.g., chargers or clothing) in the image.

[1427] The server collects data from sensors installed in the room (e.g., weight sensors and power connection sensors) and integrates this data with the results of image analysis.

[1428] The system determines whether any items have been left behind and notifies the guest's terminal and management system of the results.

[1429] 3. In-room sensors

[1430] A weight sensor inside the refrigerator detects whether there is anything left inside the refrigerator.

[1431] The power connection sensor detects chargers and other electronic devices that remain plugged in.

[1432] Specific processing of the program

[1433] Before checking out, users launch the app on their smartphone and follow the app's instructions to take pictures of specific points in the room (at their feet, under the bed, inside the refrigerator, inside the closet).The images are temporarily saved on the device.

[1434] The terminal transmits the captured image data to the server using a secure protocol (for example, HTTPS).

[1435] The server stores the received image data in a database and sends it to the AI ​​module for analysis. The AI ​​module detects specific items in the image (e.g., chargers, clothing, household items). The server also collects data from weight sensors and power connection sensors installed in the room and integrates the image analysis results with this sensor data.

[1436] The server determines whether any items have been left behind based on image analysis and sensor information. The results are sent to the management system and the guest's device in real time. For example, a message such as "There is a charger in the closet. Please check." is displayed.

[1437] Specific examples

[1438] The user launches the room inspection app and follows the instructions to take images of the floor, under the bed, inside the refrigerator, and inside the closet. The device sends this image data to a server, which analyzes it using an AI module. If the analysis detects a charger in the closet, the server notifies the management system and the guest's device of this information, prompting the guest to check if they have left anything behind. The user confirms the notification and retrieves the forgotten item, completing the check-out process without any problems.

[1439] Example prompt sentence:

[1440] "Please explain how a hotel can capture specific images of a room upon check-out to detect lost items. Please include the specific steps, technology used, and examples."

[1441] Through the above process, the present invention can improve the efficiency of lost property handling at accommodation facilities, reduce operational costs, and improve service quality.

[1442] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1443] Step 1:

[1444] Before checking out, users launch the app on their smartphone and follow the app's instructions to take images of specific locations in the room (at their feet, under the bed, inside the refrigerator, inside the closet), which allows visual information about specific locations in the room to be collected.

[1445] Input: App instructions

[1446] Output: Captured image data

[1447] Specific operation: The user presses the camera button in the app to take a picture of a specific location, and the image is saved on the device.

[1448] Step 2:

[1449] The device sends the captured image data to the server using a secure protocol (e.g., HTTPS). At this stage, the image data is encrypted and sent securely.

[1450] Input: Captured image data

[1451] Output: Transfer of image data to the server

[1452] Specific operation: The device encrypts the image data using the HTTPS protocol and sends it to the server over the network.

[1453] Step 3:

[1454] The server stores the received image data in a database, then sends it to the AI ​​module to begin analysis, which uses image analysis technology to detect specific items.

[1455] Input: Received image data

[1456] Output: Analysis results (detection results of objects in the image)

[1457] Data processing: Image data is input into the AI ​​module for image segmentation and object detection.

[1458] How it works: The server reads the image data, and the AI ​​module identifies specific items in the image (e.g., chargers or clothing).

[1459] Step 4:

[1460] The server collects real-time data from sensors installed in the room (e.g., weight sensors and power connection sensors), and stores the sensor data in a database.

[1461] Input: Data from sensors

[1462] Output: Sensor data stored in a database

[1463] Data processing: Determine whether or not there is an item from the weight sensor data, and whether or not there is a device connected to a power source from the power connection sensor data.

[1464] Specific operation: A weight sensor inside the refrigerator detects whether something is left inside, and a power connection sensor detects whether a device such as a charger is connected.

[1465] Step 5:

[1466] The server combines the image analysis results with sensor data to determine whether an item has been left behind, and the AI ​​module's analysis results with sensor information to confirm whether an item has been left behind.

[1467] Input: Image analysis results, sensor data

[1468] Output: Result of lost item detection

[1469] Data calculation: Image analysis results are combined with sensor data to calculate the probability of a lost item being found.

[1470] Specific operation: The server compares the image analysis results with the sensor data and makes a determination such as "There is a charger in the closet."

[1471] Step 6:

[1472] The server notifies the management system and the user's terminal of the determination result in real time. The notification content is a specific message (for example, "There is a charger in the closet. Please check it.").

[1473] Input: Lost item detection result

[1474] Output: Information message

[1475] Specific operation: The server generates push notifications or emails based on the judgment results and sends them to the management system and the user's device.

[1476] (Application example 1)

[1477] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1478] There is a need to solve the problem of workers leaving parts or tools behind in their work area when they finish work in a factory. In large factories in particular, the effort required to check and find forgotten items increases, so a system that can efficiently detect lost items and notify workers is needed.

[1479] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1480] In this invention, the server includes: means for a guest to take an image of a specific location in a hotel room; means for transmitting the captured image data to the server; means for the server to analyze the received image data and detect items in the image; means for the server to collect data from sensors in the room; means for the server to combine the image analysis results and the sensor data to determine whether any items have been left behind; means for notifying the front desk and the guest's terminal of the determination result; means for a factory worker to take an image of a specific point in a specific work area; means for a robot to analyze and combine the sensor data and image data in the work area to detect left-behind parts or tools; and means for notifying the worker's terminal of the detection result. This makes it possible to efficiently and quickly detect items left behind in a factory and notify the worker.

[1481] "Guest" refers to a person using an accommodation facility.

[1482] "Means of capturing images of a specific location" refers to the process by which a user uses a smartphone or device to take a photo of a specific area.

[1483] "Server" refers to a central computing device that analyzes and stores received data and manages various processes.

[1484] A "sensor" refers to a device that detects physical data (such as weight or position) and outputs it as electronic data.

[1485] "Means for analyzing image data" refers to technology that identifies and detects specific items or structures based on received images.

[1486] "Means for determining whether an item has been left behind by integrating image analysis results and data from sensors" refers to the process of combining analyzed image information with sensor data to determine whether an item has been left behind.

[1487] "Means for notifying the front desk and guest terminals of the results of the determination" refers to a system that transmits information regarding whether or not an item has been left behind to the accommodation staff and guest terminals in real time.

[1488] "Factory workers" refers to employees who perform production or management work within a factory.

[1489] "Work area" refers to a specific area or section within a factory where specific work is carried out.

[1490] "Robot" refers to a programmable mechanical device that operates automatically to perform specific tasks.

[1491] "Means for detecting left-behind parts and tools" refers to technology that identifies parts and tools left behind in the work area after work is completed.

[1492] "Means of notification" refers to the communication methods and protocols used to convey specific information to users and related systems.

[1493] The present invention provides a system for efficiently detecting parts and tools left behind in work areas within a factory and notifying workers. This system combines image analysis technology and sensor technology to detect left-behind items in real time and improve work efficiency. Specific embodiments of the present invention are described below.

[1494] System configuration

[1495] The system consists of terminals for factory workers, a server, and robots and sensors within the work area.

[1496] 1. Factory worker terminals

[1497] Before finishing work, workers use their smartphones or tablets to take images of specific locations within their work area (on their desks, inside shelves, under the floor).

[1498] The device sends the captured image data to the server via a secure protocol (e.g., HTTPS).

[1499] 2. Server

[1500] The server stores the received image data in an internal database.

[1501] The server is equipped with an image analysis module based on a generative AI model, which analyzes the captured image data and detects parts and tools within the image.

[1502] Meanwhile, the server also collects data from sensors (weight sensors, position sensors) installed within the work area, and integrates this data with the image analysis results to determine whether any items have been left behind.

[1503] The results of the assessment are sent to the factory workers' terminals in real time.

[1504] 3. Robots and sensors in the work area

[1505] The robot is equipped with a camera that scans specific locations to capture real-time images, allowing it to automatically detect parts or tools left behind after a task is completed.

[1506] Weight sensors and position sensors are used as sensors, and this data is sent to a server.

[1507] Hardware and Software Use

[1508] Hardware: Factory robots (with cameras), sensors (weight sensors, position sensors)

[1509] Software: Image analysis software (TensorFlow, OpenCV), notification system (Firebase, Push notifications)

[1510] Process Overview

[1511] The server receives, stores, and analyzes image data sent from the terminal. A generative AI model using TensorFlow and OpenCV is used for the analysis to detect parts and tools in the image. The detection results are integrated with data from weight and position sensors to ultimately determine whether any items have been left behind. The results are then sent to the worker's terminal in real time via Firebase. This allows for quick and efficient confirmation of lost items in the work area.

[1512] Specific examples

[1513] After completing their work, factory workers use their smartphones to take pictures of their designated work areas. The images are sent to a server where they are analyzed by a generative AI model. At the same time, a robot scans the work area and captures real-time images. Based on this data, any lost items are identified and a notification is sent to the worker's smartphone.

[1514] Prompt Sentence Examples

[1515] "Implement a system to detect items left behind after work is completed. The system uses image analysis and sensor technology to automatically detect parts or tools left behind in the work area and notify the worker of the results in real time."

[1516] Through the above process, the present invention can improve the efficiency of dealing with lost items in factories and significantly improve work efficiency and safety.

[1517] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1518] Step 1:

[1519] A factory worker takes an image of a specific location in the work area (a work desk, inside a shelf, under the floor) using a smartphone or tablet. The input is the image data captured using the smartphone or tablet, and the output is the image data temporarily stored on the device.

[1520] Step 2:

[1521] The device sends the captured image data to the server using a secure protocol (e.g., HTTPS). The input in this step is the image data, and the output is the image data uploaded to the server.

[1522] Step 3:

[1523] The server stores the received image data in an internal database. The input in this step is the image data sent from the terminal, and the output is the image data stored in the database.

[1524] Step 4:

[1525] The server performs image analysis. It uses its generative AI model (using TensorFlow and OpenCV) to analyze the image data and detect specific parts and tools. The input is the image data stored in the database, and the output is the analysis results (a list of detected parts and tools).

[1526] Step 5:

[1527] The server collects data from sensors (weight sensors, position sensors) installed in the work area. The input in this step is the data measured by the sensors, and the output is the sensor data sent to the server.

[1528] Step 6:

[1529] The server combines the image analysis results and sensor data to determine whether an item has been left behind. The input is the analysis results and sensor data, and based on this, it calculates whether an item has been left behind. The output is the result of the lost item determination (whether an item has been left behind or not, and the name of the specific part or tool).

[1530] Step 7:

[1531] The server notifies the factory worker of the result via their smartphone or tablet. The server sends the result in real time using a notification system such as Firebase. The input is the result of the lost item detection, and the output is a notification message that is displayed on the worker's device.

[1532] Step 8:

[1533] The factory worker checks the notification and returns to the site to retrieve any items that have been left behind. The input is the notification message displayed on the terminal, and the output is the parts or tools that have actually been retrieved.

[1534] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1535] The present invention combines a system for preventing items left behind at check-out of accommodation facilities with an emotion engine that recognizes the user's emotions, providing notifications and instructions according to the user's state. This system can improve the efficiency of handling lost items and the user experience. Specific embodiments of the present invention are described below.

[1536] System configuration

[1537] This system consists of a guest terminal, a server, sensors in the room, and an emotion engine.

[1538] 1. User (guest) terminal

[1539] Before checking out, guests use a smartphone app to take pictures of specific areas in their room (at their feet, under the bed, inside the refrigerator, inside the closet).

[1540] The terminal transmits the captured image data to the server via a secure protocol (for example, HTTPS).

[1541] The device is equipped with an emotion engine that analyzes the user's voice and facial expressions to recognize their emotions.

[1542] 2. Server

[1543] The server receives the image data sent from the terminal and stores it in an internal database.

[1544] It is equipped with an AI module for analyzing the received image data, which uses image analysis technology to detect items (lost items) in the image.

[1545] Data is collected from sensors installed in the room (such as weight sensors and power connection sensors), and this data is integrated with the results of image analysis.

[1546] The system determines whether any items have been left behind and notifies the guest's terminal and the front desk of the results.

[1547] 3. In-room sensors

[1548] A weight sensor inside the refrigerator detects whether there is anything left inside the refrigerator.

[1549] The power connection sensor detects chargers and other electronic devices that remain plugged in.

[1550] 4. Emotion Engine

[1551] The emotion engine recognizes the user's emotions by analyzing their voice and facial expressions, and if the user is feeling stressed, a warning message is displayed urging them to check for forgotten items.

[1552] Program processing

[1553] A way for guests to take photos

[1554] Before checking out, users launch the app on their smartphone and follow the app's instructions to take pictures of specific points in the room (at their feet, under the bed, inside the refrigerator, inside the closet).The images are temporarily saved on the device.

[1555] Image data transmission means

[1556] The terminal transmits the captured image data to the server using a secure protocol (for example, HTTPS).

[1557] Image analysis and sensor information integration

[1558] The server stores the received image data in a database and sends it to the AI ​​module for analysis. The AI ​​module detects specific items in the image (e.g., chargers, clothing, household items). The server also collects data from weight sensors and power connection sensors installed in the room and integrates it with the image analysis results.

[1559] How to determine whether an item has been left behind and how to notify the person

[1560] The server determines whether any items have been left behind based on image analysis and sensor information. The results are sent in real time to the front desk management system and the guest's device. For example, a message will be displayed saying, "There is a charger in the closet. Please check."

[1561] Emotion recognition and notification using an emotion engine

[1562] The emotion engine installed on the device analyzes the user's voice and facial expressions to recognize their emotions. If the user is feeling stressed, the emotion engine displays a warning message to encourage them to check for forgotten items. Conversely, if the user is relaxed, the engine displays a reminder to help ensure a smooth checkout.

[1563] Specific examples

[1564] The user launches the room inspection app and follows the instructions to take pictures of their feet, under the bed, inside the refrigerator, and inside the closet. The device sends this image data to a server, which analyzes it using an AI module. If the analysis results indicate that a charger is found in the closet, the server notifies the front desk and the guest's device of this information. The user checks the notification and retrieves the forgotten item. During this time, the emotion engine analyzes the user's emotions, and if the user is feeling stressed, it displays a warning saying, "You may have left something behind. Please check," and if the user is relaxed, it displays a reminder saying, "You've checked everything. Good job."

[1565] In this way, by combining an emotion engine, the present invention enables flexible responses according to the user's condition, improving the efficiency of lost property responses at accommodation facilities while also improving the user experience.

[1566] The processing flow will be explained below.

[1567] Step 1:

[1568] Before checking out, the user launches the smartphone app in their accommodation room and selects the "Check Room" function.

[1569] Step 2:

[1570] The app prompts the user to take pictures of each point in turn: at their feet, under the bed, inside the refrigerator, and inside the closet.

[1571] Step 3:

[1572] The user follows the app's instructions and takes pictures of each point using their smartphone camera.

[1573] Step 4:

[1574] The device temporarily stores the captured image in its internal storage and prepares it for transmission to a server using a secure protocol (e.g., HTTPS).

[1575] Step 5:

[1576] The terminal transmits the captured image data to the server.

[1577] Step 6:

[1578] The server stores the received image data in a database.

[1579] Step 7:

[1580] The server calls the AI ​​module and begins image analysis using the image data as input.

[1581] Step 8:

[1582] The AI ​​module extracts features within the image and detects specific items (e.g., chargers, clothing, everyday items).

[1583] Step 9:

[1584] At the same time, the server collects data from weight sensors and power connection sensors installed in the room.

[1585] Step 10:

[1586] The server combines the image analysis results with data from the sensors to determine whether any items have been left behind.

[1587] Step 11:

[1588] The server notifies the judgment results in real time to the front management system and the user's terminal.

[1589] Step 12:

[1590] The device receives the notification from the server and displays a warning or confirmation message to the user about whether they have left something behind. For example, it displays "There is a charger in the closet. Please check."

[1591] Step 13:

[1592] The emotion engine analyzes the user's emotions from their voice and facial expressions, and if the user is feeling stressed, a warning message is displayed urging them to check for forgotten items. For example, it displays "You may have left something behind. Please check."

[1593] Step 14:

[1594] Conversely, if the emotion engine detects a relaxed state in the user, it will display a reminder to make sure they have not forgotten anything, for example, "You've checked everything, good job."

[1595] Step 15:

[1596] The user checks the message and retrieves any items left in the room as necessary.

[1597] Step 16:

[1598] Once the user has confirmed that all of their lost items have been collected, they report this to the server via the app.

[1599] Step 17:

[1600] The server receives the user's report and notifies the front desk that the user has not left anything behind.

[1601] This allows users to check out with peace of mind and allows accommodation facilities to efficiently deal with lost items.

[1602] Example 2

[1603] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1604] Discovering and dealing with items left behind at check-out has traditionally been a major challenge at hotels. Users often neglect to check their belongings when checking out, which can lead to items being left behind. Furthermore, depending on the guest's condition, stress or fatigue can lead to neglecting to check their belongings, creating a need for appropriate notifications and support tailored to the user's emotional state. To solve these issues, it is necessary to efficiently and accurately determine whether an item has been left behind, as well as to respond flexibly by taking the user's emotional state into account.

[1605] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1606] In this invention, the server includes means for allowing guests to take images of specific locations in their rooms, means for transmitting the captured image data to the server, means for the server to analyze the received image data and detect items in the images, means for the server to collect data from sensors in the room, means for the server to integrate the image analysis results with the sensor data to determine whether any items have been left behind, means for notifying the front desk and the guest's terminal of the determination result, a terminal including an emotion engine that recognizes emotions by analyzing the guest's voice and facial expressions, and means for providing notifications and warning messages according to the guest's emotional state. This allows guests to efficiently and accurately check their belongings when checking out and receive appropriate notifications and support according to their emotional state.

[1607] "Guest" refers to a user staying at an accommodation facility.

[1608] "Inside the room" refers to the interior of the room in which the guest stays within the accommodation facility.

[1609] "Specific locations" refers to areas within a hotel room where items are likely to be left behind upon check-out (such as at the feet, under the bed, in the refrigerator, in the closet, etc.).

[1610] "Means of taking images" refers to the process by which guests take photos of specific locations in their rooms using a camera-equipped device such as a smartphone or tablet.

[1611] "Means of transmission" refers to the method of transferring the captured image data to a server via a communication means such as the Internet. Specifically, this is done using a secure protocol (e.g., HTTPS).

[1612] "Server" refers to a computer system for receiving, storing, and analyzing image data.

[1613] "Means of analyzing and detecting items in images" refers to the process of recognizing and identifying items (lost items) from received image data using AI technology.

[1614] "Sensor" refers to a device installed in a room of a lodging facility, including a weight sensor and a power connection sensor, for detecting various information within the room.

[1615] "Means of collecting data" refers to the process of obtaining information from sensors installed in the room.

[1616] "Method of integrating to determine whether or not an item has been left behind" refers to the process of combining the results of image analysis with data obtained from sensors to determine whether or not an item has been left behind in the room.

[1617] "Means of notification" refers to the method of transmitting the results of the determination of whether or not an item has been left behind to the front desk and the guest's terminal.

[1618] An "emotion engine" refers to software or hardware that analyzes a user's voice and facial expressions to recognize their emotions.

[1619] "Emotional state" refers to the psychological state of the guest, such as stress or relaxation.

[1620] "Means for providing notification or warning messages" refers to the process of displaying appropriate messages depending on the emotional state of the guest.

[1621] MODE FOR CARRYING OUT THE INVENTION

[1622] The present invention is a system for preventing users from leaving items behind when checking out of accommodation facilities, and provides notifications and instructions according to the user's state by combining it with an emotion engine that recognizes the user's emotions. Specific embodiments of the present invention are described below.

[1623] System configuration

[1624] This system consists of a guest terminal, a server, sensors in the room, and an emotion engine.

[1625] 1. User (guest) terminal

[1626] Before checking out, guests use a smartphone app to take images of specific locations in their room (at their feet, under the bed, inside the refrigerator, inside the closet). The captured image data is temporarily stored on the device and sent to a server via a secure protocol (e.g., HTTPS). An emotion engine is installed on the device, which analyzes the user's voice and facial expressions to recognize their emotions.

[1627] 2. Server

[1628] The server receives the image data sent from the device and stores it in an internal database. The received image data is sent to an AI module, which uses image analysis technology to detect items (lost items) in the image. The server also collects data from sensors in the room (weight sensors, power connection sensors, etc.) and combines this data with the results of the image analysis. Finally, it determines whether any items have been left behind and notifies the guest's device and the front desk of the results.

[1629] 3. In-room sensors

[1630] Weight sensors inside the refrigerator detect whether there are any items left inside, and power connection sensors detect chargers and other electronic devices that are still plugged in.

[1631] 4. Emotion Engine

[1632] The emotion engine recognizes the user's emotions by analyzing their voice and facial expressions. If the user is stressed, it displays a warning message urging them to check for forgotten items. If the user is relaxed, it displays a reminder to help ensure a smooth checkout.

[1633] Specific examples

[1634] For example, a user launches a room inspection app and takes a photo following the instruction, "First, check under the bed." Next, they are instructed to "check inside the refrigerator," and take a photo of the inside of the refrigerator. At this time, the device sends this image data to the server. The server receives the image data and analyzes it using an AI module, while also collecting data from sensors in the room for integrated analysis. For example, if image analysis reveals a charger in the closet, the server notifies this information to the front desk and the user's device. The user receives this notification and actually checks the closet, finds the charger, and retrieves it. During this time, the emotion engine observes the user, and if it determines that the user is stressed, it displays a warning message saying, "You may have left something behind. Please check." If the user is relaxed, it displays a reminder such as, "You've checked everything, good job."

[1635] Prompt Sentence Examples

[1636] "Please tell me more about the app that helps you check for forgotten items when checking out of a room. I'd especially like to know about the notification feature that takes user emotions into consideration."

[1637] "Please tell me how the system works to prevent guests from leaving their belongings when checking out of a hotel. I'm interested in how image analysis and emotion recognition technology are used."

[1638] "Please explain with specific examples the operating procedures of a lost property prevention system used in lodging facilities and how the emotion engine works."

[1639] In this way, the present invention combines an emotion engine with AI image analysis to provide flexible responses according to the user's condition, thereby improving the efficiency of lost property responses in accommodation facilities and improving the user experience.

[1640] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1641] Step 1:

[1642] Before checking out, the user launches the app and takes images of specific locations in the room (at their feet, under the bed, inside the refrigerator, inside the closet).

[1643] Input: User's smartphone device

[1644] Output: Image data for each location

[1645] How it works: The user follows the app's instructions to take pictures of each location with the camera. During this process, the device analyzes the user's voice and facial expressions with its emotion engine to recognize their current emotional state.

[1646] Step 2:

[1647] The device sends the captured image data to the server using HTTPS.

[1648] Input: photographed image data, user emotion data

[1649] Output: Image data and emotion data sent to the server

[1650] Specific operation: Image data and emotion data are temporarily stored on the device and then transferred to the server via the appropriate protocol. During the transfer, a progress bar and a completion message are displayed to inform the user.

[1651] Step 3:

[1652] The server stores the received image data in a database and simultaneously sends it to the AI ​​module.

[1653] Input: Image data sent from the device

[1654] Output: Image data sent to the AI ​​module

[1655] Specific operation: The server receives the image data and automatically stores it in a database. It then passes the data to the AI ​​module to begin analysis.

[1656] Step 4:

[1657] The server's AI module analyzes the image data and detects objects within the image.

[1658] Input: Image data passed to the AI ​​module

[1659] Output: Information about the detected item

[1660] How it works: The AI ​​module analyzes patterns and features in the image to identify potentially lost items (e.g., chargers, clothing, household items).

[1661] Step 5:

[1662] The server collects data from sensors in the room and integrates it with the image analysis results.

[1663] Input: Data from sensors, analysis results of AI modules

[1664] Output: Consolidated data

[1665] How it works: The server collects data from weight sensors and power connection sensors inside the refrigerator and combines it with image analysis results to create a single integrated data set.

[1666] Step 6:

[1667] The server determines whether any items have been left behind and notifies the terminal and front desk of the results.

[1668] Input: Integrated data

[1669] Output: Notification of judgment results (terminal and front-end management system)

[1670] Specific operation: Based on the integrated data, it determines whether an item has been left behind. Once the determination result is generated, a notification is sent in real time to the front desk and the user's device. For example, a message such as "There is a charger in the closet. Please check it" is displayed.

[1671] Step 7:

[1672] The emotion engine analyzes the user's emotional state and provides notification and warning messages accordingly.

[1673] Input: User's voice and facial expression data

[1674] Output: Sentiment-based notification message

[1675] Specific operation: The emotion engine analyzes the user's current emotional state, and if the user is stressed, it displays a warning message such as "You may have forgotten something, please check." Conversely, if the user is relaxed, it displays a reminder such as "You've checked everything, good job."

[1676] (Application example 2)

[1677] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1678] Preventing customers from forgetting items in physical stores and improving the customer experience are important issues. Conventional loss prevention systems have difficulty in providing real-time notifications and responding flexibly to the emotional state of customers, so there was a need to solve these issues.

[1679] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes a means for a customer to take an image of a specific location in the store, a means for transmitting the captured image data to the server, a means for the server to analyze the received image data and detect items in the image, a means for the server to collect data from sensors in the store, a means for the server to integrate the image analysis results with the data from the sensors and determine whether an item has been left behind, a means for notifying the customer of the determination result to their terminal, and a means for analyzing the customer's emotions and adjusting the content of the notification. This prevents customers from leaving things behind in physical stores and enables real-time notifications and flexible responses based on the customer's emotional state.

[1680] A "customer" is a person who purchases or browses products in a physical store.

[1681] "In-store" refers to the interior space of a physical store, the area where customers can move around.

[1682] "Means for taking images" refers to the method by which a customer takes an image of a particular location using a smartphone or similar device.

[1683] The "means for transmitting image data to a server" refers to a set of protocols and technologies for transmitting captured image data to a server via the Internet.

[1684] "Server" refers to a computer system that receives, stores, and analyzes image data and sensor data.

[1685] "Means for detecting items in images" refers to technologies or algorithms that use an AI module to recognize and detect specific items from received image data.

[1686] "Sensors" are devices such as weight sensors and camera sensors that are installed in stores to detect the presence or movement of items.

[1687] "Means of collecting data" refers to the methods and technologies used to input measurement data from sensors installed in the store into a server.

[1688] The "means for determining whether or not an item has been left behind" is a technology in which the server integrates the image analysis results and sensor data to determine whether or not the customer has left any items behind.

[1689] "Means for notifying the customer's device of the judgment result" refers to a technology or method by which the server notifies the customer's smartphone or similar device in real time based on the judgment result.

[1690] "Means for analyzing emotions and adjusting notification content" refers to methods and technologies that analyze the customer's voice and facial expressions and appropriately change the notification content based on the results.

[1691] The present invention provides a system for preventing customers from forgetting items in a physical store and improving the customer experience. Specific embodiments for implementing this system will be described below.

[1692] System configuration

[1693] This system consists of a customer's device (e.g., a smartphone), a server, sensors installed in the store, and an emotion engine.

[1694] Customer's device

[1695] Customers use a smartphone app to take pictures of specific locations in the store. The captured image data is sent to a server via a secure protocol (e.g., HTTPS). An emotion engine is installed on the device, which analyzes the customer's voice and facial expressions to recognize their emotions.

[1696] server

[1697] The server receives image data sent from the terminal and stores it in an internal database. It also has an AI module for analyzing the received image data. The AI ​​module uses image analysis technology to detect items (lost items) in the image. It also collects data from sensors installed in the store (weight sensors, camera sensors, etc.) and integrates this data with the results of image analysis. It determines whether an item has been left behind and notifies the customer's terminal of the results.

[1698] In-store sensors

[1699] Weight sensors detect whether an item is left in a specific location in the store, while camera sensors record customer behavior and movements relative to products.

[1700] Emotion Engine

[1701] The emotion engine recognizes the customer's emotions by analyzing their voice and facial expressions. If the customer is stressed, it displays a warning message urging them to check for forgotten items. If the customer is relaxed, it displays a reminder to check for forgotten items.

[1702] Program processing

[1703] Taking and sending images

[1704] The user launches the point app in the store and follows the instructions to take a picture of a specific location. The image is temporarily stored on the device and then sent to the server using HTTPS.

[1705] Image data analysis and sensor information integration

[1706] The server stores the received image data in a database and sends it to the AI ​​module for analysis. The AI ​​module detects specific items in the image (e.g., forgotten purchases or personal items left behind). The server also collects data from weight sensors and camera sensors installed in the store and integrates it with the image analysis results.

[1707] Determine whether or not there is anything lost and notify you

[1708] The server determines whether an item has been left behind based on image analysis and sensor information. The result of the determination is sent to the customer's device in real time. For example, a message will be displayed saying, "An item has been left behind in the showcase. Please check."

[1709] Emotion recognition and notification content adjustment using an emotion engine

[1710] The emotion engine installed on the device analyzes the customer's voice and facial expressions to recognize their emotions. If the customer is stressed, the emotion engine displays a warning message to encourage them to check for forgotten items. If the customer is relaxed, the emotion engine responds by displaying a reminder to check.

[1711] Specific examples

[1712] A customer launches the smartphone app while in the store and follows the instructions to take a picture of a specific location. The device then sends this image data to a server, which analyzes it using an AI module. If the analysis results indicate that an item has been left behind in the display case, the server notifies the customer's device of this information. The user checks the notification and retrieves the item. During this time, the emotion engine analyzes the user's emotions, and if the user is feeling stressed, it displays a warning saying, "Did you forget to pick up an item?", or if the user is relaxed, it displays a reminder saying, "You've checked everything, good job."

[1713] Prompt Sentence Examples

[1714] "The weight sensor at the entrance reacted. Have you left anything behind?"

[1715] "Based on your recent purchasing behavior, you've forgotten a big-ticket item. Please check."

[1716] This will enable improved customer experience and more efficient operations in physical stores.

[1717] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1718] Step 1:

[1719] The user launches the smartphone app while in the store and follows the instructions to take a picture of a specific location.

[1720] Specific operation: When the app starts, it displays a guide to the user, instructing the location of the shooting point (e.g., shelf, showcase). The user then takes a picture of the specified location.

[1721] Input: An image of a specific location in the store.

[1722] Output: Image data temporarily stored on the smartphone.

[1723] Step 2:

[1724] The device sends the captured image data to the server using HTTPS.

[1725] Specific operation: After the user finishes taking a photo, the app automatically encrypts the image data and sends it to the server using a secure protocol (HTTPS).

[1726] Input: Image data stored on a smartphone.

[1727] Output: Image data sent to the server.

[1728] Step 3:

[1729] The server stores the received image data in a database and sends it to the AI ​​module to begin analysis.

[1730] Specific operation: When the server receives the image data, it stores it in a database and then passes the image data to an AI module (e.g., TensorFlow, PyTorch), which then begins image analysis.

[1731] Input: Image data sent to the server.

[1732] Output: Information on specific items detected through image analysis.

[1733] Step 4:

[1734] The server also collects data from weight sensors and camera sensors installed in the store and integrates it with the image analysis results.

[1735] Specific operation: The server periodically collects data from weight sensors and camera sensors, and combines it with analytical results to improve the accuracy of lost item detection.

[1736] Input: Data from sensors (weight sensor readings, camera sensor footage).

[1737] Output: The integrated result of determining whether or not an item has been lost.

[1738] Step 5:

[1739] The server determines whether any items have been left behind and notifies the customer's terminal of the result.

[1740] Specific operation: Based on the analysis results and sensor information, the server determines whether an item has been left behind and sends a real-time notification to the customer's device. The notification content may be something like, "An item has been left behind in the showcase. Please check it."

[1741] Input: Integrated judgement data.

[1742] Output: Notification message sent to customer's device.

[1743] Step 6:

[1744] An emotion engine installed on the device analyzes the customer's voice and facial expressions to recognize their emotions.

[1745] How it works: The device uses audio and camera sensors to collect the customer's facial expressions and voice, and then passes the data to an emotion engine (e.g., Microsoft Azure Cognitive Services), which then analyzes the data.

[1746] Input: Customer voice and facial expression data.

[1747] Output: Emotion recognition result (stress, relaxed, etc.).

[1748] Step 7:

[1749] The emotion engine adjusts notifications based on the customer's emotions, prompting them to check for lost items and displaying reminders.

[1750] Specific behavior: Based on the emotion recognition results, the device will display an appropriate notification. For example, if you are feeling stressed, it will display a warning saying, "Did you forget to pick up an item?", and if you are feeling relaxed, it will display a reminder saying, "You've checked everything, good job."

[1751] Input: Emotion recognition results.

[1752] Output: The adjusted notification message.

[1753] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1754] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1755] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1756] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1757] FIG. 9 illustrates an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and behaviors arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1758] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1759] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1760] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1761] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1762] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1763] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1764] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1765] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1766] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1767] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1768] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1769] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1770] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1771] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1772] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1773] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1774] The following is further disclosed regarding the above embodiment.

[1775] (Claim 1)

[1776] A means for a guest to take an image of a specific location in a room of the accommodation facility;

[1777] means for transmitting the captured image data to a server;

[1778] means for analyzing the image data received by the server and detecting an item in the image;

[1779] A means for the server to collect data from sensors in the room;

[1780] The server integrates the image analysis results and data from the sensors to determine whether an item has been left behind.

[1781] A means for notifying the determination result to the front desk and the guest's terminal;

[1782] A system including:

[1783] (Claim 2)

[1784] Further included is a means for displaying specific instructions to the guest when taking pictures of the inside of the room.

[1785] 10. The system of claim 1.

[1786] (Claim 3)

[1787] The sensor includes a weight sensor and a power supply connection sensor.

[1788] 10. The system of claim 1.

[1789] "Example 1"

[1790] (Claim 1)

[1791] A means for a guest to take an image of a specific location in a room of the accommodation facility;

[1792] means for transmitting the captured image data to a server;

[1793] means for analyzing the image data received by the server and detecting an item in the image;

[1794] A means for the server to collect data from sensors in the room;

[1795] The server integrates the image analysis results and data from the sensors to determine whether an item has been left behind.

[1796] a means for notifying the management system and the guest's terminal of the determination result;

[1797] A system including:

[1798] (Claim 2)

[1799] 10. The system according to claim 1, further comprising means for displaying specific instructions to a guest when taking an image of the inside of the room.

[1800] (Claim 3)

[1801] 10. The system of claim 1, wherein the sensors include a weight sensor and a power connection sensor.

[1802] "Application Example 1"

[1803] (Claim 1)

[1804] A means for a guest to take an image of a specific location in a room of the accommodation facility;

[1805] means for transmitting the captured image data to a server;

[1806] means for analyzing the image data received by the server and detecting an item in the image;

[1807] A means for the server to collect data from sensors in the room;

[1808] The server integrates the image analysis results and data from the sensors to determine whether an item has been left behind.

[1809] A means for notifying the determination result to the front desk and the guest's terminal;

[1810] A means for a factory worker to take an image of a specific point within a specific work area;

[1811] The robot analyzes and integrates sensor data and image data within the work area to detect any parts or tools that have been left behind.

[1812] a means for notifying a terminal of an operator of the detection result;

[1813] A system including:

[1814] (Claim 2)

[1815] 10. The system according to claim 1, further comprising means for displaying specific instructions to a guest when taking an image of the inside of the room.

[1816] (Claim 3)

[1817] 10. The system of claim 1, wherein the sensors include a weight sensor and a power connection sensor.

[1818] "Example 2: Combining Emotion Engines"

[1819] (Claim 1)

[1820] A means for a guest to take an image of a specific location in a room of the accommodation facility;

[1821] means for transmitting the captured image data to a server;

[1822] means for analyzing the image data received by the server and detecting an item in the image;

[1823] A means for the server to collect data from sensors in the room;

[1824] The server integrates the image analysis results and data from the sensors to determine whether an item has been left behind.

[1825] A means for notifying the determination result to the front desk and the guest's terminal;

[1826] A terminal that includes an emotion engine that analyzes the guest's voice and facial expressions to recognize their emotions, and

[1827] a means of providing notifications and warning messages depending on the emotional state of the guest;

[1828] A system including:

[1829] (Claim 2)

[1830] Further included is a means for displaying specific instructions to the guest when taking pictures of the inside of the room.

[1831] 10. The system of claim 1.

[1832] (Claim 3)

[1833] The sensor includes a weight sensor and a power supply connection sensor.

[1834] 10. The system of claim 1.

[1835] "Application example 2 when combining emotion engines"

[1836] (Claim 1)

[1837] A means for customers to take pictures of specific locations within the store;

[1838] means for transmitting the captured image data to a server;

[1839] means for analyzing the image data received by the server and detecting an item in the image;

[1840] A means for the server to collect data from sensors in the store;

[1841] The server integrates the image analysis results and data from the sensors to determine whether an item has been left behind.

[1842] means for notifying the customer terminal of the determination result;

[1843] A means to analyze customer sentiment and adjust notification content;

[1844] A system including:

[1845] (Claim 2)

[1846] Further includes means for displaying specific instructions to the customer when taking an image.

[1847] 10. The system of claim 1.

[1848] (Claim 3)

[1849] The sensor includes a weight sensor and a camera sensor.

[1850] 10. The system of claim 1. [Explanation of symbols]

[1851] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. A means for a guest to take an image of a specific location in a room of the accommodation facility; means for transmitting the captured image data to a server; means for analyzing the image data received by the server and detecting an item in the image; A means for the server to collect data from sensors in the room; The server integrates the image analysis results and data from the sensors to determine whether an item has been left behind. A means for notifying the determination result to the front desk and the guest's terminal; A system including:

2. Further included is a means for displaying specific instructions to the guest when taking pictures of the inside of the room. The system of claim 1 .

3. The sensor includes a weight sensor and a power supply connection sensor. The system of claim 1 .

Images