System

Abstract

A system is provided.SOLUTION: A system comprising: image capture means for recording activities of children; audio recording means for recording audio during the activities; data analysis means for analyzing the recorded image and audio data to extract important information; document generation means for automatically generating a report or communication based on the extracted information; confirmation and modification means for allowing a user to confirm and modify the document; and transmission means for transmitting the finally confirmed document to a parent.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] The work of childcare workers not only involves looking after children, but also a wide range of administrative tasks such as contacting parents and compiling reports on the children's status, resulting in a heavy workload. It is necessary to reduce this burden and improve smooth communication between childcare workers and parents. Specifically, a system is needed that efficiently records children's activities and quickly and accurately reports and contacts parents. [Means for solving the problem]

[0005] The present invention provides the following system. It includes an image capturing means for recording children's activities and an audio recording means for recording sounds made during the activities. It also includes a data analysis means for analyzing the recorded image and audio data to extract important information, and a document generation means for automatically generating reports or correspondence based on the extracted information. It also includes a confirmation and correction means that allows the user to confirm and correct the generated document, and a transmission means for sending the final confirmed document to parents. This provides a system that significantly reduces the administrative burden on childcare workers and enables quick and accurate reporting and communication to parents. Furthermore, by recording images and audio in real time and transmitting them to a server for analysis, it is possible to share information efficiently and in a timely manner.

[0006] "Image capture means" refers to cameras or other devices used to record children's activities in real time.

[0007] "Audio recording means" refers to devices such as microphones that record children's voices and surrounding sounds in real time during activities.

[0008] "Data analysis means" refers to software and hardware for analyzing recorded image and audio data to extract important information.

[0009] "Document generation means" refers to software for automatically generating reports or correspondence based on extracted information.

[0010] "Verification and correction means" refers to an interface that allows users to verify the automatically generated document and make corrections as necessary.

[0011] "Transmission Means" refers to the communications equipment and software used to transmit the finalized document to the parent.

[0012] "Server" refers to a computer system for receiving, analyzing, and storing recorded data.

[0013] "Devices" refers to mobile information devices such as smartphones and tablets used by childcare workers.

[0014] "Users" refer to the childcare workers who operate the system, record children's activities, and contact parents. [Brief explanation of the drawings]

[0015] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13]FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0016] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0017] First, the terms used in the following description will be explained.

[0018] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0019] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0020] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0021] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0022] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0023] [First embodiment]

[0024] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0025] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0026] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0027] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0028] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0029] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0030] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0031] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0032] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0033] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0034] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0035] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0036] The present invention is a system for reducing the workload of childcare workers and improving communication with parents. This system efficiently records children's activities and automatically generates reports and correspondence for parents. The program processing of the system and specific examples are described below.

[0037] System Overview

[0038] This system consists of a device held by the user (childcare worker), a cloud server, and a device held by the parent. The user uses the device to record the children's activities, and the server analyzes the data to automatically generate reports and correspondence. The final generated documents are checked and revised by the user before being sent to the parent.

[0039] Program processing

[0040] Starting and recording an activity

[0041] 1. User Action:

[0042] The user turns on the device and opens the app.

[0043] Press the "Start Activity" button in the app to begin recording.

[0044] 2. Device behavior:

[0045] The device activates the camera and microphone to record the children's activities in real time using video and audio.

[0046] The recorded data is sent to a server at regular intervals.

[0047] Data Analysis and Document Generation

[0048] 3. Server Operation:

[0049] The server analyzes the received video data and uses image recognition technology to identify important scenes (e.g., scenes of people playing on playground equipment).

[0050] Similarly, voice data is analyzed and specific keywords (e.g., "fun!", "lunch") are extracted using voice recognition technology.

[0051] Based on the analysis results, reports and messages to parents are automatically generated.

[0052] Checking, correcting and sending documents

[0053] 4. User confirmation and correction:

[0054] The user checks the automatically generated document on the terminal.

[0055] Modify the document as needed.

[0056] 5. Sending documents:

[0057] When the user presses the "send" button, the terminal sends the final verified document to the server.

[0058] The server sends the document to the parent via email or in-app message.

[0059] Specific examples

[0060] Example 1: Field trip record

[0061] Below is a specific example of when a user (a nursery teacher) takes children on a field trip.

[0062] 1. User Action:

[0063] The user turns on the device and opens the app to record the field trip.

[0064] Press "Start Activity" to begin recording.

[0065] 2. Device behavior:

[0066] The device uses a camera to record the children playing and simultaneously records audio using a microphone.

[0067] The recorded data is sent to a server at regular intervals.

[0068] 3. Server Operation:

[0069] The server analyzes the video data and extracts scenes such as "scenes of children playing on playground equipment" and "scenes of children gathering together."

[0070] The statements made by nursery teachers and children are converted into text from the audio data, and important phrases (e.g., "It's fun," "Be careful") are extracted.

[0071] A field trip report is automatically generated based on this information.

[0072] 4. User confirmation and correction:

[0073] The user checks the generated report on the terminal and makes corrections as necessary.

[0074] You can also add photos and additional comments to your report.

[0075] 5. Sending documents:

[0076] Once the report has been revised, click the "Send" button to send it to the parent.

[0077] The server will send the final report to the parent's email address or within the app.

[0078] The above is a form for implementing the invention of a system for reducing the workload of childcare workers and sharing information with parents quickly and accurately.

[0079] The processing flow will be explained below.

[0080] Step 1:

[0081] The user launches the app on their device and presses the "Start Activity" button.

[0082] How it works: To record their child's activity, the user launches the app and selects the "Start Activity" button.

[0083] Step 2:

[0084] The device will turn on the camera and microphone and begin recording.

[0085] Action Details: The device activates the camera and microphone and begins real-time video and audio recording.

[0086] Step 3:

[0087] The device stores recorded video and audio data locally.

[0088] Operation details: The device captures video and audio data in real time and stores it in local storage.

[0089] Step 4:

[0090] The terminal transmits the recorded data to the server at regular intervals.

[0091] Operation details: The device uploads collected video and audio data to the server at regular intervals.

[0092] Step 5:

[0093] The server analyzes the received video data and extracts important scenes.

[0094] How it works: The server uses image recognition technology to analyze the content of the video and identify specific activities or behaviors.

[0095] Step 6:

[0096] The server analyzes the received audio data and extracts specific keywords and phrases.

[0097] How it works: The server uses speech recognition technology to convert speech into text and extract key information.

[0098] Step 7:

[0099] The server automatically generates reports and correspondence based on the analysis results.

[0100] Operation details: The server automatically generates reports and correspondence based on the analyzed data and makes them available to the user for review.

[0101] Step 8:

[0102] The server sends the generated document to the terminal.

[0103] Operation details: The server sends the generated reports and messages to the terminal so that the user can check and edit them.

[0104] Step 9:

[0105] The user checks and modifies the document generated on the terminal.

[0106] Operation details: The user checks the automatically generated document on the device and corrects any necessary parts.

[0107] Step 10:

[0108] The user confirms the document after making corrections and presses the "Send" button.

[0109] Operation details: The user confirms the final confirmed document and presses the "Send" button to send it.

[0110] Step 11:

[0111] The terminal transmits the final confirmed document to the server.

[0112] Operation details: The terminal sends the modified document data to the server.

[0113] Step 12:

[0114] The server sends a final report or communication to the parent.

[0115] Operation details: The server sends the document to the parent's email address or via the in-app messaging function.

[0116] Step 13:

[0117] The parent receives and reviews the document.

[0118] How it works: Parents receive and review the documents sent to them, and can reply in-app or via email if they have any feedback or questions.

[0119] The above are the specific steps in the processing flow of the program.

[0120] Example 1

[0121] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0122] There is a need to reduce the workload of childcare workers and improve communication with parents. However, conventional manual recording and reporting tasks require a great deal of time and effort, placing a heavy burden on childcare workers. Furthermore, manual report creation is prone to subjectivity and can lack accuracy. The objective of this invention is to solve these problems and provide a system that enables efficient and accurate activity recording and reporting.

[0123] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0124] In this invention, the server includes a data analysis means, a document generation means, and a transmission means. This makes it possible to analyze received data and extract specific scenes. Furthermore, it is possible to automatically generate reports or communication messages based on the analysis results using a generative AI model, and the final confirmed documents can be quickly sent to the guardian.

[0125] The "image capturing means" is a video capturing device for recording children's activities in real time.

[0126] The "audio recording means" is a recording device for recording the voices of children during their activities.

[0127] The "transmission means" is a communication device or system for transmitting the recorded image and audio data to a cloud server via the Internet.

[0128] A "cloud server" is a remote server accessible via the Internet that performs processes such as data storage, analysis, and document generation.

[0129] "Data analysis means" refers to software or a system for analyzing received image and audio data and extracting specific scenes and important information.

[0130] "Document generation means" refers to software or systems such as generative AI models that automatically generate reports or correspondence based on analysis results.

[0131] "Verification and correction means" refers to the means by which a user can verify the automatically generated document and correct it if necessary.

[0132] "Transmission means" refers to a communication device or system for transmitting the finalized document to the parent.

[0133] The present invention is a system for reducing the workload of childcare workers and improving communication with parents. This system efficiently records children's activities and automatically generates reports and correspondence for parents. Specific embodiments of the system are described below.

[0134] This system consists of a device held by the user (childcare worker), a cloud server, and a device held by the parent. The user uses the device to record the children's activities, and the cloud server analyzes the data to automatically generate reports and correspondence. The final generated documents are checked and revised by the user before being sent to the parent.

[0135] Hardware and software used

[0136] Device: Smartphone or tablet (e.g. iPad) used by the childcare worker

[0137] Cloud server: A cloud server running on AWS (Amazon Web Services)

[0138] Software components:

[0139] Image recognition technology: OpenCV

[0140] Speech recognition technology: Google Cloud Speech-to-Text

[0141] Natural Language Generation: GPT-3 (generative AI model)

[0142] Communication method: Firebase

[0143] A detailed description of the system's program processing

[0144] 1. To record the activities of the children under the user's care, the user starts up the device and opens a dedicated app (e.g., a childcare activity recording app). By tapping the "Start Activity" button, the device prepares to record activities in real time.

[0145] 2. The device activates the camera and microphone, recording the children's activities in real time with video and audio. For example, it records them playing on playground equipment or lunchtime. The user can check the recording status on the screen.

[0146] 3. The device sends the recorded video and audio data to a cloud server (e.g., AWS) at regular intervals (e.g., every 5 minutes). The transmission is performed in the background so as not to interfere with user operations.

[0147] 4. The server analyzes the received video data using OpenCV and extracts specific keyframes (e.g., scenes of people playing on playground equipment, group photos, etc.). At the same time, it converts the audio data into text using Google Cloud Speech-to-Text and detects important phrases (e.g., "It's fun!", "Be careful").

[0148] 5. Based on the analysis results, the server uses a generative AI model (e.g., GPT-3) to automatically generate reports and messages to parents. The reports automatically include photos based on the extracted keyframes and include text containing important phrases.

[0149] 6. The user can review the report generated on their device, make corrections or add comments as needed, and in some cases even add their own photos or videos.

[0150] 7. When the user taps the "Send" button, the device resends the final confirmed document to the cloud server, which then sends the received document to the parent's email address or as an in-app message.

[0151] Specific examples

[0152] As a specific example, we will show a case where a user (a nursery teacher) takes children on a field trip.

[0153] 1. Before the field trip begins, the user starts the iPad and opens the childcare activity recording app. They tap the "Start Activity" button to begin recording.

[0154] 2. Use the device's camera to record children having fun playing on the playground equipment, and record their audio with the microphone.

[0155] 3. Recorded video and audio data is sent to the AWS cloud server every five minutes.

[0156] 4. The server analyzes the received data, identifies "scenes of playing on playground equipment" and "group photos," and extracts important phrases from the audio, such as "This is fun" and "Be careful."

[0157] 5. A generative AI model (GPT-3) is used to automatically generate a report of the field trip, forming a sentence containing the extracted scenes and key phrases.

[0158] 6. The nursery teacher checks the generated report on the iPad and makes corrections or adds photos as necessary.

[0159] 7. The teacher taps the "Send" button to send the verified report to the server, and the final report is sent to the parent via email or in-app message.

[0160] Prompt Sentence Examples

[0161] "Automatically generate a report that includes scenes of children playing on playground equipment on a field trip and comments that sound like they're having fun."

[0162] The above is a specific embodiment for carrying out the invention, which reduces the workload of childcare workers and enables quick and accurate information sharing with parents.

[0163] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0164] Step 1:

[0165] The user starts the activity

[0166] The user starts up the device, opens the dedicated app, and taps the "Start Activity" button to begin recording.

[0167] Input: Activity start operation (button tap)

[0168] Output: The device will turn on the camera and microphone and be ready for real-time recording.

[0169] Step 2:

[0170] The device records the data

[0171] The device activates a camera and microphone to record video and audio of the children's activities in real time, such as taking pictures of them playing on playground equipment or taking group photos, and recording audio.

[0172] Input: Video and audio of children's activities

[0173] Output: Real-time recorded video and audio data

[0174] Step 3:

[0175] The device sends data to the server

[0176] The device sends recorded video and audio data to a cloud server at regular intervals (e.g., every 5 minutes). The transmission occurs in the background, so it does not interfere with user operations.

[0177] Input: Recorded video and audio data

[0178] Output: Video and audio data sent to the server

[0179] Step 4:

[0180] The server analyzes the data

[0181] The server analyzes the received video data using OpenCV to extract specific keyframes (e.g., scenes of children playing on playground equipment), and converts the audio data into text using Google Cloud Speech-to-Text to detect important phrases (e.g., "This is fun!", "Be careful!").

[0182] Input: Video and audio data sent to the server

[0183] Output: Extracted keyframes, key phrases converted to text

[0184] Step 5:

[0185] The server generates the document

[0186] Based on the analyzed data, the server uses a generative AI model (e.g., GPT-3) to automatically generate reports and parental communications. The reports include photos based on the extracted keyframes and contain text containing important phrases.

[0187] Input: Extracted keyframes, key phrases converted to text

[0188] Output: Automatically generated reports and correspondence

[0189] Step 6:

[0190] The user checks and modifies the document

[0191] Users can review the report generated on their device, make corrections or add comments as needed, and can also add their own photos and videos.

[0192] Input: Auto-generated reports and correspondence

[0193] Output: Revised final report and correspondence

[0194] Step 7:

[0195] A user submits a document

[0196] When the user taps the "Send" button, the device resends the finalized document to the cloud server, which then sends the received document to the parent's email address or as an in-app message.

[0197] Input: Revised Final Report, Correspondence

[0198] Output: Final report and correspondence sent to parents

[0199] The above are the specific processing steps of the system program and their detailed operations.

[0200] (Application example 1)

[0201] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0202] Customer service work in brick-and-mortar stores often places a heavy burden on employees. Recording customer interactions and creating reports afterward takes time and effort, hindering efficient business operations. Furthermore, if the content of customer interactions is not accurately recorded, it becomes difficult to improve service quality and customer satisfaction. To solve these issues, a system is needed that automatically records customer interactions, analyzes the content of the interactions, and generates reports.

[0203] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0204] In this invention, the server includes an image capturing means for recording customer activities, an audio recording means for recording audio during the activities, a data analysis means for analyzing the recorded image and audio data to extract important information, a document generation means for automatically generating a report or communication based on the extracted information, a confirmation / modification means for allowing a user to confirm and modify the document, and a transmission means for transmitting the final confirmed document to relevant parties. This automates the recording of customer interactions in physical stores, reduces the workload of employees, and improves the quality of customer service.

[0205] "Customer activity" refers to purchases, inquiries, complaints, and other actions taken by customers in the store.

[0206] The "image capturing means" is a means for recording customer activities as video using a device such as a camera.

[0207] "Audio recording means" refers to a means for recording the voice of a customer during an activity using a device such as a microphone.

[0208] The "data analysis means" is a means for analyzing the recorded image and audio data and extracting important information.

[0209] The "document generation means" is a means for automatically generating a report or communication based on the extracted information.

[0210] "Means for checking and correcting" refers to the means by which a user can check the automatically generated document and correct its contents if necessary.

[0211] "Transmission means" refers to a means for transmitting the final confirmed document to the parties involved.

[0212] "Real-time" refers to data being processed with minimal delay from the moment it is created.

[0213] "Server" refers to a computer system for receiving recorded data, analyzing it, and generating reports.

[0214] "Stakeholders" refers to store employees and managers involved in customer service, or the customers themselves.

[0215] The present invention is a system for improving the efficiency of customer service operations in brick-and-mortar stores and reducing the workload of employees. This system records customer activity, extracts important information, and automatically generates reports or correspondence. The following describes the system's program processing and specific examples.

[0216] System configuration

[0217] This system consists of devices held by store employees (users), a cloud server, and devices for employees and related departments. Users use their devices to record customer activity, and the server analyzes the data to automatically generate reports and correspondence. The final generated documents are then checked and revised by the users before being sent to the relevant parties.

[0218] Program processing configuration

[0219] The system includes the following major processing steps:

[0220] Initiating and recording customer activity

[0221] 1. User Action:

[0222] The user turns on the device and opens the app.

[0223] Press the "Start Activity" button in the app to begin recording.

[0224] 2. Device behavior:

[0225] The device activates a camera and microphone to record customer activity in real-time video and audio.

[0226] The recorded data is sent to a server at regular intervals.

[0227] Data Analysis and Document Generation

[0228] 3. Server Operation:

[0229] The server analyzes the received video data and identifies important scenes using image recognition technology (e.g., OpenCV or TensorFlow).

[0230] Similarly, voice data is analyzed and specific keywords are extracted using voice recognition technology (e.g., Google Speech Recognition API).

[0231] Based on the analysis results, reports and correspondence to stakeholders are automatically generated. Generative AI models are used to create natural-looking documents.

[0232] Checking, correcting and sending documents

[0233] 4. User confirmation and correction:

[0234] The user checks the automatically generated document on the terminal.

[0235] Modify the document as needed.

[0236] 5. Sending documents:

[0237] When the user presses the "send" button, the terminal sends the final verified document to the server.

[0238] The server sends the document to the relevant parties via email or in-app message.

[0239] Specific examples

[0240] Example 1: Handling complaints at a store

[0241] A specific example when a user (store employee) receives a complaint from a customer is shown below.

[0242] 1. User Action:

[0243] The user turns on the device and opens the app to record the complaint.

[0244] Press "Start Activity" to begin recording.

[0245] 2. Device behavior:

[0246] The device uses a camera to record interactions between customers and employees, while simultaneously recording audio using a microphone.

[0247] The recorded data is sent to a server at regular intervals.

[0248] 3. Server Operation:

[0249] The server analyzes the video data and extracts "scenes where complaints are handled" and "important comments from customers."

[0250] The speech of employees and customers is converted into text from the audio data, and important phrases are extracted, such as keywords such as "I want to return the product" or "I'm dissatisfied."

[0251] A claim report is automatically generated based on this information.

[0252] 4. User confirmation and correction:

[0253] The user checks the generated report on the terminal and makes corrections as necessary.

[0254] You can also add photos and additional comments to your report.

[0255] 5. Sending documents:

[0256] Once the report has been revised, click the "Send" button to send it to the relevant parties.

[0257] The server will send the final report to the relevant parties' email addresses or within the app.

[0258] Prompt Sentence Examples

[0259] "Please start recording the customer interaction."

[0260] "We have completed the customer service recording. We will generate a report based on the analysis results."

[0261] This system automates the recording of customer interactions in physical stores, reducing the workload of employees and improving the quality of customer service.

[0262] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0263] Step 1:

[0264] User Action:

[0265] The user starts up the device, opens the app, and presses the "Start Activity" button within the app to begin recording.

[0266] input:

[0267] App launch information, activity start instructions

[0268] output:

[0269] Recording start state

[0270] Specific operation:

[0271] The device displays the application interface, and the user can enter recording mode by tapping the "Start Activity" button.

[0272] Step 2:

[0273] Terminal behavior:

[0274] The device activates the camera and microphone to record the customer's activities in real time using video and audio, and the recorded data is sent to a server at regular intervals.

[0275] input:

[0276] Recording start command

[0277] output:

[0278] Real-time video and audio data

[0279] Specific operation:

[0280] The device's camera records video of interactions between customers and employees, and the microphone records audio, which is then uploaded to a server at regular intervals.

[0281] Step 3:

[0282] Server behavior:

[0283] The server analyzes the received video data and identifies important scenes using image recognition technology, as well as audio data and extracts specific keywords using voice recognition technology.

[0284] input:

[0285] Video data, audio data

[0286] output:

[0287] Text data (extracted important information)

[0288] Specific operation:

[0289] The server analyzes the video data using image recognition software such as OpenCV and TensorFlow to identify specific scenes, and converts the audio data into text using the Google Speech Recognition API to extract key phrases.

[0290] Step 4:

[0291] Server behavior:

[0292] The server automatically generates reports and correspondence for stakeholders based on the analysis results, using a generative AI model to create natural-looking documents.

[0293] input:

[0294] Text data (extracted important information)

[0295] output:

[0296] Auto-generated reports or correspondence

[0297] Specific operation:

[0298] The server uses a generative AI model to automatically generate appropriate documents based on the extracted key information, using Natural Language Generation (NLG) technology.

[0299] Step 5:

[0300] User Action:

[0301] The user checks the automatically generated document on the terminal and corrects the content as necessary.

[0302] input:

[0303] Auto-generated reports or correspondence

[0304] output:

[0305] Corrected document

[0306] Specific operation:

[0307] The user checks the automatically generated report displayed on the terminal and uses the editing function to correct any errors or necessary additional information.

[0308] Step 6:

[0309] User Action:

[0310] When the user presses the "Send" button, the device sends the finalized document to the server, which then sends it to the relevant parties via email or in-app message.

[0311] input:

[0312] Corrected documents, transmission instructions

[0313] output:

[0314] Final report sent to stakeholders

[0315] Specific operation:

[0316] The user taps the send button, and the device sends the finalized document to the server, which then sends it to the relevant parties via email service or notification API.

[0317] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0318] This invention combines a system for reducing the workload of childcare workers and improving communication with parents with an emotion engine that recognizes the user's emotions. This system efficiently records children's activities and has the function of automatically generating reports and correspondence for parents, as well as generating documents that take the user's emotions into consideration. The system's program processing and specific examples are described below.

[0319] System Overview

[0320] This system consists of a device held by the childcare worker (user), a cloud server, and a parent's device. The user uses the device to record the children's activities, which is then analyzed by the server, and an emotion engine is used to recognize the user's emotions. Based on this, reports and correspondence are automatically generated, and the user can then review and edit the final documents before sending them to the parents.

[0321] Program processing

[0322] Starting and recording an activity

[0323] 1. User Action:

[0324] The user turns on the device and opens the app.

[0325] Press the "Start Activity" button in the app to begin recording.

[0326] 2. Device behavior:

[0327] The device activates the camera and microphone to record the children's activities in real time using video and audio.

[0328] The recorded data is sent to a server at regular intervals.

[0329] Data analysis, emotion recognition and document generation

[0330] 3. Server Operation:

[0331] The server analyzes the received video data and uses image recognition technology to identify important scenes.

[0332] Similarly, voice data is analyzed and specific keywords and phrases are extracted using voice recognition technology.

[0333] The server then uses an emotion engine to recognize the user's emotions from the received video and audio data. For example, it analyzes the user's tone of voice and facial expressions to extract emotions such as joy, sadness, and surprise.

[0334] The server automatically generates reports and correspondence based on the analysis and emotion recognition results, incorporating phrases and tones that correspond to the user's emotions to better tailor the content of the documents.

[0335] Checking, correcting and sending documents

[0336] 4. User confirmation and correction:

[0337] The user checks the automatically generated document on the terminal.

[0338] If necessary, the contents of the document are modified to suit the user's wishes.

[0339] 5. Sending documents:

[0340] When the user presses the "send" button, the terminal sends the final verified document to the server.

[0341] The server sends the document to the parent via email or in-app message.

[0342] Specific examples

[0343] Example 1: Field trip record

[0344] Below is a specific example of when a user (a nursery teacher) takes children on a field trip.

[0345] 1. User Action:

[0346] The user turns on the device and opens the app to record the field trip.

[0347] Press "Start Activity" to begin recording.

[0348] 2. Device behavior:

[0349] The device uses a camera to record the children playing and simultaneously records audio using a microphone.

[0350] The recorded data is sent to a server at regular intervals.

[0351] 3. Server Operation:

[0352] The server analyzes the video data and extracts scenes such as "scenes of children playing on playground equipment" and "scenes of children gathering together."

[0353] The statements made by nursery teachers and children are converted into text from the audio data, and important phrases (e.g., "It's fun," "Be careful") are extracted.

[0354] At the same time, an emotion engine is used to identify the emotions of the caregivers and children.

[0355] Based on this information, a report on the field trip is automatically generated. For example, if the children appear to be having a good time, the document will be adjusted to include more upbeat phrases and photos of smiling children.

[0356] 4. User confirmation and correction:

[0357] The user checks the generated report on the terminal and makes corrections as necessary.

[0358] You can also add photos and additional comments to your report.

[0359] 5. Sending documents:

[0360] Once the report has been revised, click the "Send" button to send it to the parent.

[0361] The server will send the final report to the parent's email address or within the app.

[0362] The above is an embodiment of the invention of a system for reducing the workload of childcare workers and sharing information with parents quickly and accurately, combined with an emotion engine.

[0363] The processing flow will be explained below.

[0364] Step 1:

[0365] The user launches the app on their device and presses the "Start Activity" button.

[0366] Operation details: At the start of an excursion, the user turns on their smartphone or tablet, opens the app, and presses the "Start Activity" button.

[0367] Step 2:

[0368] The device will turn on the camera and microphone and begin recording.

[0369] What it does: The device activates its camera and microphone and begins recording video and audio of the child's activities in real time.

[0370] Step 3:

[0371] The device stores recorded video and audio data locally.

[0372] Operation details: The device captures video and audio data in real time and stores it in local storage.

[0373] Step 4:

[0374] The terminal transmits the recorded data to the server at regular intervals.

[0375] Operation details: The device uploads collected video and audio data to the server at regular intervals.

[0376] Step 5:

[0377] The server analyzes the received video data and extracts important scenes.

[0378] Operation details: The server uses a video analysis algorithm to automatically extract important scenes, such as playing on playground equipment or group photos.

[0379] Step 6:

[0380] The server analyzes the received audio data and extracts specific keywords and phrases.

[0381] How it works: The server uses speech recognition technology to convert the voice data into text and extracts key phrases (e.g., "fun," "next is lunchtime").

[0382] Step 7:

[0383] The server uses an emotion engine to recognize the user's emotions based on video and audio data.

[0384] How it works: The server uses an emotion engine to analyze facial expressions and vocal tone in the video to identify emotions such as joy, surprise, and calm.

[0385] Step 8:

[0386] The server automatically generates reports and correspondence based on the analysis results and emotion recognition results.

[0387] Operation details: The server automatically generates reports and communication messages that reflect the children's activities and the feelings of the caregivers. When children are having fun, it uses a lot of positive expressions to make the reports more appealing.

[0388] Step 9:

[0389] The server sends the generated document to the terminal.

[0390] Operation details: Automatically generated reports and correspondence are sent to the terminal so that the user can check and edit them.

[0391] Step 10:

[0392] The user checks the document generated on the terminal and corrects it if necessary.

[0393] How it works: The user reviews the report and correspondence, enters any necessary corrections or additions, and creates the final version.

[0394] Step 11:

[0395] The user confirms the document after making corrections and presses the "Send" button.

[0396] Operation details: The user confirms the final confirmed document and presses the "Send" button to complete the transmission.

[0397] Step 12:

[0398] The terminal transmits the final confirmed document to the server.

[0399] Operation details: The terminal sends the corrected document data to the server.

[0400] Step 13:

[0401] The server sends a final report or communication to the parent.

[0402] Operational Details: The server will send a final report or communication to the parent via email address or in-app messaging.

[0403] Step 14:

[0404] The parent receives and reviews the document.

[0405] How it works: Parents receive the report or communication sent to them, review the contents, and provide feedback if necessary.

[0406] The above is a specific processing flow of the present invention. By combining it with an emotion engine, it becomes possible to report information that reflects the user's emotions, further improving communication with parents.

[0407] Example 2

[0408] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0409] In today's childcare environment, childcare workers are overwhelmed with their busy daily work, making it difficult for them to record children's activities in detail and report them to parents quickly and accurately. Furthermore, in order to communicate with parents, appropriate document generation that takes emotions into account is required, but doing this manually requires a great deal of time and effort. Therefore, a technology is needed that can accurately record children's activities and automatically generate reports that reflect their emotions while reducing the workload of childcare workers.

[0410] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes a data analysis means, an emotion recognition means, and a document generation means. This makes it possible to efficiently analyze the children's activity data and automatically generate appropriate reports and communication messages that take the user's emotions into consideration.

[0411] "Filming means" refers to devices and functions for recording children's activities as video in real time.

[0412] "Audio recording means" refers to a device or function for recording audio during an activity.

[0413] "Data analysis means" refers to devices or functions that analyze recorded image and audio data and extract important information.

[0414] The "document generation means" refers to a device or function for automatically generating reports and correspondence based on the extracted information and the user's feelings.

[0415] "Emotion recognition means" refers to a device or function for recognizing emotions from the user's tone of voice, facial expression, etc.

[0416] "Verification and correction means" refers to devices and functions that allow users to verify automatically generated documents and correct them as necessary.

[0417] "Transmission means" refers to a device or function for transmitting the final confirmed document to the guardian.

[0418] This invention combines an emotion engine with a system that reduces the workload of childcare workers and improves communication with parents. This system efficiently records children's activities, has the function of automatically generating reports and correspondence for parents, and is capable of generating documents that take the user's emotions into consideration.

[0419] System Overview

[0420] This system consists of a device held by the childcare worker (user), a cloud server, and a parent's device. The user uses the device to record the children's activities, which is then analyzed by the server, and an emotion engine is used to recognize the user's emotions. Based on this, reports and correspondence are automatically generated, and the user can then review and edit the final documents before sending them to the parents.

[0421] Hardware and software used

[0422] Device hardware:

[0423] Smartphone or tablet

[0424] Built-in camera and microphone

[0425] Server software:

[0426] Image recognition technology (e.g., OpenCV, TensorFlow)

[0427] Speech recognition technology (e.g., Google Cloud Speech-to-Text, Amazon Transcribe)

[0428] Emotion recognition engine (e.g. Microsoft Azure Emotion API, Affectiva)

[0429] Generative AI models (e.g., OpenAI GPT-3)

[0430] Program processing overview

[0431] Activity recording

[0432] User Action:

[0433] The user starts up the terminal and launches the dedicated application.

[0434] Press the "Start Activity" button in the app to begin recording mode.

[0435] Terminal behavior:

[0436] The device's camera and microphone will be activated and will record the children's activities in real time.

[0437] Video and audio data is sent to the server at regular intervals (for example, every 10 seconds).

[0438] Data analysis and emotion recognition

[0439] Server behavior:

[0440] The server analyzes the received video and audio data and extracts important information.

[0441] Image recognition technology is used to identify important scenes from video.

[0442] It uses voice recognition technology to extract specific keywords and phrases from speech.

[0443] It uses an emotion engine to recognize emotions from the user's tone of voice and facial expressions.

[0444] Document generation and transmission

[0445] Server behavior:

[0446] Based on the analysis results and emotion recognition results, the server automatically generates reports and correspondence using a generative AI model.

[0447] Tailor your document content with phrases and tones that reflect your emotions.

[0448] User Action:

[0449] The user can check the generated document on their own terminal and make corrections as necessary.

[0450] Click the "Send" button to send the revised document to the parent.

[0451] Examples:

[0452] For example, when a user takes their children on a field trip, the system operates as follows:

[0453] User Action:

[0454] The user starts the device and opens the application to record the field trip.

[0455] Press the "Start Activity" button to begin recording.

[0456] Terminal behavior:

[0457] The device's camera records the children's play and the microphone records their audio.

[0458] The recorded data is sent to the server at regular intervals.

[0459] Server behavior:

[0460] The server analyzes the video data and extracts scenes such as "scenes of children playing on playground equipment" and "scenes of children gathering together."

[0461] The system analyzes audio data and extracts important phrases from the speech of caregivers and children.

[0462] An emotion engine is used to identify the emotions of caregivers and children.

[0463] Based on the above information, a report on the field trip is automatically generated, describing the fun the students had in cheerful phrases.

[0464] User confirmation and correction:

[0465] The user reviews the generated report and makes corrections as necessary.

[0466] For example, add photos and additional comments to the report.

[0467] Sending documents:

[0468] Once the report has been revised, click the "Send" button to send it to the parent.

[0469] The server will send the final report to the parent's email address or within the app.

[0470] Prompt Sentence Examples

[0471] "Childcare workers record the children's field trips, and based on the records, automatically generate reports for parents. The generated documents also include emotional information from the childcare workers and the children. For example, fun scenes are described in cheerful language, while scenes requiring caution are described in polite language."

[0472] The above is an embodiment of the invention.

[0473] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0474] Step 1: Starting the activity

[0475] User Action:

[0476] The user starts up the terminal and launches the dedicated application.

[0477] Press the "Start Activity" button in the app to begin recording mode.

[0478] Specific behavior:

[0479] A user taps an app on their smartphone or tablet to launch it.

[0480] Pressing the "Start Activity" button will take you to the activity recording screen.

[0481] Step 2: Record and send data

[0482] Terminal behavior:

[0483] The device activates a camera and microphone to record the children's activities in real time.

[0484] Video and audio data is sent to the server at regular intervals (for example, every 10 seconds).

[0485] Input and Output:

[0486] (Input) Photographed video and recorded audio data

[0487] (Output) Split video and audio files sent to the server

[0488] Specific behavior:

[0489] Images captured by the built-in camera and audio recorded by the microphone are saved in a data buffer in real time.

[0490] The divided data is uploaded to the server sequentially (e.g., a data file is sent every 10 seconds).

[0491] Step 3: Data analysis

[0492] Server behavior:

[0493] The server analyzes the received video and audio data and extracts important information.

[0494] Image recognition technology is used to identify important scenes from video.

[0495] Using voice recognition technology, the voice data is converted into text and important keywords and phrases are extracted.

[0496] Input and Output:

[0497] (Input) Received video and audio data

[0498] (Output) Analyzed important scene information and textual keywords

[0499] Specific behavior:

[0500] Image recognition technology (e.g., OpenCV, TensorFlow) is used to analyze specific frames in the video and extract "smiling scenes" and "group photo scenes."

[0501] Voice recognition technology (e.g., Google Cloud Speech-to-Text) is used to convert the voice data into text, and important phrases such as "It's fun" and "Be careful" are extracted from the comments of the caregivers and children.

[0502] Step 4: Emotion Recognition

[0503] Server behavior:

[0504] The server uses an emotion recognition engine to recognize emotions from the user's tone of voice and facial expressions.

[0505] Input and Output:

[0506] (Input) Analyzed video and audio data

[0507] (Output) Recognized user emotion data

[0508] Specific behavior:

[0509] An emotion recognition engine (e.g., Microsoft Azure Emotion API) is used to identify emotions (e.g., joy, sadness, surprise) from facial expressions and tone of voice in the video.

[0510] The recognized emotion data is stored in a database as emotion tags.

[0511] Step 5: Document Generation

[0512] Server behavior:

[0513] Based on the analysis results and emotion recognition results, the server automatically generates reports and correspondence using a generative AI model.

[0514] Tailor your document content with phrases and tones that reflect your emotions.

[0515] Input and Output:

[0516] (Input) Important scene information, text keywords, emotion data

[0517] (Output) Automatically generated reports and correspondence

[0518] Specific behavior:

[0519] Using a generative AI model (e.g., OpenAI GPT-3), documents are automatically generated based on analysis and emotion recognition results.

[0520] The document will contain information such as "We had a great time on our field trip today," and depending on the emotion tag, it will add details such as "We saw lots of smiling children."

[0521] Step 6: Check and correct the document

[0522] User Action:

[0523] The user checks the automatically generated document on the terminal and corrects it if necessary.

[0524] Click the "Save" button to confirm the changes.

[0525] Input and Output:

[0526] (Input) Automatically generated report

[0527] (Output) Revised Final Report

[0528] Specific behavior:

[0529] Tap the generated document on the app screen to enter edit mode and manually insert additional comments or photos.

[0530] Save the amendments and prepare them as a final confirmation document.

[0531] Step 7: Sending documents

[0532] User Action:

[0533] When the user presses the "send" button, the terminal sends the final verified document to the server.

[0534] The server sends the final document to the parent.

[0535] Input and Output:

[0536] (Input) Final confirmed report

[0537] (Output) Report sent to parents

[0538] Specific behavior:

[0539] When the user taps the "Send" button, a confirmation message will be displayed.

[0540] Once you agree to the confirmation message, the document is instantly sent to the server and delivered to the parent.

[0541] (Application example 2)

[0542] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0543] Conventional virtual store customer support systems lacked the ability to recognize user emotions in real time, making it difficult to improve the user experience and provide smooth support. They also lacked the ability to immediately detect user confusion or difficulties and provide appropriate real-time support accordingly. This resulted in a poor user experience and inefficient customer support. Furthermore, there was a need for improved accuracy in automatically generated reports based on user behavior.

[0544] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0545] In this invention, the server includes a means for analyzing a user's behavior and emotions in real time, a means for recognizing emotions in real time, and a means for providing feedback based on the recognition results. This makes it possible to provide appropriate support and product suggestions in real time according to the user's emotions. In addition, the efficiency of customer support can be improved by creating highly accurate automatically generated reports based on the user's experience records.

[0546] "Image capture means" refers to video capture devices and their functions for recording children's activities.

[0547] "Audio recording means" refers to a device and its function that records audio during an activity.

[0548] "Data analysis means" refers to the function of analyzing recorded image and audio data to extract important information.

[0549] "Document generation means" refers to a function that automatically generates a report or communication based on the extracted information.

[0550] "Emotion recognition means" refers to a function that uses an emotion engine to analyze a user's emotions in real time.

[0551] "Feedback means" refers to a function that suggests appropriate actions to the user based on real-time emotion recognition results.

[0552] "Means for checking and correcting" refers to a function that allows users to check and correct automatically generated documents.

[0553] "Transmission means" refers to the function of sending the final confirmed document to the guardian.

[0554] The system for implementing this invention consists of a user's device, a cloud server, and a parent's device. The user uses the device to record their child's activities, and the data is analyzed on the server. Furthermore, an emotion engine is used to recognize the user's emotions in real time and automatically generate reports and correspondence.

[0555] Hardware and software used

[0556] Hardware:

[0557] Terminal: A device such as a smartphone, smart glasses, or head-mounted display.

[0558] Camera: The camera built into the device.

[0559] Microphone: The microphone built into the device.

[0560] software:

[0561] OpenCV: A library for image processing and face detection.

[0562] DeepFace: A library for determining emotions from images.

[0563] Hugging Face's transformers: A library for speech recognition and tone analysis.

[0564] Data processing and calculation

[0565] 1. Image and audio recording:

[0566] The user turns on the device and records their children's activities in real time.

[0567] The device's camera captures video and microphone records audio, and these data are sent to a cloud server at regular intervals.

[0568] 2. Data Analysis and Emotion Recognition:

[0569] The server uses OpenCV to analyze the video data and perform face detection.

[0570] The facial image is analyzed using the DeepFace library to determine emotions (e.g., joy, sadness, surprise, etc.).

[0571] The voice data is recognized using Hugging Face's transformers to extract the user's tone of voice and specific keywords (e.g., "fun," "troubled," etc.).

[0572] 3. Real-time feedback:

[0573] Based on the results of emotion recognition, the server will provide appropriate feedback to the user, for example, if a confused expression is detected, it will immediately display a supportive message.

[0574] 4. Document Generation:

[0575] The server automatically generates reports and correspondence based on the analyzed data and emotion recognition results.

[0576] The user can review the generated document on the terminal and make corrections as necessary.

[0577] 5. Sending documents:

[0578] The final, verified document is sent from the server to the parent via email or in-app message.

[0579] Specific examples

[0580] Use case 1: Recording a field trip

[0581] When a user takes their children on a field trip, they use the system in the following steps:

[0582] 1. The user starts up the device and presses the "Start Activity" button to record the field trip.

[0583] 2. The device uses a camera to record the children's play and a microphone to simultaneously record their voices. This data is then sent to a server at regular intervals.

[0584] 3. The server performs video and audio analysis, and extracts important scenes and keywords using facial expression and voice recognition.

[0585] 4. The server automatically generates a report on the field trip based on this information. For example, if it recognizes that the field trip is going well, it will write a report in a cheerful tone and include many photos of the children smiling.

[0586] 5. The user reviews the generated report and makes any necessary corrections.

[0587] 6. Click the "Send" button to send the finalized report to the parent.

[0588] Example prompt sentence:

[0589] Generate appropriate support messages based on the user's emotional state:

[0590] "Emotion: Confused" "User says: 'I don't know which product to buy.'"

[0591] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0592] Step 1:

[0593] The user starts the device, opens the application and presses the "Start Activity" button.

[0594] Input: Launching a terminal and operating applications.

[0595] What it does: The device activates the camera and microphone, ready to record your child's activities in real time.

[0596] Output: Camera and microphone are activated.

[0597] Step 2:

[0598] The device records video and audio in real time and transmits the data to a server in the cloud at regular intervals.

[0599] Input: Real-time data from camera and microphone.

[0600] How it works: The device's camera captures video and microphone records audio. The data is packetized at regular intervals and sent to a server.

[0601] Output: Transmitted video and audio data.

[0602] Step 3:

[0603] The server analyzes the received video and audio data.

[0604] Input: Video and audio data sent from the device.

[0605] How it works: The server uses OpenCV to detect faces from video data and DeepFace to analyze emotions, and Hugging Face's transformers to recognize audio data and extract tones and specific keywords.

[0606] Output: Analysis results (e.g. emotion labels, speech recognition results).

[0607] Step 4:

[0608] The server automatically generates a report or communication based on the analysis results.

[0609] Input: Emotion recognition results from video data, speech recognition results from audio data.

[0610] How it works: The server combines the analysis results and uses natural language generation technology to create reports and correspondence. It selects appropriate expressions based on the user's emotions and composes the text.

[0611] Output: Auto-generated report or correspondence.

[0612] Step 5:

[0613] The user reviews the automatically generated report or correspondence and corrects it if necessary.

[0614] Input: An automatically generated report or correspondence.

[0615] How it works: The user views the generated document on their device, reviews the content, and then edits the text and adds images and comments if necessary.

[0616] Output: The final verified and corrected document.

[0617] Step 6:

[0618] The final confirmed document is sent from the server to the parent.

[0619] Input: Reviewed and amended final document.

[0620] How it works: When the user presses the "Send" button, the device sends the final document to the server, which then sends it to the parent's email address or as an in-app message.

[0621] Output: Document sent to parents.

[0622] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0623] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0624] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0625] [Second embodiment]

[0626] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0627] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0628] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0629] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0630] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0631] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0632] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0633] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0634] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0635] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0636] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0637] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0638] The present invention is a system for reducing the workload of childcare workers and improving communication with parents. This system efficiently records children's activities and automatically generates reports and correspondence for parents. The program processing of the system and specific examples are described below.

[0639] System Overview

[0640] This system consists of a device held by the user (childcare worker), a cloud server, and a device held by the parent. The user uses the device to record the children's activities, and the server analyzes the data to automatically generate reports and correspondence. The final generated documents are checked and revised by the user before being sent to the parent.

[0641] Program processing

[0642] Starting and recording an activity

[0643] 1. User Action:

[0644] The user turns on the device and opens the app.

[0645] Press the "Start Activity" button in the app to begin recording.

[0646] 2. Device behavior:

[0647] The device activates the camera and microphone to record the children's activities in real time using video and audio.

[0648] The recorded data is sent to a server at regular intervals.

[0649] Data Analysis and Document Generation

[0650] 3. Server Operation:

[0651] The server analyzes the received video data and uses image recognition technology to identify important scenes (e.g., scenes of people playing on playground equipment).

[0652] Similarly, voice data is analyzed and specific keywords (e.g., "fun!", "lunch") are extracted using voice recognition technology.

[0653] Based on the analysis results, reports and messages to parents are automatically generated.

[0654] Checking, correcting and sending documents

[0655] 4. User confirmation and correction:

[0656] The user checks the automatically generated document on the terminal.

[0657] Modify the document as needed.

[0658] 5. Sending documents:

[0659] When the user presses the "send" button, the terminal sends the final verified document to the server.

[0660] The server sends the document to the parent via email or in-app message.

[0661] Specific examples

[0662] Example 1: Field trip record

[0663] Below is a specific example of when a user (a nursery teacher) takes children on a field trip.

[0664] 1. User Action:

[0665] The user turns on the device and opens the app to record the field trip.

[0666] Press "Start Activity" to begin recording.

[0667] 2. Device behavior:

[0668] The device uses a camera to record the children playing and simultaneously records audio using a microphone.

[0669] The recorded data is sent to a server at regular intervals.

[0670] 3. Server Operation:

[0671] The server analyzes the video data and extracts scenes such as "scenes of children playing on playground equipment" and "scenes of children gathering together."

[0672] The statements made by nursery teachers and children are converted into text from the audio data, and important phrases (e.g., "It's fun," "Be careful") are extracted.

[0673] A field trip report is automatically generated based on this information.

[0674] 4. User confirmation and correction:

[0675] The user checks the generated report on the terminal and makes corrections as necessary.

[0676] You can also add photos and additional comments to your report.

[0677] 5. Sending documents:

[0678] Once the report has been revised, click the "Send" button to send it to the parent.

[0679] The server will send the final report to the parent's email address or within the app.

[0680] The above is a form for implementing the invention of a system for reducing the workload of childcare workers and sharing information with parents quickly and accurately.

[0681] The processing flow will be explained below.

[0682] Step 1:

[0683] The user launches the app on their device and presses the "Start Activity" button.

[0684] How it works: To record their child's activity, the user launches the app and selects the "Start Activity" button.

[0685] Step 2:

[0686] The device will turn on the camera and microphone and begin recording.

[0687] Action Details: The device activates the camera and microphone and begins real-time video and audio recording.

[0688] Step 3:

[0689] The device stores recorded video and audio data locally.

[0690] Operation details: The device captures video and audio data in real time and stores it in local storage.

[0691] Step 4:

[0692] The terminal transmits the recorded data to the server at regular intervals.

[0693] Operation details: The device uploads collected video and audio data to the server at regular intervals.

[0694] Step 5:

[0695] The server analyzes the received video data and extracts important scenes.

[0696] How it works: The server uses image recognition technology to analyze the content of the video and identify specific activities or behaviors.

[0697] Step 6:

[0698] The server analyzes the received audio data and extracts specific keywords and phrases.

[0699] How it works: The server uses speech recognition technology to convert speech into text and extract key information.

[0700] Step 7:

[0701] The server automatically generates reports and correspondence based on the analysis results.

[0702] Operation details: The server automatically generates reports and correspondence based on the analyzed data and makes them available to the user for review.

[0703] Step 8:

[0704] The server sends the generated document to the terminal.

[0705] Operation details: The server sends the generated reports and messages to the terminal so that the user can check and edit them.

[0706] Step 9:

[0707] The user checks and modifies the document generated on the terminal.

[0708] Operation details: The user checks the automatically generated document on the device and corrects any necessary parts.

[0709] Step 10:

[0710] The user confirms the document after making corrections and presses the "Send" button.

[0711] Operation details: The user confirms the final confirmed document and presses the "Send" button to send it.

[0712] Step 11:

[0713] The terminal transmits the final confirmed document to the server.

[0714] Operation details: The terminal sends the corrected document data to the server.

[0715] Step 12:

[0716] The server sends a final report or communication to the parent.

[0717] Operation details: The server sends the document to the parent's email address or via the in-app messaging function.

[0718] Step 13:

[0719] The parent receives and reviews the document.

[0720] How it works: Parents receive and review the documents sent to them, and can reply in-app or via email if they have any feedback or questions.

[0721] The above are the specific steps in the processing flow of the program.

[0722] Example 1

[0723] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0724] There is a need to reduce the workload of childcare workers and improve communication with parents. However, conventional manual recording and reporting tasks require a great deal of time and effort, placing a heavy burden on childcare workers. Furthermore, manual report creation is prone to subjectivity and can lack accuracy. The objective of this invention is to solve these problems and provide a system that enables efficient and accurate activity recording and reporting.

[0725] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0726] In this invention, the server includes a data analysis means, a document generation means, and a transmission means. This makes it possible to analyze received data and extract specific scenes. Furthermore, it is possible to automatically generate reports or communication messages based on the analysis results using a generative AI model, and the final confirmed documents can be quickly sent to the guardian.

[0727] The "image capturing means" is a video capturing device for recording children's activities in real time.

[0728] The "audio recording means" is a recording device for recording the voices of children during their activities.

[0729] The "transmission means" is a communication device or system for transmitting the recorded image and audio data to a cloud server via the Internet.

[0730] A "cloud server" is a remote server accessible via the Internet that performs processes such as data storage, analysis, and document generation.

[0731] "Data analysis means" refers to software or a system for analyzing received image and audio data and extracting specific scenes and important information.

[0732] "Document generation means" refers to software or systems such as generative AI models that automatically generate reports or correspondence based on analysis results.

[0733] "Verification and correction means" refers to the means by which a user can verify the automatically generated document and correct it if necessary.

[0734] "Transmission means" refers to a communication device or system for transmitting the finalized document to the parent.

[0735] The present invention is a system for reducing the workload of childcare workers and improving communication with parents. This system efficiently records children's activities and automatically generates reports and correspondence for parents. Specific embodiments of the system are described below.

[0736] This system consists of a device held by the user (childcare worker), a cloud server, and a device held by the parent. The user uses the device to record the children's activities, and the cloud server analyzes the data to automatically generate reports and correspondence. The final generated documents are checked and revised by the user before being sent to the parent.

[0737] Hardware and software used

[0738] Device: Smartphone or tablet (e.g. iPad) used by the childcare worker

[0739] Cloud server: A cloud server running on AWS (Amazon Web Services)

[0740] Software components:

[0741] Image recognition technology: OpenCV

[0742] Speech recognition technology: Google Cloud Speech-to-Text

[0743] Natural Language Generation: GPT-3 (generative AI model)

[0744] Communication method: Firebase

[0745] A detailed description of the system's program processing

[0746] 1. To record the activities of the children under the user's care, the user starts up the device and opens a dedicated app (e.g., a childcare activity recording app). By tapping the "Start Activity" button, the device prepares to record activities in real time.

[0747] 2. The device activates the camera and microphone, recording the children's activities in real time with video and audio. For example, it records them playing on playground equipment or lunchtime. The user can check the recording status on the screen.

[0748] 3. The device sends the recorded video and audio data to a cloud server (e.g., AWS) at regular intervals (e.g., every 5 minutes). The transmission is performed in the background so as not to interfere with user operations.

[0749] 4. The server analyzes the received video data using OpenCV and extracts specific keyframes (e.g., scenes of people playing on playground equipment, group photos, etc.). At the same time, it converts the audio data into text using Google Cloud Speech-to-Text and detects important phrases (e.g., "It's fun!", "Be careful").

[0750] 5. Based on the analysis results, the server uses a generative AI model (e.g., GPT-3) to automatically generate reports and messages to parents. The reports automatically include photos based on the extracted keyframes and include text containing important phrases.

[0751] 6. The user can review the report generated on their device, make corrections or add comments as needed, and in some cases even add their own photos or videos.

[0752] 7. When the user taps the "Send" button, the device resends the final confirmed document to the cloud server, which then sends the received document to the parent's email address or as an in-app message.

[0753] Specific examples

[0754] As a specific example, we will show a case where a user (a nursery teacher) takes children on a field trip.

[0755] 1. Before the field trip begins, the user starts the iPad and opens the childcare activity recording app. They tap the "Start Activity" button to begin recording.

[0756] 2. Use the device's camera to record children having fun playing on the playground equipment, and record their audio with the microphone.

[0757] 3. Recorded video and audio data is sent to the AWS cloud server every five minutes.

[0758] 4. The server analyzes the received data, identifies "scenes of playing on playground equipment" and "group photos," and extracts important phrases from the audio, such as "This is fun" and "Be careful."

[0759] 5. A generative AI model (GPT-3) is used to automatically generate a report of the field trip, forming a sentence containing the extracted scenes and key phrases.

[0760] 6. The nursery teacher checks the generated report on the iPad and makes corrections or adds photos as necessary.

[0761] 7. The teacher taps the "Send" button to send the verified report to the server, and the final report is sent to the parent via email or in-app message.

[0762] Prompt Sentence Examples

[0763] "Automatically generate a report that includes scenes of children playing on playground equipment on a field trip and comments that sound like they're having fun."

[0764] The above is a specific embodiment for carrying out the invention, which reduces the workload of childcare workers and enables quick and accurate information sharing with parents.

[0765] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0766] Step 1:

[0767] The user starts the activity

[0768] The user starts up the device, opens the dedicated app, and taps the "Start Activity" button to begin recording.

[0769] Input: Activity start operation (button tap)

[0770] Output: The device will turn on the camera and microphone and be ready for real-time recording.

[0771] Step 2:

[0772] The device records the data

[0773] The device activates a camera and microphone to record video and audio of the children's activities in real time, such as taking pictures of them playing on playground equipment or taking group photos, and recording audio.

[0774] Input: Video and audio of children's activities

[0775] Output: Real-time recorded video and audio data

[0776] Step 3:

[0777] The device sends data to the server

[0778] The device sends recorded video and audio data to a cloud server at regular intervals (e.g., every 5 minutes). The transmission occurs in the background, so it does not interfere with user operations.

[0779] Input: Recorded video and audio data

[0780] Output: Video and audio data sent to the server

[0781] Step 4:

[0782] The server analyzes the data

[0783] The server analyzes the received video data using OpenCV to extract specific keyframes (e.g., scenes of children playing on playground equipment), and converts the audio data into text using Google Cloud Speech-to-Text to detect important phrases (e.g., "This is fun!", "Be careful!").

[0784] Input: Video and audio data sent to the server

[0785] Output: Extracted keyframes, key phrases converted to text

[0786] Step 5:

[0787] The server generates the document

[0788] Based on the analyzed data, the server uses a generative AI model (e.g., GPT-3) to automatically generate reports and parental communications. The reports include photos based on the extracted keyframes and contain text containing important phrases.

[0789] Input: Extracted keyframes, key phrases converted to text

[0790] Output: Automatically generated reports and correspondence

[0791] Step 6:

[0792] The user checks and modifies the document

[0793] Users can review the report generated on their device, make corrections or add comments as needed, and can also add their own photos and videos.

[0794] Input: Auto-generated reports and correspondence

[0795] Output: Revised final report and correspondence

[0796] Step 7:

[0797] A user submits a document

[0798] When the user taps the "Send" button, the device resends the finalized document to the cloud server, which then sends the received document to the parent's email address or as an in-app message.

[0799] Input: Revised Final Report, Correspondence

[0800] Output: Final report and correspondence sent to parents

[0801] The above are the specific processing steps of the system program and their detailed operations.

[0802] (Application example 1)

[0803] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0804] Customer service work in brick-and-mortar stores often places a heavy burden on employees. Recording customer interactions and creating reports afterward takes time and effort, hindering efficient business operations. Furthermore, if the content of customer interactions is not accurately recorded, it becomes difficult to improve service quality and customer satisfaction. To solve these issues, a system is needed that automatically records customer interactions, analyzes the content of the interactions, and generates reports.

[0805] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0806] In this invention, the server includes an image capturing means for recording customer activities, an audio recording means for recording audio during the activities, a data analysis means for analyzing the recorded image and audio data to extract important information, a document generation means for automatically generating a report or communication based on the extracted information, a confirmation / modification means for allowing a user to confirm and modify the document, and a transmission means for transmitting the final confirmed document to relevant parties. This automates the recording of customer interactions in physical stores, reduces the workload of employees, and improves the quality of customer service.

[0807] "Customer activity" refers to purchases, inquiries, complaints, and other actions taken by customers in the store.

[0808] The "image capturing means" is a means for recording customer activities as video using a device such as a camera.

[0809] "Audio recording means" refers to a means for recording the voice of a customer during an activity using a device such as a microphone.

[0810] The "data analysis means" is a means for analyzing the recorded image and audio data and extracting important information.

[0811] The "document generation means" is a means for automatically generating a report or communication based on the extracted information.

[0812] "Means for checking and correcting" refers to the means by which a user can check the automatically generated document and correct its contents if necessary.

[0813] "Transmission means" refers to a means for transmitting the final confirmed document to the parties involved.

[0814] "Real-time" refers to data being processed with minimal delay from the moment it is created.

[0815] "Server" refers to a computer system for receiving recorded data, analyzing it, and generating reports.

[0816] "Stakeholders" refers to store employees and managers involved in customer service, or the customers themselves.

[0817] The present invention is a system for improving the efficiency of customer service operations in brick-and-mortar stores and reducing the workload of employees. This system records customer activity, extracts important information, and automatically generates reports or correspondence. The following describes the system's program processing and specific examples.

[0818] System configuration

[0819] This system consists of devices held by store employees (users), a cloud server, and devices for employees and related departments. Users use their devices to record customer activity, and the server analyzes the data to automatically generate reports and correspondence. The final generated documents are then checked and revised by the users before being sent to the relevant parties.

[0820] Program processing configuration

[0821] The system includes the following major processing steps:

[0822] Initiating and recording customer activity

[0823] 1. User Action:

[0824] The user turns on the device and opens the app.

[0825] Press the "Start Activity" button in the app to begin recording.

[0826] 2. Device behavior:

[0827] The device activates a camera and microphone to record customer activity in real-time video and audio.

[0828] The recorded data is sent to a server at regular intervals.

[0829] Data Analysis and Document Generation

[0830] 3. Server Operation:

[0831] The server analyzes the received video data and identifies important scenes using image recognition technology (e.g., OpenCV or TensorFlow).

[0832] Similarly, voice data is analyzed and specific keywords are extracted using voice recognition technology (e.g., Google Speech Recognition API).

[0833] Based on the analysis results, reports and correspondence to stakeholders are automatically generated. Generative AI models are used to create natural-looking documents.

[0834] Checking, correcting and sending documents

[0835] 4. User confirmation and correction:

[0836] The user checks the automatically generated document on the terminal.

[0837] Modify the document as needed.

[0838] 5. Sending documents:

[0839] When the user presses the "send" button, the terminal sends the final verified document to the server.

[0840] The server sends the document to the relevant parties via email or in-app message.

[0841] Specific examples

[0842] Example 1: Handling complaints at a store

[0843] A specific example when a user (store employee) receives a complaint from a customer is shown below.

[0844] 1. User Action:

[0845] The user turns on the device and opens the app to record the complaint.

[0846] Press "Start Activity" to begin recording.

[0847] 2. Device behavior:

[0848] The device uses a camera to record interactions between customers and employees, while simultaneously recording audio using a microphone.

[0849] The recorded data is sent to a server at regular intervals.

[0850] 3. Server Operation:

[0851] The server analyzes the video data and extracts "scenes where complaints are handled" and "important comments from customers."

[0852] The speech of employees and customers is converted into text from the audio data, and important phrases are extracted, such as keywords such as "I want to return the product" or "I'm dissatisfied."

[0853] A claim report is automatically generated based on this information.

[0854] 4. User confirmation and correction:

[0855] The user checks the generated report on the terminal and makes corrections as necessary.

[0856] You can also add photos and additional comments to your report.

[0857] 5. Sending documents:

[0858] Once the report has been revised, click the "Send" button to send it to the relevant parties.

[0859] The server will send the final report to the relevant parties' email addresses or within the app.

[0860] Prompt Sentence Examples

[0861] "Please start recording the customer interaction."

[0862] "We have completed the customer service recording. We will generate a report based on the analysis results."

[0863] This system automates the recording of customer interactions in physical stores, reducing the workload of employees and improving the quality of customer service.

[0864] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0865] Step 1:

[0866] User Action:

[0867] The user starts up the device, opens the app, and presses the "Start Activity" button within the app to begin recording.

[0868] input:

[0869] App launch information, activity start instructions

[0870] output:

[0871] Recording start state

[0872] Specific operation:

[0873] The device displays the application interface, and the user can enter recording mode by tapping the "Start Activity" button.

[0874] Step 2:

[0875] Terminal behavior:

[0876] The device activates the camera and microphone to record the customer's activities in real time using video and audio, and the recorded data is sent to a server at regular intervals.

[0877] input:

[0878] Recording start command

[0879] output:

[0880] Real-time video and audio data

[0881] Specific operation:

[0882] The device's camera records video of interactions between customers and employees, and the microphone records audio, which is then uploaded to a server at regular intervals.

[0883] Step 3:

[0884] Server behavior:

[0885] The server analyzes the received video data and identifies important scenes using image recognition technology, as well as audio data and extracts specific keywords using voice recognition technology.

[0886] input:

[0887] Video data, audio data

[0888] output:

[0889] Text data (extracted important information)

[0890] Specific operation:

[0891] The server analyzes the video data using image recognition software such as OpenCV and TensorFlow to identify specific scenes, and converts the audio data into text using the Google Speech Recognition API to extract key phrases.

[0892] Step 4:

[0893] Server behavior:

[0894] The server automatically generates reports and correspondence for stakeholders based on the analysis results, using a generative AI model to create natural-looking documents.

[0895] input:

[0896] Text data (extracted important information)

[0897] output:

[0898] Auto-generated reports or correspondence

[0899] Specific operation:

[0900] The server uses a generative AI model to automatically generate appropriate documents based on the extracted key information, using Natural Language Generation (NLG) technology.

[0901] Step 5:

[0902] User Action:

[0903] The user checks the automatically generated document on the terminal and corrects the content as necessary.

[0904] input:

[0905] Auto-generated reports or correspondence

[0906] output:

[0907] Corrected document

[0908] Specific operation:

[0909] The user checks the automatically generated report displayed on the terminal and uses the editing function to correct any errors or necessary additional information.

[0910] Step 6:

[0911] User Action:

[0912] When the user presses the "Send" button, the device sends the finalized document to the server, which then sends it to the relevant parties via email or in-app message.

[0913] input:

[0914] Corrected documents, transmission instructions

[0915] output:

[0916] Final report sent to stakeholders

[0917] Specific operation:

[0918] The user taps the send button, and the device sends the finalized document to the server, which then sends it to the relevant parties via email service or notification API.

[0919] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0920] This invention combines a system for reducing the workload of childcare workers and improving communication with parents with an emotion engine that recognizes the user's emotions. This system efficiently records children's activities and has the function of automatically generating reports and correspondence for parents, as well as generating documents that take the user's emotions into consideration. The system's program processing and specific examples are described below.

[0921] System Overview

[0922] This system consists of a device held by the childcare worker (user), a cloud server, and a parent's device. The user uses the device to record the children's activities, which is then analyzed by the server, and an emotion engine is used to recognize the user's emotions. Based on this, reports and correspondence are automatically generated, and the user can then review and edit the final documents before sending them to the parents.

[0923] Program processing

[0924] Starting and recording an activity

[0925] 1. User Action:

[0926] The user turns on the device and opens the app.

[0927] Press the "Start Activity" button in the app to begin recording.

[0928] 2. Device behavior:

[0929] The device activates the camera and microphone to record the children's activities in real time using video and audio.

[0930] The recorded data is sent to a server at regular intervals.

[0931] Data analysis, emotion recognition and document generation

[0932] 3. Server Operation:

[0933] The server analyzes the received video data and uses image recognition technology to identify important scenes.

[0934] Similarly, voice data is analyzed and specific keywords and phrases are extracted using voice recognition technology.

[0935] The server then uses an emotion engine to recognize the user's emotions from the received video and audio data. For example, it analyzes the user's tone of voice and facial expressions to extract emotions such as joy, sadness, and surprise.

[0936] The server automatically generates reports and correspondence based on the analysis and emotion recognition results, incorporating phrases and tones that correspond to the user's emotions to better tailor the content of the documents.

[0937] Checking, correcting and sending documents

[0938] 4. User confirmation and correction:

[0939] The user checks the automatically generated document on the terminal.

[0940] If necessary, the contents of the document are modified to suit the user's wishes.

[0941] 5. Sending documents:

[0942] When the user presses the "send" button, the terminal sends the final verified document to the server.

[0943] The server sends the document to the parent via email or in-app message.

[0944] Specific examples

[0945] Example 1: Field trip record

[0946] Below is a specific example of when a user (a nursery teacher) takes children on a field trip.

[0947] 1. User Action:

[0948] The user turns on the device and opens the app to record the field trip.

[0949] Press "Start Activity" to begin recording.

[0950] 2. Device behavior:

[0951] The device uses a camera to record the children playing and simultaneously records audio using a microphone.

[0952] The recorded data is sent to a server at regular intervals.

[0953] 3. Server Operation:

[0954] The server analyzes the video data and extracts scenes such as "scenes of children playing on playground equipment" and "scenes of children gathering together."

[0955] The statements made by nursery teachers and children are converted into text from the audio data, and important phrases (e.g., "It's fun," "Be careful") are extracted.

[0956] At the same time, an emotion engine is used to identify the emotions of the caregivers and children.

[0957] Based on this information, a report on the field trip is automatically generated. For example, if the children appear to be having a good time, the document will be adjusted to include more upbeat phrases and photos of smiling children.

[0958] 4. User confirmation and correction:

[0959] The user checks the generated report on the terminal and makes corrections as necessary.

[0960] You can also add photos and additional comments to your report.

[0961] 5. Sending documents:

[0962] Once the report has been revised, click the "Send" button to send it to the parent.

[0963] The server will send the final report to the parent's email address or within the app.

[0964] The above is an embodiment of the invention of a system for reducing the workload of childcare workers and sharing information with parents quickly and accurately, combined with an emotion engine.

[0965] The processing flow will be explained below.

[0966] Step 1:

[0967] The user launches the app on their device and presses the "Start Activity" button.

[0968] Operation details: At the start of an excursion, the user turns on their smartphone or tablet, opens the app, and presses the "Start Activity" button.

[0969] Step 2:

[0970] The device will turn on the camera and microphone and begin recording.

[0971] What it does: The device activates its camera and microphone and begins recording video and audio of the child's activities in real time.

[0972] Step 3:

[0973] The device stores recorded video and audio data locally.

[0974] Operation details: The device captures video and audio data in real time and stores it in local storage.

[0975] Step 4:

[0976] The terminal transmits the recorded data to the server at regular intervals.

[0977] Operation details: The device uploads collected video and audio data to the server at regular intervals.

[0978] Step 5:

[0979] The server analyzes the received video data and extracts important scenes.

[0980] Operation details: The server uses a video analysis algorithm to automatically extract important scenes, such as playing on playground equipment or group photos.

[0981] Step 6:

[0982] The server analyzes the received audio data and extracts specific keywords and phrases.

[0983] How it works: The server uses speech recognition technology to convert the voice data into text and extracts key phrases (e.g., "fun," "next is lunchtime").

[0984] Step 7:

[0985] The server uses an emotion engine to recognize the user's emotions based on video and audio data.

[0986] How it works: The server uses an emotion engine to analyze facial expressions and vocal tone in the video to identify emotions such as joy, surprise, and calm.

[0987] Step 8:

[0988] The server automatically generates reports and correspondence based on the analysis results and emotion recognition results.

[0989] Operation details: The server automatically generates reports and communication messages that reflect the children's activities and the feelings of the caregivers. When children are having fun, it uses a lot of positive expressions to make the reports more appealing.

[0990] Step 9:

[0991] The server sends the generated document to the terminal.

[0992] Operation details: Automatically generated reports and correspondence are sent to the terminal so that the user can check and edit them.

[0993] Step 10:

[0994] The user checks the document generated on the terminal and corrects it if necessary.

[0995] How it works: The user reviews the report and correspondence, enters any necessary corrections or additions, and creates the final version.

[0996] Step 11:

[0997] The user confirms the document after making corrections and presses the "Send" button.

[0998] Operation details: The user confirms the final confirmed document and presses the "Send" button to complete the transmission.

[0999] Step 12:

[1000] The terminal transmits the final confirmed document to the server.

[1001] Operation details: The terminal sends the corrected document data to the server.

[1002] Step 13:

[1003] The server sends a final report or communication to the parent.

[1004] Operational Details: The server will send a final report or communication to the parent via email address or in-app messaging.

[1005] Step 14:

[1006] The parent receives and reviews the document.

[1007] How it works: Parents receive the report or communication sent to them, review the contents, and provide feedback if necessary.

[1008] The above is a specific processing flow of the present invention. By combining it with an emotion engine, it becomes possible to report information that reflects the user's emotions, further improving communication with parents.

[1009] Example 2

[1010] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1011] In today's childcare environment, childcare workers are overwhelmed with their busy daily work, making it difficult for them to record children's activities in detail and report them to parents quickly and accurately. Furthermore, in order to communicate with parents, appropriate document generation that takes emotions into account is required, but doing this manually requires a great deal of time and effort. Therefore, a technology is needed that can accurately record children's activities and automatically generate reports that reflect their emotions while reducing the workload of childcare workers.

[1012] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes a data analysis means, an emotion recognition means, and a document generation means. This makes it possible to efficiently analyze the children's activity data and automatically generate appropriate reports and communication messages that take the user's emotions into consideration.

[1013] "Filming means" refers to devices and functions for recording children's activities as video in real time.

[1014] "Audio recording means" refers to a device or function for recording audio during an activity.

[1015] "Data analysis means" refers to devices or functions that analyze recorded image and audio data and extract important information.

[1016] The "document generation means" refers to a device or function for automatically generating reports and correspondence based on the extracted information and the user's feelings.

[1017] "Emotion recognition means" refers to a device or function for recognizing emotions from the user's tone of voice, facial expression, etc.

[1018] "Verification and correction means" refers to devices and functions that allow users to verify automatically generated documents and correct them as necessary.

[1019] "Transmission means" refers to a device or function for transmitting the final confirmed document to the guardian.

[1020] This invention combines an emotion engine with a system that reduces the workload of childcare workers and improves communication with parents. This system efficiently records children's activities, has the function of automatically generating reports and correspondence for parents, and is capable of generating documents that take the user's emotions into consideration.

[1021] System Overview

[1022] This system consists of a device held by the childcare worker (user), a cloud server, and a parent's device. The user uses the device to record the children's activities, which is then analyzed by the server, and an emotion engine is used to recognize the user's emotions. Based on this, reports and correspondence are automatically generated, and the user can then review and edit the final documents before sending them to the parents.

[1023] Hardware and software used

[1024] Device hardware:

[1025] Smartphone or tablet

[1026] Built-in camera and microphone

[1027] Server software:

[1028] Image recognition technology (e.g., OpenCV, TensorFlow)

[1029] Speech recognition technology (e.g., Google Cloud Speech-to-Text, Amazon Transcribe)

[1030] Emotion recognition engine (e.g. Microsoft Azure Emotion API, Affectiva)

[1031] Generative AI models (e.g., OpenAI GPT-3)

[1032] Program processing overview

[1033] Activity recording

[1034] User Action:

[1035] The user starts up the terminal and launches the dedicated application.

[1036] Press the "Start Activity" button in the app to begin recording mode.

[1037] Terminal behavior:

[1038] The device's camera and microphone will be activated and will record the children's activities in real time.

[1039] Video and audio data is sent to the server at regular intervals (for example, every 10 seconds).

[1040] Data analysis and emotion recognition

[1041] Server behavior:

[1042] The server analyzes the received video and audio data and extracts important information.

[1043] Image recognition technology is used to identify important scenes from video.

[1044] It uses voice recognition technology to extract specific keywords and phrases from speech.

[1045] It uses an emotion engine to recognize emotions from the user's tone of voice and facial expressions.

[1046] Document generation and transmission

[1047] Server behavior:

[1048] Based on the analysis results and emotion recognition results, the server automatically generates reports and correspondence using a generative AI model.

[1049] Tailor your document content with phrases and tones that reflect your emotions.

[1050] User Action:

[1051] The user can check the generated document on their own terminal and make corrections as necessary.

[1052] Click the "Send" button to send the revised document to the parent.

[1053] Examples:

[1054] For example, when a user takes their children on a field trip, the system operates as follows:

[1055] User Action:

[1056] The user starts the device and opens the application to record the field trip.

[1057] Press the "Start Activity" button to begin recording.

[1058] Terminal behavior:

[1059] The device's camera records the children's play and the microphone records their audio.

[1060] The recorded data is sent to the server at regular intervals.

[1061] Server behavior:

[1062] The server analyzes the video data and extracts scenes such as "scenes of children playing on playground equipment" and "scenes of children gathering together."

[1063] The system analyzes audio data and extracts important phrases from the speech of caregivers and children.

[1064] An emotion engine is used to identify the emotions of caregivers and children.

[1065] Based on the above information, a report on the field trip is automatically generated, describing the fun the students had in cheerful phrases.

[1066] User confirmation and correction:

[1067] The user reviews the generated report and makes corrections as necessary.

[1068] For example, add photos and additional comments to the report.

[1069] Sending documents:

[1070] Once the report has been revised, click the "Send" button to send it to the parent.

[1071] The server will send the final report to the parent's email address or within the app.

[1072] Prompt Sentence Examples

[1073] "Childcare workers record the children's field trips, and based on the records, automatically generate reports for parents. The generated documents also include emotional information from the childcare workers and the children. For example, fun scenes are described in cheerful language, while scenes requiring caution are described in polite language."

[1074] The above is an embodiment of the invention.

[1075] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1076] Step 1: Starting the activity

[1077] User Action:

[1078] The user starts up the terminal and launches the dedicated application.

[1079] Press the "Start Activity" button in the app to begin recording mode.

[1080] Specific behavior:

[1081] A user taps an app on their smartphone or tablet to launch it.

[1082] Pressing the "Start Activity" button will take you to the activity recording screen.

[1083] Step 2: Record and send data

[1084] Terminal behavior:

[1085] The device activates a camera and microphone to record the children's activities in real time.

[1086] Video and audio data is sent to the server at regular intervals (for example, every 10 seconds).

[1087] Input and Output:

[1088] (Input) Photographed video and recorded audio data

[1089] (Output) Split video and audio files sent to the server

[1090] Specific behavior:

[1091] Images captured by the built-in camera and audio recorded by the microphone are saved in a data buffer in real time.

[1092] The divided data is uploaded to the server sequentially (e.g., a data file is sent every 10 seconds).

[1093] Step 3: Data analysis

[1094] Server behavior:

[1095] The server analyzes the received video and audio data and extracts important information.

[1096] Image recognition technology is used to identify important scenes from video.

[1097] Using voice recognition technology, the voice data is converted into text and important keywords and phrases are extracted.

[1098] Input and Output:

[1099] (Input) Received video and audio data

[1100] (Output) Analyzed important scene information and textual keywords

[1101] Specific behavior:

[1102] Image recognition technology (e.g., OpenCV, TensorFlow) is used to analyze specific frames in the video and extract "smiling scenes" and "group photo scenes."

[1103] Voice recognition technology (e.g., Google Cloud Speech-to-Text) is used to convert the voice data into text, and important phrases such as "It's fun" and "Be careful" are extracted from the comments of the caregivers and children.

[1104] Step 4: Emotion Recognition

[1105] Server behavior:

[1106] The server uses an emotion recognition engine to recognize emotions from the user's tone of voice and facial expressions.

[1107] Input and Output:

[1108] (Input) Analyzed video and audio data

[1109] (Output) Recognized user emotion data

[1110] Specific behavior:

[1111] An emotion recognition engine (e.g., Microsoft Azure Emotion API) is used to identify emotions (e.g., joy, sadness, surprise) from facial expressions and tone of voice in the video.

[1112] The recognized emotion data is stored in a database as emotion tags.

[1113] Step 5: Document Generation

[1114] Server behavior:

[1115] Based on the analysis results and emotion recognition results, the server automatically generates reports and correspondence using a generative AI model.

[1116] Tailor your document content with phrases and tones that reflect your emotions.

[1117] Input and Output:

[1118] (Input) Important scene information, text keywords, emotion data

[1119] (Output) Automatically generated reports and correspondence

[1120] Specific behavior:

[1121] Using a generative AI model (e.g., OpenAI GPT-3), documents are automatically generated based on analysis and emotion recognition results.

[1122] The document will contain information such as "We had a great time on our field trip today," and depending on the emotion tag, it will add details such as "We saw lots of smiling children."

[1123] Step 6: Check and correct the document

[1124] User Action:

[1125] The user checks the automatically generated document on the terminal and corrects it if necessary.

[1126] Click the "Save" button to confirm the changes.

[1127] Input and Output:

[1128] (Input) Automatically generated report

[1129] (Output) Revised Final Report

[1130] Specific behavior:

[1131] Tap the generated document on the app screen to enter edit mode and manually insert additional comments or photos.

[1132] Save the amendments and prepare them as a final confirmation document.

[1133] Step 7: Sending documents

[1134] User Action:

[1135] When the user presses the "send" button, the terminal sends the final verified document to the server.

[1136] The server sends the final document to the parent.

[1137] Input and Output:

[1138] (Input) Final confirmed report

[1139] (Output) Report sent to parents

[1140] Specific behavior:

[1141] When the user taps the "Send" button, a confirmation message will be displayed.

[1142] Once you agree to the confirmation message, the document is instantly sent to the server and delivered to the parent.

[1143] (Application example 2)

[1144] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1145] Conventional virtual store customer support systems lacked the ability to recognize user emotions in real time, making it difficult to improve the user experience and provide smooth support. They also lacked the ability to immediately detect user confusion or difficulties and provide appropriate real-time support accordingly. This resulted in a poor user experience and inefficient customer support. Furthermore, there was a need for improved accuracy in automatically generated reports based on user behavior.

[1146] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1147] In this invention, the server includes a means for analyzing a user's behavior and emotions in real time, a means for recognizing emotions in real time, and a means for providing feedback based on the recognition results. This makes it possible to provide appropriate support and product suggestions in real time according to the user's emotions. In addition, the efficiency of customer support can be improved by creating highly accurate automatically generated reports based on the user's experience records.

[1148] "Image capture means" refers to video capture devices and their functions for recording children's activities.

[1149] "Audio recording means" refers to a device and its function that records audio during an activity.

[1150] "Data analysis means" refers to the function of analyzing recorded image and audio data to extract important information.

[1151] "Document generation means" refers to a function that automatically generates a report or communication based on the extracted information.

[1152] "Emotion recognition means" refers to a function that uses an emotion engine to analyze a user's emotions in real time.

[1153] "Feedback means" refers to a function that suggests appropriate actions to the user based on real-time emotion recognition results.

[1154] "Means for checking and correcting" refers to a function that allows users to check and correct automatically generated documents.

[1155] "Transmission means" refers to the function of sending the final confirmed document to the guardian.

[1156] The system for implementing this invention consists of a user's device, a cloud server, and a parent's device. The user uses the device to record their child's activities, and the data is analyzed on the server. Furthermore, an emotion engine is used to recognize the user's emotions in real time and automatically generate reports and correspondence.

[1157] Hardware and software used

[1158] Hardware:

[1159] Terminal: A device such as a smartphone, smart glasses, or head-mounted display.

[1160] Camera: The camera built into the device.

[1161] Microphone: The microphone built into the device.

[1162] software:

[1163] OpenCV: A library for image processing and face detection.

[1164] DeepFace: A library for determining emotions from images.

[1165] Hugging Face's transformers: A library for speech recognition and tone analysis.

[1166] Data processing and calculation

[1167] 1. Image and audio recording:

[1168] The user turns on the device and records their children's activities in real time.

[1169] The device's camera captures video and microphone records audio, and these data are sent to a cloud server at regular intervals.

[1170] 2. Data Analysis and Emotion Recognition:

[1171] The server uses OpenCV to analyze the video data and perform face detection.

[1172] The facial image is analyzed using the DeepFace library to determine emotions (e.g., joy, sadness, surprise, etc.).

[1173] The voice data is recognized using Hugging Face's transformers to extract the user's tone of voice and specific keywords (e.g., "fun," "troubled," etc.).

[1174] 3. Real-time feedback:

[1175] Based on the results of emotion recognition, the server will provide appropriate feedback to the user, for example, if a confused expression is detected, it will immediately display a supportive message.

[1176] 4. Document Generation:

[1177] The server automatically generates reports and correspondence based on the analyzed data and emotion recognition results.

[1178] The user can review the generated document on the terminal and make corrections as necessary.

[1179] 5. Sending documents:

[1180] The final, verified document is sent from the server to the parent via email or in-app message.

[1181] Specific examples

[1182] Use case 1: Recording a field trip

[1183] When a user takes their children on a field trip, they use the system in the following steps:

[1184] 1. The user starts up the device and presses the "Start Activity" button to record the field trip.

[1185] 2. The device uses a camera to record the children's play and a microphone to simultaneously record their voices. This data is then sent to a server at regular intervals.

[1186] 3. The server performs video and audio analysis, and extracts important scenes and keywords using facial expression and voice recognition.

[1187] 4. The server automatically generates a report on the field trip based on this information. For example, if it recognizes that the field trip is going well, it will write a report in a cheerful tone and include many photos of the children smiling.

[1188] 5. The user reviews the generated report and makes any necessary corrections.

[1189] 6. Click the "Send" button to send the finalized report to the parent.

[1190] Example prompt sentence:

[1191] Generate appropriate support messages based on the user's emotional state:

[1192] "Emotion: Confused" "User says: 'I don't know which product to buy.'"

[1193] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1194] Step 1:

[1195] The user starts the device, opens the application and presses the "Start Activity" button.

[1196] Input: Launching a terminal and operating applications.

[1197] What it does: The device activates the camera and microphone, ready to record your child's activities in real time.

[1198] Output: Camera and microphone are activated.

[1199] Step 2:

[1200] The device records video and audio in real time and transmits the data to a server in the cloud at regular intervals.

[1201] Input: Real-time data from camera and microphone.

[1202] How it works: The device's camera captures video and microphone records audio. The data is packetized at regular intervals and sent to a server.

[1203] Output: Transmitted video and audio data.

[1204] Step 3:

[1205] The server analyzes the received video and audio data.

[1206] Input: Video and audio data sent from the device.

[1207] How it works: The server uses OpenCV to detect faces from video data and DeepFace to analyze emotions, and Hugging Face's transformers to recognize audio data and extract tones and specific keywords.

[1208] Output: Analysis results (e.g. emotion labels, speech recognition results).

[1209] Step 4:

[1210] The server automatically generates a report or communication based on the analysis results.

[1211] Input: Emotion recognition results from video data, speech recognition results from audio data.

[1212] How it works: The server combines the analysis results and uses natural language generation technology to create reports and correspondence. It selects appropriate expressions based on the user's emotions and composes the text.

[1213] Output: Auto-generated report or correspondence.

[1214] Step 5:

[1215] The user reviews the automatically generated report or correspondence and corrects it if necessary.

[1216] Input: An automatically generated report or correspondence.

[1217] How it works: The user views the generated document on their device, reviews the content, and then edits the text and adds images and comments if necessary.

[1218] Output: The final verified and corrected document.

[1219] Step 6:

[1220] The final confirmed document is sent from the server to the parent.

[1221] Input: Reviewed and amended final document.

[1222] How it works: When the user presses the "Send" button, the device sends the final document to the server, which then sends it to the parent's email address or as an in-app message.

[1223] Output: Document sent to parents.

[1224] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1225] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1226] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[1227] [Third embodiment]

[1228] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[1229] 5, the data processing system 310 includes the data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[1230] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1231] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[1232] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1233] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1234] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1235] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1236] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1237] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1238] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1239] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[1240] The present invention is a system for reducing the workload of childcare workers and improving communication with parents. This system efficiently records children's activities and automatically generates reports and correspondence for parents. The program processing of the system and specific examples are described below.

[1241] System Overview

[1242] This system consists of a device held by the user (childcare worker), a cloud server, and a device held by the parent. The user uses the device to record the children's activities, and the server analyzes the data to automatically generate reports and correspondence. The final generated documents are checked and revised by the user before being sent to the parent.

[1243] Program processing

[1244] Starting and recording an activity

[1245] 1. User Action:

[1246] The user turns on the device and opens the app.

[1247] Press the "Start Activity" button in the app to begin recording.

[1248] 2. Device behavior:

[1249] The device activates the camera and microphone to record the children's activities in real time using video and audio.

[1250] The recorded data is sent to a server at regular intervals.

[1251] Data Analysis and Document Generation

[1252] 3. Server Operation:

[1253] The server analyzes the received video data and uses image recognition technology to identify important scenes (e.g., scenes of people playing on playground equipment).

[1254] Similarly, voice data is analyzed and specific keywords (e.g., "fun!", "lunch") are extracted using voice recognition technology.

[1255] Based on the analysis results, reports and messages to parents are automatically generated.

[1256] Checking, correcting and sending documents

[1257] 4. User confirmation and correction:

[1258] The user checks the automatically generated document on the terminal.

[1259] Modify the document as needed.

[1260] 5. Sending documents:

[1261] When the user presses the "send" button, the terminal sends the final verified document to the server.

[1262] The server sends the document to the parent via email or in-app message.

[1263] Specific examples

[1264] Example 1: Field trip record

[1265] Below is a specific example of when a user (a nursery teacher) takes children on a field trip.

[1266] 1. User Action:

[1267] The user turns on the device and opens the app to record the field trip.

[1268] Press "Start Activity" to begin recording.

[1269] 2. Device behavior:

[1270] The device uses a camera to record the children playing and simultaneously records audio using a microphone.

[1271] The recorded data is sent to a server at regular intervals.

[1272] 3. Server Operation:

[1273] The server analyzes the video data and extracts scenes such as "scenes of children playing on playground equipment" and "scenes of children gathering together."

[1274] The statements made by nursery teachers and children are converted into text from the audio data, and important phrases (e.g., "It's fun," "Be careful") are extracted.

[1275] A field trip report is automatically generated based on this information.

[1276] 4. User confirmation and correction:

[1277] The user checks the generated report on the terminal and makes corrections as necessary.

[1278] You can also add photos and additional comments to your report.

[1279] 5. Sending documents:

[1280] Once the report has been revised, click the "Send" button to send it to the parent.

[1281] The server will send the final report to the parent's email address or within the app.

[1282] The above is a form for implementing the invention of a system for reducing the workload of childcare workers and sharing information with parents quickly and accurately.

[1283] The processing flow will be explained below.

[1284] Step 1:

[1285] The user launches the app on their device and presses the "Start Activity" button.

[1286] How it works: To record their child's activity, the user launches the app and selects the "Start Activity" button.

[1287] Step 2:

[1288] The device will turn on the camera and microphone and begin recording.

[1289] Action Details: The device activates the camera and microphone and begins real-time video and audio recording.

[1290] Step 3:

[1291] The device stores recorded video and audio data locally.

[1292] Operation details: The device captures video and audio data in real time and stores it in local storage.

[1293] Step 4:

[1294] The terminal transmits the recorded data to the server at regular intervals.

[1295] Operation details: The device uploads collected video and audio data to the server at regular intervals.

[1296] Step 5:

[1297] The server analyzes the received video data and extracts important scenes.

[1298] How it works: The server uses image recognition technology to analyze the content of the video and identify specific activities or behaviors.

[1299] Step 6:

[1300] The server analyzes the received audio data and extracts specific keywords and phrases.

[1301] How it works: The server uses speech recognition technology to convert speech into text and extract key information.

[1302] Step 7:

[1303] The server automatically generates reports and correspondence based on the analysis results.

[1304] Operation details: The server automatically generates reports and correspondence based on the analyzed data and makes them available to the user for review.

[1305] Step 8:

[1306] The server sends the generated document to the terminal.

[1307] Operation details: The server sends the generated reports and messages to the terminal so that the user can check and edit them.

[1308] Step 9:

[1309] The user checks and modifies the document generated on the terminal.

[1310] Operation details: The user checks the automatically generated document on the device and corrects any necessary parts.

[1311] Step 10:

[1312] The user confirms the document after making corrections and presses the "Send" button.

[1313] Operation details: The user confirms the final confirmed document and presses the "Send" button to send it.

[1314] Step 11:

[1315] The terminal transmits the final confirmed document to the server.

[1316] Operation details: The terminal sends the corrected document data to the server.

[1317] Step 12:

[1318] The server sends a final report or communication to the parent.

[1319] Operation details: The server sends the document to the parent's email address or via the in-app messaging function.

[1320] Step 13:

[1321] The parent receives and reviews the document.

[1322] How it works: Parents receive and review the documents sent to them, and can reply in-app or via email if they have any feedback or questions.

[1323] The above are the specific steps in the processing flow of the program.

[1324] Example 1

[1325] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1326] There is a need to reduce the workload of childcare workers and improve communication with parents. However, conventional manual recording and reporting tasks require a great deal of time and effort, placing a heavy burden on childcare workers. Furthermore, manual report creation is prone to subjectivity and can lack accuracy. The objective of this invention is to solve these problems and provide a system that enables efficient and accurate activity recording and reporting.

[1327] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1328] In this invention, the server includes a data analysis means, a document generation means, and a transmission means. This makes it possible to analyze received data and extract specific scenes. Furthermore, it is possible to automatically generate reports or communication messages based on the analysis results using a generative AI model, and the final confirmed documents can be quickly sent to the guardian.

[1329] The "image capturing means" is a video capturing device for recording children's activities in real time.

[1330] The "audio recording means" is a recording device for recording the voices of children during their activities.

[1331] The "transmission means" is a communication device or system for transmitting the recorded image and audio data to a cloud server via the Internet.

[1332] A "cloud server" is a remote server accessible via the Internet that performs processes such as data storage, analysis, and document generation.

[1333] "Data analysis means" refers to software or a system for analyzing received image and audio data and extracting specific scenes and important information.

[1334] "Document generation means" refers to software or systems such as generative AI models that automatically generate reports or correspondence based on analysis results.

[1335] "Verification and correction means" refers to the means by which a user can verify the automatically generated document and correct it if necessary.

[1336] "Transmission means" refers to a communication device or system for transmitting the finalized document to the parent.

[1337] The present invention is a system for reducing the workload of childcare workers and improving communication with parents. This system efficiently records children's activities and automatically generates reports and correspondence for parents. Specific embodiments of the system are described below.

[1338] This system consists of a device held by the user (childcare worker), a cloud server, and a device held by the parent. The user uses the device to record the children's activities, and the cloud server analyzes the data to automatically generate reports and correspondence. The final generated documents are checked and revised by the user before being sent to the parent.

[1339] Hardware and software used

[1340] Device: Smartphone or tablet (e.g. iPad) used by the childcare worker

[1341] Cloud server: A cloud server running on AWS (Amazon Web Services)

[1342] Software components:

[1343] Image recognition technology: OpenCV

[1344] Speech recognition technology: Google Cloud Speech-to-Text

[1345] Natural Language Generation: GPT-3 (generative AI model)

[1346] Communication method: Firebase

[1347] A detailed description of the system's program processing

[1348] 1. To record the activities of the children under the user's care, the user starts up the device and opens a dedicated app (e.g., a childcare activity recording app). By tapping the "Start Activity" button, the device prepares to record activities in real time.

[1349] 2. The device activates the camera and microphone, recording the children's activities in real time with video and audio. For example, it records them playing on playground equipment or lunchtime. The user can check the recording status on the screen.

[1350] 3. The device sends the recorded video and audio data to a cloud server (e.g., AWS) at regular intervals (e.g., every 5 minutes). The transmission is performed in the background so as not to interfere with user operations.

[1351] 4. The server analyzes the received video data using OpenCV and extracts specific keyframes (e.g., scenes of people playing on playground equipment, group photos, etc.). At the same time, it converts the audio data into text using Google Cloud Speech-to-Text and detects important phrases (e.g., "It's fun!", "Be careful").

[1352] 5. Based on the analysis results, the server uses a generative AI model (e.g., GPT-3) to automatically generate reports and messages to parents. The reports automatically include photos based on the extracted keyframes and include text containing important phrases.

[1353] 6. The user can review the report generated on their device, make corrections or add comments as needed, and in some cases even add their own photos or videos.

[1354] 7. When the user taps the "Send" button, the device resends the final confirmed document to the cloud server, which then sends the received document to the parent's email address or as an in-app message.

[1355] Specific examples

[1356] As a specific example, we will show a case where a user (a nursery teacher) takes children on a field trip.

[1357] 1. Before the field trip begins, the user starts the iPad and opens the childcare activity recording app. They tap the "Start Activity" button to begin recording.

[1358] 2. Use the device's camera to record children having fun playing on the playground equipment, and record their audio with the microphone.

[1359] 3. Recorded video and audio data is sent to the AWS cloud server every five minutes.

[1360] 4. The server analyzes the received data, identifies "scenes of playing on playground equipment" and "group photos," and extracts important phrases from the audio, such as "This is fun" and "Be careful."

[1361] 5. A generative AI model (GPT-3) is used to automatically generate a report of the field trip, forming a sentence containing the extracted scenes and key phrases.

[1362] 6. The nursery teacher checks the generated report on the iPad and makes corrections or adds photos as necessary.

[1363] 7. The teacher taps the "Send" button to send the verified report to the server, and the final report is sent to the parent via email or in-app message.

[1364] Prompt Sentence Examples

[1365] "Automatically generate a report that includes scenes of children playing on playground equipment on a field trip and comments that sound like they're having fun."

[1366] The above is a specific embodiment for carrying out the invention, which reduces the workload of childcare workers and enables quick and accurate information sharing with parents.

[1367] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1368] Step 1:

[1369] The user starts the activity

[1370] The user starts up the device, opens the dedicated app, and taps the "Start Activity" button to begin recording.

[1371] Input: Activity start operation (button tap)

[1372] Output: The device will turn on the camera and microphone and be ready for real-time recording.

[1373] Step 2:

[1374] The device records the data

[1375] The device activates a camera and microphone to record video and audio of the children's activities in real time, such as taking pictures of them playing on playground equipment or taking group photos, and recording audio.

[1376] Input: Video and audio of children's activities

[1377] Output: Real-time recorded video and audio data

[1378] Step 3:

[1379] The device sends data to the server

[1380] The device sends recorded video and audio data to a cloud server at regular intervals (e.g., every 5 minutes). The transmission occurs in the background, so it does not interfere with user operations.

[1381] Input: Recorded video and audio data

[1382] Output: Video and audio data sent to the server

[1383] Step 4:

[1384] The server analyzes the data

[1385] The server analyzes the received video data using OpenCV to extract specific keyframes (e.g., scenes of children playing on playground equipment), and converts the audio data into text using Google Cloud Speech-to-Text to detect important phrases (e.g., "This is fun!", "Be careful!").

[1386] Input: Video and audio data sent to the server

[1387] Output: Extracted keyframes, key phrases converted to text

[1388] Step 5:

[1389] The server generates the document

[1390] Based on the analyzed data, the server uses a generative AI model (e.g., GPT-3) to automatically generate reports and parental communications. The reports include photos based on the extracted keyframes and contain text containing important phrases.

[1391] Input: Extracted keyframes, key phrases converted to text

[1392] Output: Automatically generated reports and correspondence

[1393] Step 6:

[1394] The user checks and modifies the document

[1395] Users can review the report generated on their device, make corrections or add comments as needed, and can also add their own photos and videos.

[1396] Input: Auto-generated reports and correspondence

[1397] Output: Revised final report and correspondence

[1398] Step 7:

[1399] A user submits a document

[1400] When the user taps the "Send" button, the device resends the finalized document to the cloud server, which then sends the received document to the parent's email address or as an in-app message.

[1401] Input: Revised Final Report, Correspondence

[1402] Output: Final report and correspondence sent to parents

[1403] The above are the specific processing steps of the system program and their detailed operations.

[1404] (Application example 1)

[1405] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1406] Customer service work in brick-and-mortar stores often places a heavy burden on employees. Recording customer interactions and creating reports afterward takes time and effort, hindering efficient business operations. Furthermore, if the content of customer interactions is not accurately recorded, it becomes difficult to improve service quality and customer satisfaction. To solve these issues, a system is needed that automatically records customer interactions, analyzes the content of the interactions, and generates reports.

[1407] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1408] In this invention, the server includes an image capturing means for recording customer activities, an audio recording means for recording audio during the activities, a data analysis means for analyzing the recorded image and audio data to extract important information, a document generation means for automatically generating a report or communication based on the extracted information, a confirmation / modification means for allowing a user to confirm and modify the document, and a transmission means for transmitting the final confirmed document to relevant parties. This automates the recording of customer interactions in physical stores, reduces the workload of employees, and improves the quality of customer service.

[1409] "Customer activity" refers to purchases, inquiries, complaints, and other actions taken by customers in the store.

[1410] The "image capturing means" is a means for recording customer activities as video using a device such as a camera.

[1411] "Audio recording means" refers to a means for recording the voice of a customer during an activity using a device such as a microphone.

[1412] The "data analysis means" is a means for analyzing the recorded image and audio data and extracting important information.

[1413] The "document generation means" is a means for automatically generating a report or communication based on the extracted information.

[1414] "Means for checking and correcting" refers to the means by which a user can check the automatically generated document and correct its contents if necessary.

[1415] "Transmission means" refers to a means for transmitting the final confirmed document to the parties involved.

[1416] "Real-time" refers to data being processed with minimal delay from the moment it is created.

[1417] "Server" refers to a computer system for receiving recorded data, analyzing it, and generating reports.

[1418] "Stakeholders" refers to store employees and managers involved in customer service, or the customers themselves.

[1419] The present invention is a system for improving the efficiency of customer service operations in brick-and-mortar stores and reducing the workload of employees. This system records customer activity, extracts important information, and automatically generates reports or correspondence. The following describes the system's program processing and specific examples.

[1420] System configuration

[1421] This system consists of devices held by store employees (users), a cloud server, and devices for employees and related departments. Users use their devices to record customer activity, and the server analyzes the data to automatically generate reports and correspondence. The final generated documents are then checked and revised by the users before being sent to the relevant parties.

[1422] Program processing configuration

[1423] The system includes the following major processing steps:

[1424] Initiating and recording customer activity

[1425] 1. User Action:

[1426] The user turns on the device and opens the app.

[1427] Press the "Start Activity" button in the app to begin recording.

[1428] 2. Device behavior:

[1429] The device activates a camera and microphone to record customer activity in real-time video and audio.

[1430] The recorded data is sent to a server at regular intervals.

[1431] Data Analysis and Document Generation

[1432] 3. Server Operation:

[1433] The server analyzes the received video data and identifies important scenes using image recognition technology (e.g., OpenCV or TensorFlow).

[1434] Similarly, voice data is analyzed and specific keywords are extracted using voice recognition technology (e.g., Google Speech Recognition API).

[1435] Based on the analysis results, reports and correspondence to stakeholders are automatically generated. Generative AI models are used to create natural-looking documents.

[1436] Checking, correcting and sending documents

[1437] 4. User confirmation and correction:

[1438] The user checks the automatically generated document on the terminal.

[1439] Modify the document as needed.

[1440] 5. Sending documents:

[1441] When the user presses the "send" button, the terminal sends the final verified document to the server.

[1442] The server sends the document to the relevant parties via email or in-app message.

[1443] Specific examples

[1444] Example 1: Handling complaints at a store

[1445] A specific example when a user (store employee) receives a complaint from a customer is shown below.

[1446] 1. User Action:

[1447] The user turns on the device and opens the app to record the complaint.

[1448] Press "Start Activity" to begin recording.

[1449] 2. Device behavior:

[1450] The device uses a camera to record interactions between customers and employees, while simultaneously recording audio using a microphone.

[1451] The recorded data is sent to a server at regular intervals.

[1452] 3. Server Operation:

[1453] The server analyzes the video data and extracts "scenes where complaints are handled" and "important comments from customers."

[1454] The speech of employees and customers is converted into text from the audio data, and important phrases are extracted, such as keywords such as "I want to return the product" or "I'm dissatisfied."

[1455] A claim report is automatically generated based on this information.

[1456] 4. User confirmation and correction:

[1457] The user checks the generated report on the terminal and makes corrections as necessary.

[1458] You can also add photos and additional comments to your report.

[1459] 5. Sending documents:

[1460] Once the report has been revised, click the "Send" button to send it to the relevant parties.

[1461] The server will send the final report to the relevant parties' email addresses or within the app.

[1462] Prompt Sentence Examples

[1463] "Please start recording the customer interaction."

[1464] "We have completed the customer service recording. We will generate a report based on the analysis results."

[1465] This system automates the recording of customer interactions in physical stores, reducing the workload of employees and improving the quality of customer service.

[1466] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1467] Step 1:

[1468] User Action:

[1469] The user starts up the device, opens the app, and presses the "Start Activity" button within the app to begin recording.

[1470] input:

[1471] App launch information, activity start instructions

[1472] output:

[1473] Recording start state

[1474] Specific operation:

[1475] The device displays the application interface, and the user can enter recording mode by tapping the "Start Activity" button.

[1476] Step 2:

[1477] Terminal behavior:

[1478] The device activates the camera and microphone to record the customer's activities in real time using video and audio, and the recorded data is sent to a server at regular intervals.

[1479] input:

[1480] Recording start command

[1481] output:

[1482] Real-time video and audio data

[1483] Specific operation:

[1484] The device's camera records video of interactions between customers and employees, and the microphone records audio, which is then uploaded to a server at regular intervals.

[1485] Step 3:

[1486] Server behavior:

[1487] The server analyzes the received video data and identifies important scenes using image recognition technology, as well as audio data and extracts specific keywords using voice recognition technology.

[1488] input:

[1489] Video data, audio data

[1490] output:

[1491] Text data (extracted important information)

[1492] Specific operation:

[1493] The server analyzes the video data using image recognition software such as OpenCV and TensorFlow to identify specific scenes, and converts the audio data into text using the Google Speech Recognition API to extract key phrases.

[1494] Step 4:

[1495] Server behavior:

[1496] The server automatically generates reports and correspondence for stakeholders based on the analysis results, using a generative AI model to create natural-looking documents.

[1497] input:

[1498] Text data (extracted important information)

[1499] output:

[1500] Auto-generated reports or correspondence

[1501] Specific operation:

[1502] The server uses a generative AI model to automatically generate appropriate documents based on the extracted key information, using Natural Language Generation (NLG) technology.

[1503] Step 5:

[1504] User Action:

[1505] The user checks the automatically generated document on the terminal and corrects the content as necessary.

[1506] input:

[1507] Auto-generated reports or correspondence

[1508] output:

[1509] Corrected document

[1510] Specific operation:

[1511] The user checks the automatically generated report displayed on the terminal and uses the editing function to correct any errors or necessary additional information.

[1512] Step 6:

[1513] User Action:

[1514] When the user presses the "Send" button, the device sends the finalized document to the server, which then sends it to the relevant parties via email or in-app message.

[1515] input:

[1516] Corrected documents, transmission instructions

[1517] output:

[1518] Final report sent to stakeholders

[1519] Specific operation:

[1520] The user taps the send button, and the device sends the finalized document to the server, which then sends it to the relevant parties via email service or notification API.

[1521] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1522] This invention combines a system for reducing the workload of childcare workers and improving communication with parents with an emotion engine that recognizes the user's emotions. This system efficiently records children's activities and has the function of automatically generating reports and correspondence for parents, as well as generating documents that take the user's emotions into consideration. The system's program processing and specific examples are described below.

[1523] System Overview

[1524] This system consists of a device held by the childcare worker (user), a cloud server, and a parent's device. The user uses the device to record the children's activities, which is then analyzed by the server, and an emotion engine is used to recognize the user's emotions. Based on this, reports and correspondence are automatically generated, and the user can then review and edit the final documents before sending them to the parents.

[1525] Program processing

[1526] Starting and recording an activity

[1527] 1. User Action:

[1528] The user turns on the device and opens the app.

[1529] Press the "Start Activity" button in the app to begin recording.

[1530] 2. Device behavior:

[1531] The device activates the camera and microphone to record the children's activities in real time using video and audio.

[1532] The recorded data is sent to a server at regular intervals.

[1533] Data analysis, emotion recognition and document generation

[1534] 3. Server Operation:

[1535] The server analyzes the received video data and uses image recognition technology to identify important scenes.

[1536] Similarly, voice data is analyzed and specific keywords and phrases are extracted using voice recognition technology.

[1537] The server then uses an emotion engine to recognize the user's emotions from the received video and audio data. For example, it analyzes the user's tone of voice and facial expressions to extract emotions such as joy, sadness, and surprise.

[1538] The server automatically generates reports and correspondence based on the analysis and emotion recognition results, incorporating phrases and tones that correspond to the user's emotions to better tailor the content of the documents.

[1539] Checking, correcting and sending documents

[1540] 4. User confirmation and correction:

[1541] The user checks the automatically generated document on the terminal.

[1542] If necessary, the contents of the document are modified to suit the user's wishes.

[1543] 5. Sending documents:

[1544] When the user presses the "send" button, the terminal sends the final verified document to the server.

[1545] The server sends the document to the parent via email or in-app message.

[1546] Specific examples

[1547] Example 1: Field trip record

[1548] Below is a specific example of when a user (a nursery teacher) takes children on a field trip.

[1549] 1. User Action:

[1550] The user turns on the device and opens the app to record the field trip.

[1551] Press "Start Activity" to begin recording.

[1552] 2. Device behavior:

[1553] The device uses a camera to record the children playing and simultaneously records audio using a microphone.

[1554] The recorded data is sent to a server at regular intervals.

[1555] 3. Server Operation:

[1556] The server analyzes the video data and extracts scenes such as "scenes of children playing on playground equipment" and "scenes of children gathering together."

[1557] The statements made by nursery teachers and children are converted into text from the audio data, and important phrases (e.g., "It's fun," "Be careful") are extracted.

[1558] At the same time, an emotion engine is used to identify the emotions of the caregivers and children.

[1559] Based on this information, a report on the field trip is automatically generated. For example, if the children appear to be having a good time, the document will be adjusted to include more upbeat phrases and photos of smiling children.

[1560] 4. User confirmation and correction:

[1561] The user checks the generated report on the terminal and makes corrections as necessary.

[1562] You can also add photos and additional comments to your report.

[1563] 5. Sending documents:

[1564] Once the report has been revised, click the "Send" button to send it to the parent.

[1565] The server will send the final report to the parent's email address or within the app.

[1566] The above is an embodiment of the invention of a system for reducing the workload of childcare workers and sharing information with parents quickly and accurately, combined with an emotion engine.

[1567] The processing flow will be explained below.

[1568] Step 1:

[1569] The user launches the app on their device and presses the "Start Activity" button.

[1570] Operation details: At the start of an excursion, the user turns on their smartphone or tablet, opens the app, and presses the "Start Activity" button.

[1571] Step 2:

[1572] The device will turn on the camera and microphone and begin recording.

[1573] What it does: The device activates its camera and microphone and begins recording video and audio of the child's activities in real time.

[1574] Step 3:

[1575] The device stores recorded video and audio data locally.

[1576] Operation details: The device captures video and audio data in real time and stores it in local storage.

[1577] Step 4:

[1578] The terminal transmits the recorded data to the server at regular intervals.

[1579] Operation details: The device uploads collected video and audio data to the server at regular intervals.

[1580] Step 5:

[1581] The server analyzes the received video data and extracts important scenes.

[1582] Operation details: The server uses a video analysis algorithm to automatically extract important scenes, such as playing on playground equipment or group photos.

[1583] Step 6:

[1584] The server analyzes the received audio data and extracts specific keywords and phrases.

[1585] How it works: The server uses speech recognition technology to convert the voice data into text and extracts key phrases (e.g., "fun," "next is lunchtime").

[1586] Step 7:

[1587] The server uses an emotion engine to recognize the user's emotions based on video and audio data.

[1588] How it works: The server uses an emotion engine to analyze facial expressions and vocal tone in the video to identify emotions such as joy, surprise, and calm.

[1589] Step 8:

[1590] The server automatically generates reports and correspondence based on the analysis results and emotion recognition results.

[1591] Operation details: The server automatically generates reports and communication messages that reflect the children's activities and the feelings of the caregivers. When children are having fun, it uses a lot of positive expressions to make the reports more appealing.

[1592] Step 9:

[1593] The server sends the generated document to the terminal.

[1594] Operation details: Automatically generated reports and correspondence are sent to the terminal so that the user can check and edit them.

[1595] Step 10:

[1596] The user checks the document generated on the terminal and corrects it if necessary.

[1597] How it works: The user reviews the report and correspondence, enters any necessary corrections or additions, and creates the final version.

[1598] Step 11:

[1599] The user confirms the document after making corrections and presses the "Send" button.

[1600] Operation details: The user confirms the final confirmed document and presses the "Send" button to complete the transmission.

[1601] Step 12:

[1602] The terminal transmits the final confirmed document to the server.

[1603] Operation details: The terminal sends the corrected document data to the server.

[1604] Step 13:

[1605] The server sends a final report or communication to the parent.

[1606] Operational Details: The server will send a final report or communication to the parent via email address or in-app messaging.

[1607] Step 14:

[1608] The parent receives and reviews the document.

[1609] How it works: Parents receive the report or communication sent to them, review the contents, and provide feedback if necessary.

[1610] The above is a specific processing flow of the present invention. By combining it with an emotion engine, it becomes possible to report information that reflects the user's emotions, further improving communication with parents.

[1611] Example 2

[1612] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1613] In today's childcare environment, childcare workers are overwhelmed with their busy daily work, making it difficult for them to record children's activities in detail and report them to parents quickly and accurately. Furthermore, in order to communicate with parents, appropriate document generation that takes emotions into account is required, but doing this manually requires a great deal of time and effort. Therefore, a technology is needed that can accurately record children's activities and automatically generate reports that reflect their emotions while reducing the workload of childcare workers.

[1614] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes a data analysis means, an emotion recognition means, and a document generation means. This makes it possible to efficiently analyze the children's activity data and automatically generate appropriate reports and communication messages that take the user's emotions into consideration.

[1615] "Filming means" refers to devices and functions for recording children's activities as video in real time.

[1616] "Audio recording means" refers to a device or function for recording audio during an activity.

[1617] "Data analysis means" refers to devices or functions that analyze recorded image and audio data and extract important information.

[1618] The "document generation means" refers to a device or function for automatically generating reports and correspondence based on the extracted information and the user's feelings.

[1619] "Emotion recognition means" refers to a device or function for recognizing emotions from the user's tone of voice, facial expression, etc.

[1620] "Verification and correction means" refers to devices and functions that allow users to verify automatically generated documents and correct them as necessary.

[1621] "Transmission means" refers to a device or function for transmitting the final confirmed document to the guardian.

[1622] This invention combines an emotion engine with a system that reduces the workload of childcare workers and improves communication with parents. This system efficiently records children's activities, has the function of automatically generating reports and correspondence for parents, and is capable of generating documents that take the user's emotions into consideration.

[1623] System Overview

[1624] This system consists of a device held by the childcare worker (user), a cloud server, and a parent's device. The user uses the device to record the children's activities, which is then analyzed by the server, and an emotion engine is used to recognize the user's emotions. Based on this, reports and correspondence are automatically generated, and the user can then review and edit the final documents before sending them to the parents.

[1625] Hardware and software used

[1626] Device hardware:

[1627] Smartphone or tablet

[1628] Built-in camera and microphone

[1629] Server software:

[1630] Image recognition technology (e.g., OpenCV, TensorFlow)

[1631] Speech recognition technology (e.g., Google Cloud Speech-to-Text, Amazon Transcribe)

[1632] Emotion recognition engine (e.g. Microsoft Azure Emotion API, Affectiva)

[1633] Generative AI models (e.g., OpenAI GPT-3)

[1634] Program processing overview

[1635] Activity recording

[1636] User Action:

[1637] The user starts up the terminal and launches the dedicated application.

[1638] Press the "Start Activity" button in the app to begin recording mode.

[1639] Terminal behavior:

[1640] The device's camera and microphone will be activated and will record the children's activities in real time.

[1641] Video and audio data is sent to the server at regular intervals (for example, every 10 seconds).

[1642] Data analysis and emotion recognition

[1643] Server behavior:

[1644] The server analyzes the received video and audio data and extracts important information.

[1645] Image recognition technology is used to identify important scenes from video.

[1646] It uses voice recognition technology to extract specific keywords and phrases from speech.

[1647] It uses an emotion engine to recognize emotions from the user's tone of voice and facial expressions.

[1648] Document generation and transmission

[1649] Server behavior:

[1650] Based on the analysis results and emotion recognition results, the server automatically generates reports and correspondence using a generative AI model.

[1651] Tailor your document content with phrases and tones that reflect your emotions.

[1652] User Action:

[1653] The user can check the generated document on their own terminal and make corrections as necessary.

[1654] Click the "Send" button to send the revised document to the parent.

[1655] Examples:

[1656] For example, when a user takes their children on a field trip, the system operates as follows:

[1657] User Action:

[1658] The user starts the device and opens the application to record the field trip.

[1659] Press the "Start Activity" button to begin recording.

[1660] Terminal behavior:

[1661] The device's camera records the children's play and the microphone records their audio.

[1662] The recorded data is sent to the server at regular intervals.

[1663] Server behavior:

[1664] The server analyzes the video data and extracts scenes such as "scenes of children playing on playground equipment" and "scenes of children gathering together."

[1665] The system analyzes audio data and extracts important phrases from the speech of caregivers and children.

[1666] An emotion engine is used to identify the emotions of caregivers and children.

[1667] Based on the above information, a report on the field trip is automatically generated, describing the fun the students had in cheerful phrases.

[1668] User confirmation and correction:

[1669] The user reviews the generated report and makes corrections as necessary.

[1670] For example, add photos and additional comments to the report.

[1671] Sending documents:

[1672] Once the report has been revised, click the "Send" button to send it to the parent.

[1673] The server will send the final report to the parent's email address or within the app.

[1674] Prompt Sentence Examples

[1675] "Childcare workers record the children's field trips, and based on the records, automatically generate reports for parents. The generated documents also include emotional information from the childcare workers and the children. For example, fun scenes are described in cheerful language, while scenes requiring caution are described in polite language."

[1676] The above is an embodiment of the invention.

[1677] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1678] Step 1: Starting the activity

[1679] User Action:

[1680] The user starts up the terminal and launches the dedicated application.

[1681] Press the "Start Activity" button in the app to begin recording mode.

[1682] Specific behavior:

[1683] A user taps an app on their smartphone or tablet to launch it.

[1684] Pressing the "Start Activity" button will take you to the activity recording screen.

[1685] Step 2: Record and send data

[1686] Terminal behavior:

[1687] The device activates a camera and microphone to record the children's activities in real time.

[1688] Video and audio data is sent to the server at regular intervals (for example, every 10 seconds).

[1689] Input and Output:

[1690] (Input) Photographed video and recorded audio data

[1691] (Output) Split video and audio files sent to the server

[1692] Specific behavior:

[1693] Images captured by the built-in camera and audio recorded by the microphone are saved in a data buffer in real time.

[1694] The divided data is uploaded to the server sequentially (e.g., a data file is sent every 10 seconds).

[1695] Step 3: Data analysis

[1696] Server behavior:

[1697] The server analyzes the received video and audio data and extracts important information.

[1698] Image recognition technology is used to identify important scenes from video.

[1699] Using voice recognition technology, the voice data is converted into text and important keywords and phrases are extracted.

[1700] Input and Output:

[1701] (Input) Received video and audio data

[1702] (Output) Analyzed important scene information and textual keywords

[1703] Specific behavior:

[1704] Image recognition technology (e.g., OpenCV, TensorFlow) is used to analyze specific frames in the video and extract "smiling scenes" and "group photo scenes."

[1705] Voice recognition technology (e.g., Google Cloud Speech-to-Text) is used to convert the voice data into text, and important phrases such as "It's fun" and "Be careful" are extracted from the comments of the caregivers and children.

[1706] Step 4: Emotion Recognition

[1707] Server behavior:

[1708] The server uses an emotion recognition engine to recognize emotions from the user's tone of voice and facial expressions.

[1709] Input and Output:

[1710] (Input) Analyzed video and audio data

[1711] (Output) Recognized user emotion data

[1712] Specific behavior:

[1713] An emotion recognition engine (e.g., Microsoft Azure Emotion API) is used to identify emotions (e.g., joy, sadness, surprise) from facial expressions and tone of voice in the video.

[1714] The recognized emotion data is stored in a database as emotion tags.

[1715] Step 5: Document Generation

[1716] Server behavior:

[1717] Based on the analysis results and emotion recognition results, the server automatically generates reports and correspondence using a generative AI model.

[1718] Tailor your document content with phrases and tones that reflect your emotions.

[1719] Input and Output:

[1720] (Input) Important scene information, text keywords, emotion data

[1721] (Output) Automatically generated reports and correspondence

[1722] Specific behavior:

[1723] Using a generative AI model (e.g., OpenAI GPT-3), documents are automatically generated based on analysis and emotion recognition results.

[1724] The document will contain information such as "We had a great time on our field trip today," and depending on the emotion tag, it will add details such as "We saw lots of smiling children."

[1725] Step 6: Check and correct the document

[1726] User Action:

[1727] The user checks the automatically generated document on the terminal and corrects it if necessary.

[1728] Click the "Save" button to confirm the changes.

[1729] Input and Output:

[1730] (Input) Automatically generated report

[1731] (Output) Revised Final Report

[1732] Specific behavior:

[1733] Tap the generated document on the app screen to enter edit mode and manually insert additional comments or photos.

[1734] Save the amendments and prepare them as a final confirmation document.

[1735] Step 7: Sending documents

[1736] User Action:

[1737] When the user presses the "send" button, the terminal sends the final verified document to the server.

[1738] The server sends the final document to the parent.

[1739] Input and Output:

[1740] (Input) Final confirmed report

[1741] (Output) Report sent to parents

[1742] Specific behavior:

[1743] When the user taps the "Send" button, a confirmation message will be displayed.

[1744] Once you agree to the confirmation message, the document is instantly sent to the server and delivered to the parent.

[1745] (Application example 2)

[1746] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1747] Conventional virtual store customer support systems lacked the ability to recognize user emotions in real time, making it difficult to improve the user experience and provide smooth support. They also lacked the ability to immediately detect user confusion or difficulties and provide appropriate real-time support accordingly. This resulted in a poor user experience and inefficient customer support. Furthermore, there was a need for improved accuracy in automatically generated reports based on user behavior.

[1748] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1749] In this invention, the server includes a means for analyzing a user's behavior and emotions in real time, a means for recognizing emotions in real time, and a means for providing feedback based on the recognition results. This makes it possible to provide appropriate support and product suggestions in real time according to the user's emotions. In addition, the efficiency of customer support can be improved by creating highly accurate automatically generated reports based on the user's experience records.

[1750] "Image capture means" refers to a video capture device and its functions for recording children's activities.

[1751] "Audio recording means" refers to a device and its function for recording audio during an activity.

[1752] "Data analysis means" refers to the function of analyzing recorded image and audio data to extract important information.

[1753] "Document generation means" refers to a function that automatically generates a report or communication based on the extracted information.

[1754] "Emotion recognition means" refers to a function that uses an emotion engine to analyze a user's emotions in real time.

[1755] "Feedback means" refers to a function that suggests appropriate actions to the user based on real-time emotion recognition results.

[1756] "Means for checking and correcting" refers to a function that allows users to check and correct automatically generated documents.

[1757] "Transmission means" refers to the function of sending the final confirmed document to the guardian.

[1758] The system for implementing this invention consists of a user's device, a cloud server, and a parent's device. The user uses the device to record their child's activities, and the data is analyzed on the server. Furthermore, an emotion engine is used to recognize the user's emotions in real time and automatically generate reports and correspondence.

[1759] Hardware and software used

[1760] Hardware:

[1761] Terminal: A device such as a smartphone, smart glasses, or head-mounted display.

[1762] Camera: The camera built into the device.

[1763] Microphone: The microphone built into the device.

[1764] software:

[1765] OpenCV: A library for image processing and face detection.

[1766] DeepFace: A library for determining emotions from images.

[1767] Hugging Face's transformers: A library for speech recognition and tone analysis.

[1768] Data processing and calculation

[1769] 1. Image and audio recording:

[1770] The user turns on the device and records their children's activities in real time.

[1771] The device's camera captures video and microphone records audio, and these data are sent to a cloud server at regular intervals.

[1772] 2. Data Analysis and Emotion Recognition:

[1773] The server uses OpenCV to analyze the video data and perform face detection.

[1774] The facial image is analyzed using the DeepFace library to determine emotions (e.g., joy, sadness, surprise, etc.).

[1775] The voice data is recognized using Hugging Face's transformers to extract the user's tone of voice and specific keywords (e.g., "fun," "troubled," etc.).

[1776] 3. Real-time feedback:

[1777] Based on the results of emotion recognition, the server will provide appropriate feedback to the user, for example, if a confused expression is detected, it will immediately display a support message.

[1778] 4. Document Generation:

[1779] The server automatically generates reports and correspondence based on the analyzed data and emotion recognition results.

[1780] The user can review the generated document on the terminal and make corrections as necessary.

[1781] 5. Sending documents:

[1782] The final, verified document is sent from the server to the parent via email or in-app message.

[1783] Specific examples

[1784] Use case 1: Recording a field trip

[1785] When a user takes their children on a field trip, they use the system in the following steps:

[1786] 1. The user starts up the device and presses the "Start Activity" button to record the field trip.

[1787] 2. The device uses a camera to record the children's play and a microphone to simultaneously record their voices. This data is then sent to a server at regular intervals.

[1788] 3. The server performs video and audio analysis, and extracts important scenes and keywords using facial expression and voice recognition.

[1789] 4. The server automatically generates a report on the field trip based on this information. For example, if it recognizes that the field trip is going well, it will write a report in a cheerful tone and include many photos of the children smiling.

[1790] 5. The user reviews the generated report and makes any necessary corrections.

[1791] 6. Click the "Send" button to send the finalized report to the parent.

[1792] Example prompt sentence:

[1793] Generate appropriate support messages based on the user's emotional state:

[1794] "Emotion: Confused" "User says: 'I don't know which product to buy.'"

[1795] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1796] Step 1:

[1797] The user starts the device, opens the application and presses the "Start Activity" button.

[1798] Input: Launching a terminal and operating applications.

[1799] What it does: The device activates the camera and microphone, ready to record your child's activities in real time.

[1800] Output: Camera and microphone are activated.

[1801] Step 2:

[1802] The device records video and audio in real time and transmits the data to a server in the cloud at regular intervals.

[1803] Input: Real-time data from camera and microphone.

[1804] How it works: The device's camera captures video and microphone records audio. The data is packetized at regular intervals and sent to a server.

[1805] Output: Transmitted video and audio data.

[1806] Step 3:

[1807] The server analyzes the received video and audio data.

[1808] Input: Video and audio data sent from the device.

[1809] How it works: The server uses OpenCV to detect faces from video data and DeepFace to analyze emotions, and Hugging Face's transformers to recognize audio data and extract tones and specific keywords.

[1810] Output: Analysis results (e.g. emotion labels, speech recognition results).

[1811] Step 4:

[1812] The server automatically generates a report or communication based on the analysis results.

[1813] Input: Emotion recognition results from video data, speech recognition results from audio data.

[1814] How it works: The server combines the analysis results and uses natural language generation technology to create reports and correspondence. It selects appropriate expressions based on the user's emotions and composes the text.

[1815] Output: Auto-generated report or correspondence.

[1816] Step 5:

[1817] The user reviews the automatically generated report or correspondence and corrects it if necessary.

[1818] Input: An automatically generated report or correspondence.

[1819] How it works: The user views the generated document on their device, reviews the content, and then edits the text and adds images and comments if necessary.

[1820] Output: The final verified and corrected document.

[1821] Step 6:

[1822] The final confirmed document is sent from the server to the parent.

[1823] Input: Reviewed and amended final document.

[1824] How it works: When the user presses the "Send" button, the device sends the final document to the server, which then sends it to the parent's email address or as an in-app message.

[1825] Output: Document sent to parents.

[1826] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1827] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1828] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1829] [Fourth embodiment]

[1830] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1831] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1832] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1833] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1834] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1835] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1836] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1837] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1838] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1839] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1840] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1841] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1842] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1843] The present invention is a system for reducing the workload of childcare workers and improving communication with parents. This system efficiently records children's activities and automatically generates reports and correspondence for parents. The program processing of the system and specific examples are described below.

[1844] System Overview

[1845] This system consists of a device held by the user (childcare worker), a cloud server, and a device held by the parent. The user uses the device to record the children's activities, and the server analyzes the data to automatically generate reports and correspondence. The final generated documents are checked and revised by the user before being sent to the parent.

[1846] Program processing

[1847] Starting and recording an activity

[1848] 1. User Action:

[1849] The user turns on the device and opens the app.

[1850] Press the "Start Activity" button in the app to begin recording.

[1851] 2. Device behavior:

[1852] The device activates the camera and microphone to record the children's activities in real time using video and audio.

[1853] The recorded data is sent to a server at regular intervals.

[1854] Data Analysis and Document Generation

[1855] 3. Server Operation:

[1856] The server analyzes the received video data and uses image recognition technology to identify important scenes (e.g., scenes of people playing on playground equipment).

[1857] Similarly, voice data is analyzed and specific keywords (e.g., "fun!", "lunch") are extracted using voice recognition technology.

[1858] Based on the analysis results, reports and messages to parents are automatically generated.

[1859] Checking, correcting and sending documents

[1860] 4. User confirmation and correction:

[1861] The user checks the automatically generated document on the terminal.

[1862] Modify the document as needed.

[1863] 5. Sending documents:

[1864] When the user presses the "send" button, the terminal sends the final verified document to the server.

[1865] The server sends the document to the parent via email or in-app message.

[1866] Specific examples

[1867] Example 1: Field trip record

[1868] Below is a specific example of when a user (a nursery teacher) takes children on a field trip.

[1869] 1. User Action:

[1870] The user turns on the device and opens the app to record the field trip.

[1871] Press "Start Activity" to begin recording.

[1872] 2. Device behavior:

[1873] The device uses a camera to record the children playing and simultaneously records audio using a microphone.

[1874] The recorded data is sent to a server at regular intervals.

[1875] 3. Server Operation:

[1876] The server analyzes the video data and extracts scenes such as "scenes of children playing on playground equipment" and "scenes of children gathering together."

[1877] The statements made by nursery teachers and children are converted into text from the audio data, and important phrases (e.g., "It's fun," "Be careful") are extracted.

[1878] A field trip report is automatically generated based on this information.

[1879] 4. User confirmation and correction:

[1880] The user checks the generated report on the terminal and makes corrections as necessary.

[1881] You can also add photos and additional comments to your report.

[1882] 5. Sending documents:

[1883] Once the report has been revised, click the "Send" button to send it to the parent.

[1884] The server will send the final report to the parent's email address or within the app.

[1885] The above is a form for implementing the invention of a system for reducing the workload of childcare workers and sharing information with parents quickly and accurately.

[1886] The processing flow will be explained below.

[1887] Step 1:

[1888] The user launches the app on their device and presses the "Start Activity" button.

[1889] How it works: To record their child's activity, the user launches the app and selects the "Start Activity" button.

[1890] Step 2:

[1891] The device will turn on the camera and microphone and begin recording.

[1892] Action Details: The device activates the camera and microphone and begins real-time video and audio recording.

[1893] Step 3:

[1894] The device stores recorded video and audio data locally.

[1895] Operation details: The device captures video and audio data in real time and stores it in local storage.

[1896] Step 4:

[1897] The terminal transmits the recorded data to the server at regular intervals.

[1898] Operation details: The device uploads collected video and audio data to the server at regular intervals.

[1899] Step 5:

[1900] The server analyzes the received video data and extracts important scenes.

[1901] How it works: The server uses image recognition technology to analyze the content of the video and identify specific activities or behaviors.

[1902] Step 6:

[1903] The server analyzes the received audio data and extracts specific keywords and phrases.

[1904] How it works: The server uses speech recognition technology to convert speech into text and extract key information.

[1905] Step 7:

[1906] The server automatically generates reports and correspondence based on the analysis results.

[1907] Operation details: The server automatically generates reports and correspondence based on the analyzed data and makes them available to the user for review.

[1908] Step 8:

[1909] The server sends the generated document to the terminal.

[1910] Operation details: The server sends the generated reports and messages to the terminal so that the user can check and edit them.

[1911] Step 9:

[1912] The user checks and modifies the document generated on the terminal.

[1913] Operation details: The user checks the automatically generated document on the device and corrects any necessary parts.

[1914] Step 10:

[1915] The user confirms the document after making corrections and presses the "Send" button.

[1916] Operation details: The user confirms the final confirmed document and presses the "Send" button to send it.

[1917] Step 11:

[1918] The terminal transmits the final confirmed document to the server.

[1919] Operation details: The terminal sends the corrected document data to the server.

[1920] Step 12:

[1921] The server sends a final report or communication to the parent.

[1922] Operation details: The server sends the document to the parent's email address or via the in-app messaging function.

[1923] Step 13:

[1924] The parent receives and reviews the document.

[1925] How it works: Parents receive and review the documents sent to them, and can reply in-app or via email if they have any feedback or questions.

[1926] The above are the specific steps in the processing flow of the program.

[1927] Example 1

[1928] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1929] There is a need to reduce the workload of childcare workers and improve communication with parents. However, conventional manual recording and reporting tasks require a great deal of time and effort, placing a heavy burden on childcare workers. Furthermore, manual report creation is prone to subjectivity and can lack accuracy. The objective of this invention is to solve these problems and provide a system that enables efficient and accurate activity recording and reporting.

[1930] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1931] In this invention, the server includes a data analysis means, a document generation means, and a transmission means. This makes it possible to analyze received data and extract specific scenes. Furthermore, it is possible to automatically generate reports or communication messages based on the analysis results using a generative AI model, and the final confirmed documents can be quickly sent to the guardian.

[1932] The "image capturing means" is a video capturing device for recording children's activities in real time.

[1933] The "audio recording means" is a recording device for recording the voices of children during their activities.

[1934] The "transmission means" is a communication device or system for transmitting the recorded image and audio data to a cloud server via the Internet.

[1935] A "cloud server" is a remote server accessible via the Internet that performs processes such as data storage, analysis, and document generation.

[1936] "Data analysis means" refers to software or a system for analyzing received image and audio data and extracting specific scenes and important information.

[1937] "Document generation means" refers to software or systems such as generative AI models that automatically generate reports or correspondence based on analysis results.

[1938] "Verification and correction means" refers to the means by which a user can verify the automatically generated document and correct it if necessary.

[1939] "Transmission means" refers to a communication device or system for transmitting the finalized document to the parent.

[1940] The present invention is a system for reducing the workload of childcare workers and improving communication with parents. This system efficiently records children's activities and automatically generates reports and correspondence for parents. Specific embodiments of the system are described below.

[1941] This system consists of a device held by the user (childcare worker), a cloud server, and a device held by the parent. The user uses the device to record the children's activities, and the cloud server analyzes the data to automatically generate reports and correspondence. The final generated documents are checked and revised by the user before being sent to the parent.

[1942] Hardware and software used

[1943] Device: Smartphone or tablet (e.g. iPad) used by the childcare worker

[1944] Cloud server: A cloud server running on AWS (Amazon Web Services)

[1945] Software components:

[1946] Image recognition technology: OpenCV

[1947] Speech recognition technology: Google Cloud Speech-to-Text

[1948] Natural Language Generation: GPT-3 (generative AI model)

[1949] Communication method: Firebase

[1950] A detailed description of the system's program processing

[1951] 1. To record the activities of the children under the user's care, the user starts up the device and opens a dedicated app (e.g., a childcare activity recording app). By tapping the "Start Activity" button, the device prepares to record activities in real time.

[1952] 2. The device activates the camera and microphone, recording the children's activities in real time with video and audio. For example, it records them playing on playground equipment or lunchtime. The user can check the recording status on the screen.

[1953] 3. The device sends the recorded video and audio data to a cloud server (e.g., AWS) at regular intervals (e.g., every 5 minutes). The transmission is performed in the background so as not to interfere with user operations.

[1954] 4. The server analyzes the received video data using OpenCV and extracts specific keyframes (e.g., scenes of people playing on playground equipment, group photos, etc.). At the same time, it converts the audio data into text using Google Cloud Speech-to-Text and detects important phrases (e.g., "It's fun!", "Be careful").

[1955] 5. Based on the analysis results, the server uses a generative AI model (e.g., GPT-3) to automatically generate reports and messages to parents. The reports automatically include photos based on the extracted keyframes and include text containing important phrases.

[1956] 6. The user can review the report generated on their device, make corrections or add comments as needed, and in some cases even add their own photos or videos.

[1957] 7. When the user taps the "Send" button, the device resends the final confirmed document to the cloud server, which then sends the received document to the parent's email address or as an in-app message.

[1958] Specific examples

[1959] As a specific example, we will show a case where a user (a nursery teacher) takes children on a field trip.

[1960] 1. Before the field trip begins, the user starts the iPad and opens the childcare activity recording app. They tap the "Start Activity" button to begin recording.

[1961] 2. Use the device's camera to record children having fun playing on the playground equipment, and record their audio with the microphone.

[1962] 3. Recorded video and audio data is sent to the AWS cloud server every five minutes.

[1963] 4. The server analyzes the received data, identifies "scenes of playing on playground equipment" and "group photos," and extracts important phrases from the audio, such as "This is fun" and "Be careful."

[1964] 5. A generative AI model (GPT-3) is used to automatically generate a report of the field trip, forming a sentence containing the extracted scenes and key phrases.

[1965] 6. The nursery teacher checks the generated report on the iPad and makes corrections or adds photos as necessary.

[1966] 7. The teacher taps the "Send" button to send the verified report to the server, and the final report is sent to the parent via email or in-app message.

[1967] Prompt Sentence Examples

[1968] "Automatically generate a report that includes scenes of children playing on playground equipment on a field trip and comments that sound like they're having fun."

[1969] The above is a specific embodiment for carrying out the invention, which reduces the workload of childcare workers and enables quick and accurate information sharing with parents.

[1970] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1971] Step 1:

[1972] The user starts the activity

[1973] The user starts up the device, opens the dedicated app, and taps the "Start Activity" button to begin recording.

[1974] Input: Activity start operation (button tap)

[1975] Output: The device will turn on the camera and microphone and be ready for real-time recording.

[1976] Step 2:

[1977] The device records the data

[1978] The device activates a camera and microphone to record video and audio of the children's activities in real time, such as taking pictures of them playing on playground equipment or taking group photos, and recording audio.

[1979] Input: Video and audio of children's activities

[1980] Output: Real-time recorded video and audio data

[1981] Step 3:

[1982] The device sends data to the server

[1983] The device sends recorded video and audio data to a cloud server at regular intervals (e.g., every 5 minutes). The transmission occurs in the background, so it does not interfere with user operations.

[1984] Input: Recorded video and audio data

[1985] Output: Video and audio data sent to the server

[1986] Step 4:

[1987] The server analyzes the data

[1988] The server analyzes the received video data using OpenCV to extract specific keyframes (e.g., scenes of children playing on playground equipment), and converts the audio data into text using Google Cloud Speech-to-Text to detect important phrases (e.g., "This is fun!", "Be careful!").

[1989] Input: Video and audio data sent to the server

[1990] Output: Extracted keyframes, key phrases converted to text

[1991] Step 5:

[1992] The server generates the document

[1993] Based on the analyzed data, the server uses a generative AI model (e.g., GPT-3) to automatically generate reports and parental communications. The reports include photos based on the extracted keyframes and contain text containing important phrases.

[1994] Input: Extracted keyframes, key phrases converted to text

[1995] Output: Automatically generated reports and correspondence

[1996] Step 6:

[1997] The user checks and modifies the document

[1998] Users can review the report generated on their device, make corrections or add comments as needed, and can also add their own photos and videos.

[1999] Input: Auto-generated reports and correspondence

[2000] Output: Revised final report and correspondence

[2001] Step 7:

[2002] A user submits a document

[2003] When the user taps the "Send" button, the device resends the finalized document to the cloud server, which then sends the received document to the parent's email address or as an in-app message.

[2004] Input: Revised Final Report, Correspondence

[2005] Output: Final report and correspondence sent to parents

[2006] The above are the specific processing steps of the system program and their detailed operations.

[2007] (Application example 1)

[2008] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[2009] Customer service work in brick-and-mortar stores often places a heavy burden on employees. Recording customer interactions and creating reports afterward takes time and effort, hindering efficient business operations. Furthermore, if the content of customer interactions is not accurately recorded, it becomes difficult to improve service quality and customer satisfaction. To solve these issues, a system is needed that automatically records customer interactions, analyzes the content of the interactions, and generates reports.

[2010] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[2011] In this invention, the server includes an image capturing means for recording customer activities, an audio recording means for recording audio during the activities, a data analysis means for analyzing the recorded image and audio data to extract important information, a document generation means for automatically generating a report or communication based on the extracted information, a confirmation / modification means for allowing a user to confirm and modify the document, and a transmission means for transmitting the final confirmed document to relevant parties. This automates the recording of customer interactions in physical stores, reduces the workload of employees, and improves the quality of customer service.

[2012] "Customer activity" refers to purchases, inquiries, complaints, and other actions taken by customers in the store.

[2013] The "image capturing means" is a means for recording customer activities as video using a device such as a camera.

[2014] "Audio recording means" refers to a means for recording the voice of a customer during an activity using a device such as a microphone.

[2015] The "data analysis means" is a means for analyzing the recorded image and audio data and extracting important information.

[2016] The "document generation means" is a means for automatically generating a report or communication based on the extracted information.

[2017] "Means for checking and correcting" refers to the means by which a user can check the automatically generated document and correct its contents if necessary.

[2018] "Transmission means" refers to a means for transmitting the final confirmed document to the parties involved.

[2019] "Real-time" refers to data being processed with minimal delay from the moment it is created.

[2020] "Server" refers to a computer system for receiving recorded data, analyzing it, and generating reports.

[2021] "Stakeholders" refers to store employees and managers involved in customer service, or the customers themselves.

[2022] The present invention is a system for improving the efficiency of customer service operations in brick-and-mortar stores and reducing the workload of employees. This system records customer activity, extracts important information, and automatically generates reports or correspondence. The following describes the system's program processing and specific examples.

[2023] System configuration

[2024] This system consists of devices held by store employees (users), a cloud server, and devices for employees and related departments. Users use their devices to record customer activity, and the server analyzes the data to automatically generate reports and correspondence. The final generated documents are then checked and revised by the users before being sent to the relevant parties.

[2025] Program processing configuration

[2026] The system includes the following major processing steps:

[2027] Initiating and recording customer activity

[2028] 1. User Action:

[2029] The user turns on the device and opens the app.

[2030] Press the "Start Activity" button in the app to begin recording.

[2031] 2. Device behavior:

[2032] The device activates a camera and microphone to record customer activity in real-time video and audio.

[2033] The recorded data is sent to a server at regular intervals.

[2034] Data Analysis and Document Generation

[2035] 3. Server Operation:

[2036] The server analyzes the received video data and identifies important scenes using image recognition technology (e.g., OpenCV or TensorFlow).

[2037] Similarly, voice data is analyzed and specific keywords are extracted using voice recognition technology (e.g., Google Speech Recognition API).

[2038] Based on the analysis results, reports and correspondence to stakeholders are automatically generated. Generative AI models are used to create natural-looking documents.

[2039] Checking, correcting and sending documents

[2040] 4. User confirmation and correction:

[2041] The user checks the automatically generated document on the terminal.

[2042] Modify the document as needed.

[2043] 5. Sending documents:

[2044] When the user presses the "send" button, the terminal sends the final verified document to the server.

[2045] The server sends the document to the relevant parties via email or in-app message.

[2046] Specific examples

[2047] Example 1: Handling complaints at a store

[2048] A specific example when a user (store employee) receives a complaint from a customer is shown below.

[2049] 1. User Action:

[2050] The user turns on the device and opens the app to record the complaint.

[2051] Press "Start Activity" to begin recording.

[2052] 2. Device behavior:

[2053] The device uses a camera to record interactions between customers and employees, while simultaneously recording audio using a microphone.

[2054] The recorded data is sent to a server at regular intervals.

[2055] 3. Server Operation:

[2056] The server analyzes the video data and extracts "scenes where complaints are handled" and "important comments from customers."

[2057] The speech of employees and customers is converted into text from the audio data, and important phrases are extracted, such as keywords such as "I want to return the product" or "I'm dissatisfied."

[2058] A claim report is automatically generated based on this information.

[2059] 4. User confirmation and correction:

[2060] The user checks the generated report on the terminal and makes corrections as necessary.

[2061] You can also add photos and additional comments to your report.

[2062] 5. Sending documents:

[2063] Once the report has been revised, click the "Send" button to send it to the relevant parties.

[2064] The server will send the final report to the relevant parties' email addresses or within the app.

[2065] Prompt Sentence Examples

[2066] "Please start recording the customer interaction."

[2067] "We have completed the customer service recording. We will generate a report based on the analysis results."

[2068] This system automates the recording of customer interactions in physical stores, reducing the workload of employees and improving the quality of customer service.

[2069] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[2070] Step 1:

[2071] User Action:

[2072] The user starts up the device, opens the app, and presses the "Start Activity" button within the app to begin recording.

[2073] input:

[2074] App launch information, activity start instructions

[2075] output:

[2076] Recording start state

[2077] Specific operation:

[2078] The device displays the application interface, and the user can enter recording mode by tapping the "Start Activity" button.

[2079] Step 2:

[2080] Terminal behavior:

[2081] The device activates the camera and microphone to record the customer's activities in real time using video and audio, and the recorded data is sent to a server at regular intervals.

[2082] input:

[2083] Recording start command

[2084] output:

[2085] Real-time video and audio data

[2086] Specific operation:

[2087] The device's camera records video of interactions between customers and employees, and the microphone records audio, which is then uploaded to a server at regular intervals.

[2088] Step 3:

[2089] Server behavior:

[2090] The server analyzes the received video data and identifies important scenes using image recognition technology, as well as audio data and extracts specific keywords using voice recognition technology.

[2091] input:

[2092] Video data, audio data

[2093] output:

[2094] Text data (extracted important information)

[2095] Specific operation:

[2096] The server analyzes the video data using image recognition software such as OpenCV and TensorFlow to identify specific scenes, and converts the audio data into text using the Google Speech Recognition API to extract key phrases.

[2097] Step 4:

[2098] Server behavior:

[2099] The server automatically generates reports and correspondence for stakeholders based on the analysis results, using a generative AI model to create natural-looking documents.

[2100] input:

[2101] Text data (extracted important information)

[2102] output:

[2103] Auto-generated reports or correspondence

[2104] Specific operation:

[2105] The server uses a generative AI model to automatically generate appropriate documents based on the extracted key information, using Natural Language Generation (NLG) technology.

[2106] Step 5:

[2107] User Action:

[2108] The user checks the automatically generated document on the terminal and corrects the content as necessary.

[2109] input:

[2110] Auto-generated reports or correspondence

[2111] output:

[2112] Corrected document

[2113] Specific operation:

[2114] The user checks the automatically generated report displayed on the terminal and uses the editing function to correct any errors or necessary additional information.

[2115] Step 6:

[2116] User Action:

[2117] When the user presses the "Send" button, the device sends the finalized document to the server, which then sends it to the relevant parties via email or in-app message.

[2118] input:

[2119] Corrected documents, transmission instructions

[2120] output:

[2121] Final report sent to stakeholders

[2122] Specific operation:

[2123] The user taps the send button, and the device sends the finalized document to the server, which then sends it to the relevant parties via email service or notification API.

[2124] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[2125] This invention combines a system for reducing the workload of childcare workers and improving communication with parents with an emotion engine that recognizes the user's emotions. This system efficiently records children's activities and has the function of automatically generating reports and correspondence for parents, as well as generating documents that take the user's emotions into consideration. The system's program processing and specific examples are described below.

[2126] System Overview

[2127] This system consists of a device held by the childcare worker (user), a cloud server, and a parent's device. The user uses the device to record the children's activities, which is then analyzed by the server, and an emotion engine is used to recognize the user's emotions. Based on this, reports and correspondence are automatically generated, and the user can then review and edit the final documents before sending them to the parents.

[2128] Program processing

[2129] Starting and recording an activity

[2130] 1. User Action:

[2131] The user turns on the device and opens the app.

[2132] Press the "Start Activity" button in the app to begin recording.

[2133] 2. Device behavior:

[2134] The device activates the camera and microphone to record the children's activities in real time using video and audio.

[2135] The recorded data is sent to a server at regular intervals.

[2136] Data analysis, emotion recognition and document generation

[2137] 3. Server Operation:

[2138] The server analyzes the received video data and uses image recognition technology to identify important scenes.

[2139] Similarly, voice data is analyzed and specific keywords and phrases are extracted using voice recognition technology.

[2140] The server then uses an emotion engine to recognize the user's emotions from the received video and audio data. For example, it analyzes the user's tone of voice and facial expressions to extract emotions such as joy, sadness, and surprise.

[2141] The server automatically generates reports and correspondence based on the analysis and emotion recognition results, incorporating phrases and tones that correspond to the user's emotions to better tailor the content of the documents.

[2142] Checking, correcting and sending documents

[2143] 4. User confirmation and correction:

[2144] The user checks the automatically generated document on the terminal.

[2145] If necessary, the contents of the document are modified to suit the user's wishes.

[2146] 5. Sending documents:

[2147] When the user presses the "send" button, the terminal sends the final verified document to the server.

[2148] The server sends the document to the parent via email or in-app message.

[2149] Specific examples

[2150] Example 1: Field trip record

[2151] Below is a specific example of when a user (a nursery teacher) takes children on a field trip.

[2152] 1. User Action:

[2153] The user turns on the device and opens the app to record the field trip.

[2154] Press "Start Activity" to begin recording.

[2155] 2. Device behavior:

[2156] The device uses a camera to record the children playing and simultaneously records audio using a microphone.

[2157] The recorded data is sent to a server at regular intervals.

[2158] 3. Server Operation:

[2159] The server analyzes the video data and extracts scenes such as "scenes of children playing on playground equipment" and "scenes of children gathering together."

[2160] The statements made by nursery teachers and children are converted into text from the audio data, and important phrases (e.g., "It's fun," "Be careful") are extracted.

[2161] At the same time, an emotion engine is used to identify the emotions of the caregivers and children.

[2162] Based on this information, a report on the field trip is automatically generated. For example, if the children appear to be having a good time, the document will be adjusted to include more upbeat phrases and photos of smiling children.

[2163] 4. User confirmation and correction:

[2164] The user checks the generated report on the terminal and makes corrections as necessary.

[2165] You can also add photos and additional comments to your report.

[2166] 5. Sending documents:

[2167] Once the report has been revised, click the "Send" button to send it to the parent.

[2168] The server will send the final report to the parent's email address or within the app.

[2169] The above is an embodiment of the invention of a system for reducing the workload of childcare workers and sharing information with parents quickly and accurately, combined with an emotion engine.

[2170] The processing flow will be explained below.

[2171] Step 1:

[2172] The user launches the app on their device and presses the "Start Activity" button.

[2173] Operation details: At the start of an excursion, the user turns on their smartphone or tablet, opens the app, and presses the "Start Activity" button.

[2174] Step 2:

[2175] The device will turn on the camera and microphone and begin recording.

[2176] What it does: The device activates its camera and microphone and begins recording video and audio of the child's activities in real time.

[2177] Step 3:

[2178] The device stores recorded video and audio data locally.

[2179] Operation details: The device captures video and audio data in real time and stores it in local storage.

[2180] Step 4:

[2181] The terminal transmits the recorded data to the server at regular intervals.

[2182] Operation details: The device uploads collected video and audio data to the server at regular intervals.

[2183] Step 5:

[2184] The server analyzes the received video data and extracts important scenes.

[2185] Operation details: The server uses a video analysis algorithm to automatically extract important scenes, such as playing on playground equipment or group photos.

[2186] Step 6:

[2187] The server analyzes the received audio data and extracts specific keywords and phrases.

[2188] How it works: The server uses speech recognition technology to convert the voice data into text and extracts key phrases (e.g., "fun," "next is lunchtime").

[2189] Step 7:

[2190] The server uses an emotion engine to recognize the user's emotions based on video and audio data.

[2191] How it works: The server uses an emotion engine to analyze facial expressions and vocal tone in the video to identify emotions such as joy, surprise, and calm.

[2192] Step 8:

[2193] The server automatically generates reports and correspondence based on the analysis results and emotion recognition results.

[2194] Operation details: The server automatically generates reports and communication messages that reflect the children's activities and the feelings of the caregivers. When children are having fun, it uses a lot of positive expressions to make the reports more appealing.

[2195] Step 9:

[2196] The server sends the generated document to the terminal.

[2197] Operation details: Automatically generated reports and correspondence are sent to the terminal so that the user can check and edit them.

[2198] Step 10:

[2199] The user checks the document generated on the terminal and corrects it if necessary.

[2200] How it works: The user reviews the report and correspondence, enters any necessary corrections or additions, and creates the final version.

[2201] Step 11:

[2202] The user confirms the document after making corrections and presses the "Send" button.

[2203] Operation details: The user confirms the final confirmed document and presses the "Send" button to complete the transmission.

[2204] Step 12:

[2205] The terminal transmits the final confirmed document to the server.

[2206] Operation details: The terminal sends the corrected document data to the server.

[2207] Step 13:

[2208] The server sends a final report or communication to the parent.

[2209] Operational Details: The server will send a final report or communication to the parent via email address or in-app messaging.

[2210] Step 14:

[2211] The parent receives and reviews the document.

[2212] How it works: Parents receive the report or communication sent to them, review the contents, and provide feedback if necessary.

[2213] The above is a specific processing flow of the present invention. By combining it with an emotion engine, it becomes possible to report information that reflects the user's emotions, further improving communication with parents.

[2214] Example 2

[2215] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[2216] In today's childcare environment, childcare workers are overwhelmed with their busy daily work, making it difficult for them to record children's activities in detail and report them to parents quickly and accurately. Furthermore, in order to communicate with parents, appropriate document generation that takes emotions into account is required, but doing this manually requires a great deal of time and effort. Therefore, a technology is needed that can accurately record children's activities and automatically generate reports that reflect their emotions while reducing the workload of childcare workers.

[2217] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes a data analysis means, an emotion recognition means, and a document generation means. This makes it possible to efficiently analyze the children's activity data and automatically generate appropriate reports and communication messages that take the user's emotions into consideration.

[2218] "Filming means" refers to devices and functions for recording children's activities as video in real time.

[2219] "Audio recording means" refers to a device or function for recording audio during an activity.

[2220] "Data analysis means" refers to devices or functions that analyze recorded image and audio data and extract important information.

[2221] The "document generation means" refers to a device or function for automatically generating reports and correspondence based on the extracted information and the user's feelings.

[2222] "Emotion recognition means" refers to a device or function for recognizing emotions from the user's tone of voice, facial expression, etc.

[2223] "Verification and correction means" refers to devices and functions that allow users to verify automatically generated documents and correct them as necessary.

[2224] "Transmission means" refers to a device or function for transmitting the final confirmed document to the guardian.

[2225] This invention combines an emotion engine with a system that reduces the workload of childcare workers and improves communication with parents. This system efficiently records children's activities, has the function of automatically generating reports and correspondence for parents, and is capable of generating documents that take the user's emotions into consideration.

[2226] System Overview

[2227] This system consists of a device held by the childcare worker (user), a cloud server, and a parent's device. The user uses the device to record the children's activities, which is then analyzed by the server, and an emotion engine is used to recognize the user's emotions. Based on this, reports and correspondence are automatically generated, and the user can then review and edit the final documents before sending them to the parents.

[2228] Hardware and software used

[2229] Device hardware:

[2230] Smartphone or tablet

[2231] Built-in camera and microphone

[2232] Server software:

[2233] Image recognition technology (e.g., OpenCV, TensorFlow)

[2234] Speech recognition technology (e.g., Google Cloud Speech-to-Text, Amazon Transcribe)

[2235] Emotion recognition engine (e.g. Microsoft Azure Emotion API, Affectiva)

[2236] Generative AI models (e.g., OpenAI GPT-3)

[2237] Program processing overview

[2238] Activity recording

[2239] User Action:

[2240] The user starts up the terminal and launches the dedicated application.

[2241] Press the "Start Activity" button in the app to begin recording mode.

[2242] Terminal behavior:

[2243] The device's camera and microphone will be activated and will record the children's activities in real time.

[2244] Video and audio data is sent to the server at regular intervals (for example, every 10 seconds).

[2245] Data analysis and emotion recognition

[2246] Server behavior:

[2247] The server analyzes the received video and audio data and extracts important information.

[2248] Image recognition technology is used to identify important scenes from video.

[2249] It uses voice recognition technology to extract specific keywords and phrases from speech.

[2250] It uses an emotion engine to recognize emotions from the user's tone of voice and facial expressions.

[2251] Document generation and transmission

[2252] Server behavior:

[2253] Based on the analysis results and emotion recognition results, the server automatically generates reports and correspondence using a generative AI model.

[2254] Tailor your document content with phrases and tones that reflect your emotions.

[2255] User Action:

[2256] The user can check the generated document on their own terminal and make corrections as necessary.

[2257] Click the "Send" button to send the revised document to the parent.

[2258] Examples:

[2259] For example, when a user takes their children on a field trip, the system operates as follows:

[2260] User Action:

[2261] The user starts the device and opens the application to record the field trip.

[2262] Press the "Start Activity" button to begin recording.

[2263] Terminal behavior:

[2264] The device's camera records the children's play and the microphone records their audio.

[2265] The recorded data is sent to the server at regular intervals.

[2266] Server behavior:

[2267] The server analyzes the video data and extracts scenes such as "scenes of children playing on playground equipment" and "scenes of children gathering together."

[2268] The system analyzes audio data and extracts important phrases from the speech of caregivers and children.

[2269] An emotion engine is used to identify the emotions of caregivers and children.

[2270] Based on the above information, a report on the field trip is automatically generated, describing the fun the students had in cheerful phrases.

[2271] User confirmation and correction:

[2272] The user reviews the generated report and makes corrections as necessary.

[2273] For example, add photos and additional comments to the report.

[2274] Sending documents:

[2275] Once the report has been revised, click the "Send" button to send it to the parent.

[2276] The server will send the final report to the parent's email address or within the app.

[2277] Prompt Sentence Examples

[2278] "Childcare workers record the children's field trips, and based on the records, automatically generate reports for parents. The generated documents also include emotional information from the childcare workers and the children. For example, fun scenes are described in cheerful language, while scenes requiring caution are described in polite language."

[2279] The above is an embodiment of the invention.

[2280] The flow of the identification process in the second embodiment will be described with reference to FIG.

[2281] Step 1: Starting the activity

[2282] User Action:

[2283] The user starts up the terminal and launches the dedicated application.

[2284] Press the "Start Activity" button in the app to begin recording mode.

[2285] Specific behavior:

[2286] A user taps an app on their smartphone or tablet to launch it.

[2287] Pressing the "Start Activity" button will take you to the activity recording screen.

[2288] Step 2: Record and send data

[2289] Terminal behavior:

[2290] The device activates a camera and microphone to record the children's activities in real time.

[2291] Video and audio data is sent to the server at regular intervals (for example, every 10 seconds).

[2292] Input and Output:

[2293] (Input) Photographed video and recorded audio data

[2294] (Output) Split video and audio files sent to the server

[2295] Specific behavior:

[2296] Images captured by the built-in camera and audio recorded by the microphone are saved in a data buffer in real time.

[2297] The divided data is uploaded to the server sequentially (e.g., a data file is sent every 10 seconds).

[2298] Step 3: Data analysis

[2299] Server behavior:

[2300] The server analyzes the received video and audio data and extracts important information.

[2301] Image recognition technology is used to identify important scenes from video.

[2302] Using voice recognition technology, the voice data is converted into text and important keywords and phrases are extracted.

[2303] Input and Output:

[2304] (Input) Received video and audio data

[2305] (Output) Analyzed important scene information and textual keywords

[2306] Specific behavior:

[2307] Image recognition technology (e.g., OpenCV, TensorFlow) is used to analyze specific frames in the video and extract "smiling scenes" and "group photo scenes."

[2308] Voice recognition technology (e.g., Google Cloud Speech-to-Text) is used to convert the voice data into text, and important phrases such as "It's fun" and "Be careful" are extracted from the comments of the caregivers and children.

[2309] Step 4: Emotion Recognition

[2310] Server behavior:

[2311] The server uses an emotion recognition engine to recognize emotions from the user's tone of voice and facial expressions.

[2312] Input and Output:

[2313] (Input) Analyzed video and audio data

[2314] (Output) Recognized user emotion data

[2315] Specific behavior:

[2316] An emotion recognition engine (e.g., Microsoft Azure Emotion API) is used to identify emotions (e.g., joy, sadness, surprise) from facial expressions and tone of voice in the video.

[2317] The recognized emotion data is stored in a database as emotion tags.

[2318] Step 5: Document Generation

[2319] Server behavior:

[2320] Based on the analysis results and emotion recognition results, the server automatically generates reports and correspondence using a generative AI model.

[2321] Tailor your document content with phrases and tones that reflect your emotions.

[2322] Input and Output:

[2323] (Input) Important scene information, text keywords, emotion data

[2324] (Output) Automatically generated reports and correspondence

[2325] Specific behavior:

[2326] Using a generative AI model (e.g., OpenAI GPT-3), documents are automatically generated based on analysis and emotion recognition results.

[2327] The document will contain information such as "We had a great time on our field trip today," and depending on the emotion tag, it will add details such as "We saw lots of smiling children."

[2328] Step 6: Check and correct the document

[2329] User Action:

[2330] The user checks the automatically generated document on the terminal and corrects it if necessary.

[2331] Click the "Save" button to confirm the changes.

[2332] Input and Output:

[2333] (Input) Automatically generated report

[2334] (Output) Revised Final Report

[2335] Specific behavior:

[2336] Tap the generated document on the app screen to enter edit mode and manually insert additional comments or photos.

[2337] Save the amendments and prepare them as a final confirmation document.

[2338] Step 7: Sending documents

[2339] User Action:

[2340] When the user presses the "send" button, the terminal sends the final verified document to the server.

[2341] The server sends the final document to the parent.

[2342] Input and Output:

[2343] (Input) Final confirmed report

[2344] (Output) Report sent to parents

[2345] Specific behavior:

[2346] When the user taps the "Send" button, a confirmation message will be displayed.

[2347] Once you agree to the confirmation message, the document is instantly sent to the server and delivered to the parent.

[2348] (Application example 2)

[2349] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[2350] Conventional virtual store customer support systems lacked the ability to recognize user emotions in real time, making it difficult to improve the user experience and provide smooth support. They also lacked the ability to immediately detect user confusion or difficulties and provide appropriate real-time support accordingly. This resulted in a poor user experience and inefficient customer support. Furthermore, there was a need for improved accuracy in automatically generated reports based on user behavior.

[2351] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[2352] In this invention, the server includes a means for analyzing a user's behavior and emotions in real time, a means for recognizing emotions in real time, and a means for providing feedback based on the recognition results. This makes it possible to provide appropriate support and product suggestions in real time according to the user's emotions. In addition, the efficiency of customer support can be improved by creating highly accurate automatically generated reports based on the user's experience records.

[2353] "Image capture means" refers to a video capture device and its functions for recording children's activities.

[2354] "Audio recording means" refers to a device and its function for recording audio during an activity.

[2355] "Data analysis means" refers to the function of analyzing recorded image and audio data to extract important information.

[2356] "Document generation means" refers to a function that automatically generates a report or communication based on the extracted information.

[2357] "Emotion recognition means" refers to a function that uses an emotion engine to analyze a user's emotions in real time.

[2358] "Feedback means" refers to a function that suggests appropriate actions to the user based on real-time emotion recognition results.

[2359] "Means for checking and correcting" refers to a function that allows users to check and correct automatically generated documents.

[2360] "Transmission means" refers to the function of sending the final confirmed document to the guardian.

[2361] The system for implementing this invention consists of a user's device, a cloud server, and a parent's device. The user uses the device to record their child's activities, and the data is analyzed on the server. Furthermore, an emotion engine is used to recognize the user's emotions in real time and automatically generate reports and correspondence.

[2362] Hardware and software used

[2363] Hardware:

[2364] Terminal: A device such as a smartphone, smart glasses, or head-mounted display.

[2365] Camera: The camera built into the device.

[2366] Microphone: The microphone built into the device.

[2367] software:

[2368] OpenCV: A library for image processing and face detection.

[2369] DeepFace: A library for determining emotions from images.

[2370] Hugging Face's transformers: A library for speech recognition and tone analysis.

[2371] Data processing and calculation

[2372] 1. Image and audio recording:

[2373] The user turns on the device and records their children's activities in real time.

[2374] The device's camera captures video and microphone records audio, and these data are sent to a cloud server at regular intervals.

[2375] 2. Data Analysis and Emotion Recognition:

[2376] The server uses OpenCV to analyze the video data and perform face detection.

[2377] The facial image is analyzed using the DeepFace library to determine emotions (e.g., joy, sadness, surprise, etc.).

[2378] The voice data is recognized using Hugging Face's transformers to extract the user's tone of voice and specific keywords (e.g., "fun," "troubled," etc.).

[2379] 3. Real-time feedback:

[2380] Based on the results of emotion recognition, the server will provide appropriate feedback to the user, for example, if a confused expression is detected, it will immediately display a support message.

[2381] 4. Document Generation:

[2382] The server automatically generates reports and correspondence based on the analyzed data and emotion recognition results.

[2383] The user can review the generated document on the terminal and make corrections as necessary.

[2384] 5. Sending documents:

[2385] The final, verified document is sent from the server to the parent via email or in-app message.

[2386] Specific examples

[2387] Use case 1: Recording a field trip

[2388] When a user takes their children on a field trip, they use the system in the following steps:

[2389] 1. The user starts up the device and presses the "Start Activity" button to record the field trip.

[2390] 2. The device uses a camera to record the children's play and a microphone to simultaneously record their voices. This data is then sent to a server at regular intervals.

[2391] 3. The server performs video and audio analysis, and extracts important scenes and keywords using facial expression and voice recognition.

[2392] 4. The server automatically generates a report on the field trip based on this information. For example, if it recognizes that the field trip is going well, it will write a report in a cheerful tone and include many photos of the children smiling.

[2393] 5. The user reviews the generated report and makes any necessary corrections.

[2394] 6. Click the "Send" button to send the finalized report to the parent.

[2395] Example prompt sentence:

[2396] Generate appropriate support messages based on the user's emotional state:

[2397] "Emotion: Confused" "User says: 'I don't know which product to buy.'"

[2398] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[2399] Step 1:

[2400] The user starts the device, opens the application and presses the "Start Activity" button.

[2401] Input: Launching a terminal and operating applications.

[2402] What it does: The device activates the camera and microphone, ready to record your child's activities in real time.

[2403] Output: Camera and microphone are activated.

[2404] Step 2:

[2405] The device records video and audio in real time and transmits the data to a server in the cloud at regular intervals.

[2406] Input: Real-time data from camera and microphone.

[2407] How it works: The device's camera captures video and microphone records audio. The data is packetized at regular intervals and sent to a server.

[2408] Output: Transmitted video and audio data.

[2409] Step 3:

[2410] The server analyzes the received video and audio data.

[2411] Input: Video and audio data sent from the device.

[2412] How it works: The server uses OpenCV to detect faces from video data and DeepFace to analyze emotions, and Hugging Face's transformers to recognize audio data and extract tones and specific keywords.

[2413] Output: Analysis results (e.g. emotion labels, speech recognition results).

[2414] Step 4:

[2415] The server automatically generates a report or communication based on the analysis results.

[2416] Input: Emotion recognition results from video data, speech recognition results from audio data.

[2417] How it works: The server combines the analysis results and uses natural language generation technology to create reports and correspondence. It selects appropriate expressions based on the user's emotions and composes the text.

[2418] Output: Auto-generated report or correspondence.

[2419] Step 5:

[2420] The user reviews the automatically generated report or correspondence and corrects it if necessary.

[2421] Input: An automatically generated report or correspondence.

[2422] How it works: The user views the generated document on their device, reviews the content, and then edits the text and adds images and comments if necessary.

[2423] Output: The final verified and corrected document.

[2424] Step 6:

[2425] The final confirmed document is sent from the server to the parent.

[2426] Input: Reviewed and amended final document.

[2427] How it works: When the user presses the "Send" button, the device sends the final document to the server, which then sends it to the parent's email address or as an in-app message.

[2428] Output: Document sent to parents.

[2429] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[2430] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[2431] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[2432] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[2433] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[2434] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[2435] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[2436] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[2437] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[2438] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[2439] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[2440] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[2441] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[2442] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[2443] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[2444] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[2445] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[2446] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[2447] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[2448] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[2449] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[2450] The following is further disclosed regarding the above embodiment.

[2451] (Claim 1)

[2452] A means of capturing images to record children's activities;

[2453] a voice recording means for recording voices during the activity;

[2454] data analysis means for analyzing the recorded image and audio data to extract important information;

[2455] a document generation means for automatically generating a report or a communication based on the extracted information;

[2456] A means for allowing a user to check and modify the document;

[2457] a transmitting means for transmitting the final confirmed document to the guardian;

[2458] A system including:

[2459] (Claim 2)

[2460] 2. The system according to claim 1, wherein the image capturing means records video in real time and transmits the recorded data to a server.

[2461] (Claim 3)

[2462] 2. The system according to claim 1, wherein the voice recording means records voice in real time and transmits the recorded data to a server.

[2463] "Example 1"

[2464] (Claim 1)

[2465] an image capturing means for recording the activity;

[2466] a voice recording means for recording voices during the activity;

[2467] a transmitting means for transmitting the recorded image and audio data to a cloud server at regular intervals;

[2468] data analysis means for analyzing the received data and extracting specific scenes;

[2469] a document generation means for automatically generating a report or a communication based on the analysis result using a generation AI model;

[2470] A means for allowing a user to check and modify the automatically generated document;

[2471] a transmitting means for transmitting the final confirmed document to the guardian;

[2472] A system including:

[2473] (Claim 2)

[2474] 2. The system according to claim 1, wherein the image capturing means records video in real time, and the recorded data is transmitted to a cloud server using a transmission means.

[2475] (Claim 3)

[2476] 2. The system according to claim 1, wherein the audio recording means records audio in real time and transmits the recorded data to a cloud server using a transmission means.

[2477] "Application Example 1"

[2478] (Claim 1)

[2479] Image capture means for recording customer activity;

[2480] a voice recording means for recording voices during the activity;

[2481] data analysis means for analyzing the recorded image and audio data to extract important information;

[2482] a document generation means for automatically generating a report or a communication based on the extracted information;

[2483] A means for allowing a user to check and modify the document;

[2484] a transmitting means for transmitting the finalized document to the parties concerned;

[2485] A system including:

[2486] (Claim 2)

[2487] 2. The system according to claim 1, wherein the image capturing means records video in real time and transmits the recorded data to a server.

[2488] (Claim 3)

[2489] 2. The system according to claim 1, wherein the voice recording means records voice in real time and transmits the recorded data to a server.

[2490] "Example 2: Combining Emotion Engines"

[2491] (Claim 1)

[2492] A means of recording children's activities,

[2493] a voice recording means for recording voices during the activity;

[2494] data analysis means for analyzing the recorded image and audio data to extract important information;

[2495] a document generation means for automatically generating a report or a message based on the extracted information and the user's feelings;

[2496] emotion recognition means for recognizing the emotion of the user;

[2497] A means for allowing a user to check and modify the document;

[2498] a transmitting means for transmitting the final confirmed document to the guardian;

[2499] A system including:

[2500] (Claim 2)

[2501] 2. The system according to claim 1, wherein the image capturing means records video in real time and transmits the recorded data to a server.

[2502] (Claim 3)

[2503] 2. The system according to claim 1, wherein the voice recording means records voice in real time and transmits the recorded data to a server.

[2504] "Application example 2 when combining emotion engines"

[2505] Claiming a new invention

[2506] (Claim 1)

[2507] A means of capturing images to record children's activities;

[2508] a voice recording means for recording voices during the activity;

[2509] a data analysis means for analyzing the recorded image and audio data to extract important information;

[2510] a document generation means for automatically generating a report or a communication based on the extracted information;

[2511] emotion recognition means for analyzing a user's emotions in real time using an emotion engine;

[2512] a feedback means for suggesting appropriate actions to the user based on the real-time emotion recognition results;

[2513] A means for allowing a user to check and modify the document;

[2514] a transmitting means for transmitting the final confirmed document to the guardian;

[2515] A system including:

[2516] (Claim 2)

[2517] 2. The system according to claim 1, wherein the image capturing means records video in real time and transmits the recorded data to a server.

[2518] (Claim 3)

[2519] 2. The system according to claim 1, wherein the voice recording means records voice in real time and transmits the recorded data to a server. [Explanation of symbols]

[2520] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. A means of capturing images to record children's activities; a voice recording means for recording voices during the activity; a data analysis means for analyzing the recorded image and audio data to extract important information; a document generation means for automatically generating a report or a communication based on the extracted information; a confirmation / modification means for allowing a user to confirm and modify the document; a transmitting means for transmitting the final confirmed document to the guardian; A system including:

2. 2. The system according to claim 1, wherein said image capturing means records images in real time and transmits the recorded data to a server.

3. 2. The system according to claim 1, wherein said voice recording means records voice in real time and transmits the recorded data to a server.

Images