system

Abstract

We provide the system. [Solution] A means of detecting and recording children's voices, A means of adding a timestamp to recorded audio and saving it, A means of analyzing saved audio and extracting important statements, A means of sharing analysis results via communication, A system that includes this.

Description

Technical Field

[0001] The technology of the present disclosure relates to a system.

Background Art

[0002] Patent Document 1 discloses a method for controlling a persona chatbot, which is performed by at least one processor, and includes steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to an explanation of a chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In modern child-rearing generations, there are many dual-income families, making it difficult to record and share the moments of a child's growth in real time. Also, family members living far away, especially grandparents, have limited opportunities to directly experience that precious growth process. Therefore, there is a need for a technology that can immediately record a child's speech and actions and efficiently share them online.

Means for Solving the Problems

[0005] The present invention solves the above problems by providing a system that includes means for detecting and recording a child's voice, means for adding a timestamp to the recorded voice, means for analyzing the stored voice to extract important statements, and means for sharing them via communication. Furthermore, by providing means for recognizing and photographing an object held by a child, means for adding a timestamp to the photographed image and saving it, and means for providing it to a specific user via a communication platform, the present invention provides a system that can record a child's growth process more easily and efficiently and facilitate communication among family members.

[0006] "Voice detection means" refers to a device or system that has the function of identifying a child's voice and starting to record it.

[0007] "Timestamping" is the process of adding the date and time of recording to data.

[0008] "Speech analysis means" refers to a device or system that processes recorded speech data and has the function of identifying important content or patterns.

[0009] "Means of sharing via communication" refers to a device or system that has the function of transmitting recorded and analyzed data to other terminals or users via a network.

[0010] "Object recognition means" refers to a device or system that has the function of identifying and recognizing a specific object from visual data.

[0011] "Image storage means" refers to a device or system that has the function of saving captured image data to storage.

[0012] A "communication platform" is a network environment or system for communicating and sharing digital data among users.

[0013] "Means of providing information to target users" refers to a device or system that has the function of appropriately delivering information to specific recipients based on pre-specified conditions. [Brief explanation of the drawing]

[0014] [Figure 1] This is a conceptual diagram showing an example of the configuration of a data processing system according to the first embodiment. [Figure 2] This is a conceptual diagram showing an example of the essential functions of a data processing device and a smart device according to the first embodiment. [Figure 3] This is a conceptual diagram showing an example of the configuration of a data processing system according to the second embodiment. [Figure 4] This is a conceptual diagram showing an example of the main functions of a data processing device and smart glasses according to the second embodiment. [Figure 5] This is a conceptual diagram showing an example of the configuration of a data processing system according to the third embodiment. [Figure 6] This is a conceptual diagram showing an example of the main functions of a data processing device and a headset-type terminal according to the third embodiment. [Figure 7] This is a conceptual diagram showing an example of the configuration of a data processing system according to the fourth embodiment. [Figure 8] This is a conceptual diagram showing an example of the main functions of a data processing device and a robot according to the fourth embodiment. [Figure 9] This shows an emotion map where multiple emotions are mapped. [Figure 10] This shows an emotion map where multiple emotions are mapped. [Figure 11] This is a sequence diagram showing the processing flow of the data processing system in Example 1. [Figure 12] This is a sequence diagram showing the processing flow of the data processing system in Application Example 1. [Figure 13] This is a sequence diagram showing the processing flow of the data processing system in Example 2, which incorporates an emotion engine. [Figure 14]It is a sequence diagram showing the processing flow of a data processing system in Application Example 2 when a sentiment engine is combined.

Embodiments for Carrying Out the Invention

[0015] Hereinafter, an example of an embodiment of a system according to the technology of the present disclosure will be described with reference to the accompanying drawings.

[0016] First, the terms used in the following description will be explained.

[0017] In the following embodiments, a numbered processor (hereinafter simply referred to as "processor") may be a single arithmetic unit or a combination of multiple arithmetic units. Also, the processor may be a single type of arithmetic unit or a combination of multiple types of arithmetic units. Examples of arithmetic units include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), an APU (Accelerated Processing Unit), and the like.

[0018] In the following embodiments, a numbered RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a work memory by the processor.

[0019] In the following embodiments, a numbered storage is one or more non-volatile storage devices that store various programs and various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), or magnetic tapes, etc.

[0020] In the following embodiments, the signed communication interface (I / F) is an interface that includes a communication processor and an antenna, etc. The communication interface manages communication between multiple computers. Examples of communication standards applicable to the communication interface include wireless communication standards such as 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), or Bluetooth (registered trademark).

[0021] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." That is, "A and / or B" means that it may be A alone, or B alone, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" applies when expressing three or more things linked by "and / or."

[0022] [First Embodiment]

[0023] Figure 1 shows an example of the configuration of the data processing system 10 according to the first embodiment.

[0024] As shown in Figure 1, the data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0025] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0026] The smart device 14 comprises a computer 36, a reception device 38, an output device 40, a camera 42, and a communication interface 44. The computer 36 comprises a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The reception device 38, output device 40, and camera 42 are also connected to the bus 52.

[0027] The reception device 38 is equipped with a touch panel 38A and a microphone 38B, etc., and receives user input. The touch panel 38A receives user input by detecting contact with an object (e.g., a pen or finger). The microphone 38B receives user input by detecting the user's voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0028] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form perceptible to the user 20 (e.g., audio and / or text). The display 40A displays visible information such as text and images according to instructions from the processor 46. The speaker 40B outputs audio according to instructions from the processor 46. The camera 42 is a small digital camera equipped with an optical system such as a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0029] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various types of information between processor 46 and processor 28 via network 54.

[0030] Figure 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0031] As shown in Figure 2, in the data processing device 12, a specific processing is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" related to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0032] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0033] In the smart device 14, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The reception output program 60 is used in conjunction with a specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0034] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0035] This invention provides a method for recording a child's growth process in audio and images and sharing it with distant family members via a network. This allows users, who are parents raising children, to capture precious moments and save memories, even amidst their busy daily lives, and share them with their families. The following describes how this invention is implemented.

[0036] First, let's discuss voice recording. The device is equipped with a voice detection mechanism to detect children's voices. When the user's child speaks, the device collects the sound through the microphone and starts recording. The recorded voice data is automatically time-stamped and temporarily stored in storage. This makes it easy to track what was said and when.

[0037] Next, the recorded audio data is periodically transmitted to a server via the network. The server processes the received audio using speech analysis tools, extracting and organizing important phrases and words. It is then transcribed as needed, and the analysis results are recorded in a database. This information is stored in a way that allows family members to easily search and review it later.

[0038] Regarding photography, the device's built-in camera is equipped with object recognition capabilities to recognize objects held by the child. As soon as an object is detected, the camera activates, capturing the child's movements and the object they are holding. The captured images are time-stamped and saved to storage.

[0039] The saved image data is also transmitted to a server via the network and stored in a database. The server then uses a communication platform to automatically share this data with specific users based on pre-configured conditions. For example, this allows for instant image sharing to a grandparent's account.

[0040] This system automates the recording of a child's growth without requiring specific manual operations from the user, ensuring that important moments are not missed even amidst the busyness of daily life. Furthermore, these records are always easy to share with family, fostering communication among users.

[0041] The following describes the processing flow.

[0042] Step 1:

[0043] The device constantly monitors ambient sounds using voice detection and detects children's voices. When a child's voice is detected, it starts recording using the built-in microphone.

[0044] Step 2:

[0045] The device adds a timestamp to the recorded audio data and temporarily saves it to storage.

[0046] Step 3:

[0047] The device sends the stored audio data to the server in conjunction with the timing of its network connection.

[0048] Step 4:

[0049] The server receives the transmitted audio data and processes it using audio analysis tools. It extracts important statements and phrases and transcribes them if necessary.

[0050] Step 5:

[0051] The server stores the analysis results in a database and manages that data so that users can view it.

[0052] Step 6:

[0053] The device uses its camera to detect an object the child is holding using object recognition technology and captures a photograph at that moment.

[0054] Step 7:

[0055] The device adds a timestamp to the image it takes and temporarily saves it to storage.

[0056] Step 8:

[0057] The device sends stored image data to the server via the network.

[0058] Step 9:

[0059] The server stores image data in a database and shares the data with specific users through a communication platform set by the user.

[0060] Step 10:

[0061] Users receive notifications, review data shared on social media and platforms, and provide feedback as needed.

[0062] (Example 1)

[0063] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0064] In modern times, families raising children need a quick and efficient way to record and share their children's growth moments. However, manual recording and photography, as well as managing and sharing that data, is time-consuming and burdensome for busy parents. This leads to challenges such as failing to record precious moments and difficulty in timely information sharing with family.

[0065] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0066] In this invention, the server includes an acoustic acquisition means for detecting and collecting a child's voice, a storage means for temporarily storing the collected voice with time information attached, and a data analysis means for analyzing the stored voice and selecting specific phrases or words. This makes it possible to effectively capture important moments in a child's development and quickly and automatically share that information with the family.

[0067] "Sound acquisition means" refers to a device equipped with the function of automatically detecting ambient sounds and collecting sound information.

[0068] "Time information" refers to data that indicates the specific date and time when audio or images were acquired, enabling later tracking and organization.

[0069] "Storage means" refers to technologies for temporarily or permanently storing collected audio and image data in a storage device.

[0070] "Data analysis means" refers to a program or device that has the function of processing stored audio or image data and extracting specific information.

[0071] "Information sharing means" refers to a system that uses communication technology to transfer analyzed information to other devices or users.

[0072] "Visual acquisition means" refers to a device that uses cameras or sensors to acquire image information of the environment and record a specific object.

[0073] "Information distribution means" refers to technology that transmits stored or analyzed data to other users or systems via a network.

[0074] A "selective transmission method" refers to a system equipped with the function of selecting and transmitting only appropriate data based on set criteria and conditions.

[0075] This invention provides an automated system for efficiently recording a child's growth process and easily sharing it among family members. This system combines voice acquisition means and image acquisition means, and has the function of efficiently acquiring, analyzing, storing, and sharing voice and images.

[0076] Audio acquisition and processing

[0077] The device is equipped with a microphone to automatically detect children's voices. This voice data is tagged with time information and temporarily stored in local storage. The collected voice data is transmitted to a server via the network. The server uses advanced voice analysis software to analyze the data and extract important phrases and words. The analysis results are transcribed into text as needed and stored in a database for later access.

[0078] Image acquisition and processing

[0079] The device has a built-in camera for image acquisition and is equipped with a function to recognize objects held by children. When an object is detected, the camera automatically activates and takes a picture. This image data is also given time information and temporarily stored in local storage. The server receives the image data and stores it in a database. Subsequently, the image data is automatically shared with family members based on predetermined conditions.

[0080] Specific example

[0081] For example, if a user's three-year-old child says "Mama" for the first time, the system instantly records and analyzes the audio. The server recognizes "Mama" as an important phrase and converts it into text data. This data is immediately sent to the grandparents' account, allowing the family to share and enjoy that moment.

[0082] Example of a prompt

[0083] For example, the following can be entered as a prompt for the generative AI model:

[0084] "Convert your child's first words into audio format, extract important words, and save them to share with your family."

[0085] This system allows users to record important moments amidst their busy daily lives and communicate smoothly with their families.

[0086] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0087] Step 1:

[0088] The device detects the child's voice. To initiate voice detection, the device constantly monitors the surrounding sounds. When the child's voice exceeds a certain threshold, the device automatically starts recording. The input here is the voice data acquired through the microphone, and the output is the data stream of the recording that has started.

[0089] Step 2:

[0090] The device adds time information to the recorded audio data. A timestamp is added to the audio data as soon as recording begins. This allows the device to identify the date and time the recording was made. The input is the audio data being recorded, and the output is the audio data with the time information added.

[0091] Step 3:

[0092] The device temporarily stores the audio data, along with the assigned time information, in local storage. The data is stored in an efficiently accessible format. The input is audio data with time information, and the output is the temporarily stored audio file.

[0093] Step 4:

[0094] The terminal sends stored audio data to the server at regular intervals. Here, the terminal securely transfers the data over the network. The input is the audio data stored in local storage, and the output is the audio data received by the server.

[0095] Step 5:

[0096] The server analyzes the received audio data. Using speech analysis software, the server analyzes the data and extracts important phrases and words. A generative AI model is used in this process. The input is the raw audio data received by the server, and the output is the analyzed phrases and words.

[0097] Step 6:

[0098] The server stores the analysis results, including extracted phrases and words, in a database. The analysis results may be converted to text format, making them easier for family members to access later. The input is the analyzed phrases and words, and the output is the text data recorded in the database.

[0099] Step 7:

[0100] The server shares the saved analysis results with specific users according to predetermined conditions. It transmits data to pre-configured family members via a communication platform. The input is the saved analysis results, and the output is the notification received by the user.

[0101] Step 8:

[0102] The device recognizes objects held by a child and detects their movement. Using a camera and sensors, the device captures an image once it confirms the child's movement. The input is the surrounding visual information captured by the camera and sensors, and the output is the captured image data.

[0103] Step 9:

[0104] The device adds time information to the captured image data. The time at the moment the image was taken is recorded. The input is the captured image, and the output is the image data with time information added.

[0105] Step 10:

[0106] The terminal sends image data with assigned time information to the server. The data is transmitted appropriately over the network. The input is image data containing time information, and the output is image data received by the server.

[0107] Step 11:

[0108] The server stores the received image data in a database. The image data is then shared among family members based on certain criteria. The input is the received image data, and the output is the record in the database and the sharing notification to the family.

[0109] (Application Example 1)

[0110] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0111] Recording and sharing a child's development is an important form of communication within the family, but the need for manual operation makes it a burden, especially for busy parents. There is also a need to record every moment of growth to create richer memories. Furthermore, there is a desire to gain a more comprehensive understanding of a child's development by utilizing the results of voice and motion analysis.

[0112] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0113] In this invention, the server includes means for analyzing voice and notifying when a specific event has occurred, means for detecting surrounding movements and triggering specific actions, and means for inputting analysis results based on voice and actions into an AI model. This makes it possible to automatically record moments of a child's growth even amidst the busyness of daily life, and to appropriately share those records with family.

[0114] "Children's voices" refers to speech or voices of minors detected through recording devices.

[0115] "Means of recording" refers to hardware or software for saving audio as recorded data.

[0116] "Means for adding and saving timestamps" refers to a system or method for adding date and time information to recorded data and saving it.

[0117] "Means for analyzing and extracting important statements" refers to technology that identifies and extracts important information from recorded audio data based on specific criteria.

[0118] "Means of sharing via communication" refers to methods and technologies for sharing data with other terminals or systems via a network.

[0119] "A means of analyzing audio and notifying users that a specific event has occurred" refers to a system that analyzes audio data and notifies the user when a specific event is detected.

[0120] "Means of detecting surrounding movements and triggering specific actions" refers to technologies that detect the movement of the environment or objects using sensors, and automatically initiate specific actions in response.

[0121] "Means of inputting into a generative AI model" refers to an interface used to input analysis results and data into an artificial intelligence model, allowing that model to perform further processing and predictions.

[0122] The system implementing this invention is based on a terminal installed in the home. The terminal is equipped with an audio recording device and an image capture device, which are used to record the child's voice and actions in real time. A timestamp is automatically added to the audio and images, and the data is temporarily stored in the terminal's storage device.

[0123] The server receives audio data transmitted from the terminal and performs analysis using speech recognition software. This analysis extracts important statements and events based on specific criteria and notifies the user as needed. Furthermore, it applies motion recognition algorithms to image data, detecting the child's movements and behavior, and inputting the results into a generating AI model. Software such as Vosk and OpenCV can be used for this process.

[0124] Notifications and analysis results from the server are shared with specific users via the network according to pre-configured conditions. This operation utilizes cloud services and communication platforms, enabling seamless data sharing among family members.

[0125] For example, if the device records a child's first words or a drawing, these recordings are immediately analyzed and processed by the server, and the family is notified. An example of a prompt for the generating AI model might be: "Describe how the robot behaves in the home. In particular, focus on how it captures the child's voice and movements and how it sends and shares that information to the server."

[0126] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0127] Step 1:

[0128] The device uses an audio recording device to detect and record children's speech in real time within the environment. It receives an audio signal as input, adds a timestamp to the audio data, and temporarily stores it in a memory device. It generates processed audio data as output.

[0129] Step 2:

[0130] The device uses an image capture device to detect the child's movements. It takes a video feed as input and applies a motion detection algorithm. If movement is detected, an image is captured and saved along with a timestamp. The output is image data captured triggered by movement.

[0131] Step 3:

[0132] The server receives audio data transmitted from the terminal. Using the processed audio data as input, speech recognition software performs analysis to extract specific important statements. In this process, text data is generated from the audio signal and output as the analysis result.

[0133] Step 4:

[0134] The server receives image data from the terminal and applies a motion recognition algorithm. The input is the captured image data, which analyzes the child's movements and generates information about those movements, which is then input into an AI model. This results in the analysis results being output.

[0135] Step 5:

[0136] The server checks pre-configured notification conditions based on analysis results obtained from audio and image data. The input consists of analysis results and condition settings. If a notification is deemed necessary, it sends a notification to the specified user. The output is a notification message.

[0137] Step 6:

[0138] Users receive notifications from the server and review data as needed. They can view or see audio and image data on the platform and access additional information based on the generated AI model. The output includes visual and audio feedback based on user selection.

[0139] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0140] This invention provides a method for recording a child's developmental stages using audio and images, and by combining this with an emotion engine, simultaneously evaluating the child's emotional state and sharing this information with family members who live far away. This allows parents to record and share their child's daily activities along with their emotions, thereby deepening communication within the family and among relatives. The following describes how this invention is implemented.

[0141] First, let's discuss voice recording. The device is equipped with voice detection capabilities to detect children's voices. When the user's child speaks, the device collects the sound through the microphone and begins recording. The recorded voice data is time-stamped, and then the emotion engine is used to analyze the emotion from the voice. This emotion data is temporarily stored in storage along with the voice data.

[0142] Next, the saved audio and emotional data are transmitted to a server via the network. The server processes the received audio using audio analysis tools, extracting important phrases and issues, and integrating and storing emotional data from an emotional engine. The analysis results are recorded in a database and are structured to make it easy to understand the emotional state of the child.

[0143] Next, let's discuss photography. The camera built into the device is equipped with object recognition capabilities to recognize objects held by the child. As soon as an object is recognized, the camera activates and photographs the child's movements and the object they are holding. The captured images are time-stamped, and an emotion engine is used to analyze the child's emotions from their facial expressions. This emotion data is stored in storage along with the image data.

[0144] Image data and emotional data are transmitted to a server via the network. The server stores this data in a database and uses a communication platform to automatically share it with specific users based on pre-configured conditions. This allows grandparents and other family members to experience the child's growth and emotional changes in real time.

[0145] This system automates the recording of a child's growth and emotions without requiring specific manual operations from the user, ensuring that important moments are never missed. These recordings can always be shared with the family, helping to foster intimate communication.

[0146] The following describes the processing flow.

[0147] Step 1:

[0148] The device constantly monitors ambient sounds using voice detection and detects children's voices. When a child's voice is detected, it starts recording using the built-in microphone.

[0149] Step 2:

[0150] The device adds a timestamp to the recorded audio data and simultaneously uses an emotion engine to analyze the emotions in the audio data. The resulting emotional information is integrated into the audio data and temporarily stored in storage.

[0151] Step 3:

[0152] The device sends stored voice data and emotional information to the server in conjunction with the timing of its network connection.

[0153] Step 4:

[0154] The server receives the transmitted audio data and emotional information, extracts important statements and phrases using an audio analysis tool, associates them with emotional information, and stores them in a database.

[0155] Step 5:

[0156] The device's camera uses object recognition technology to detect the object the child is holding. When an object is detected, the camera activates and takes an image of that moment.

[0157] Step 6:

[0158] The device adds a timestamp to the image it captures, and then uses an emotion engine to analyze the facial expressions in the image and determine the emotion. The obtained emotion information is integrated into the image data and saved to storage.

[0159] Step 7:

[0160] The device transmits stored image data and emotional information to the server via the network.

[0161] Step 8:

[0162] The server stores image data and emotional information in a database and automatically provides the data to specific users via a communication platform.

[0163] Step 9:

[0164] Users receive notifications, review data shared on social media and other platforms, and gain real-time insights into their child's emotional state and development.

[0165] (Example 2)

[0166] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0167] This invention aims to solve the problem of difficulty for families living in remote locations to share their children's daily growth and emotional changes in real time and to engage in deep communication. Furthermore, conventional technologies lacked a mechanism to effectively combine voice and image recording with emotion analysis, so the challenge is to provide a means to efficiently achieve this.

[0168] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0169] In this invention, the server includes means for analyzing and evaluating voice using an emotion processing device, a device for analyzing images and evaluating emotional states, and a device for transmitting the analysis results to a remote recipient using communication technology. This makes it possible to automatically and instantly share a child's growth and emotional changes with family members in remote locations using voice and image data.

[0170] A "sound detection and recording device" is a device equipped with the function of detecting sounds emitted from children or the environment using a microphone and recording them in digital format.

[0171] A "device for adding and saving date and time information" is a device that has the function of adding a timestamp to recorded or photographed data and recording it as digital data along with that information.

[0172] A "device for extracting important information" is a device equipped with the function of selecting necessary sentences and words from recorded audio data based on predetermined criteria.

[0173] A "device that evaluates emotional states using an emotion processing device" is a device that analyzes voice and image data and uses an emotion engine to identify and evaluate an individual's emotional state.

[0174] A "device that identifies and photographs objects" is a device that uses a camera and artificial intelligence to automatically take a photograph when it recognizes a specific object.

[0175] A "device for sharing with remote locations via communication technology" is a device that has the function of transmitting and sharing recorded or photographed data to other locations or devices using the internet or wireless communication networks.

[0176] A "device that provides information to specific recipients via an information transmission platform" is a device that uses dedicated software or network services to provide information to specific individuals or groups according to pre-configured conditions.

[0177] This invention is a system that uses a terminal and a server to acquire and analyze a child's voice and images, evaluate their emotional state, and share it with family members in a remote location. The detailed procedure for implementing this system is shown below.

[0178] First, the device is equipped with a microphone and camera, which are used to acquire audio and images. Voice detection software on the device detects the child's vocalizations in real time, starts recording the audio, and generates audio data. Next, the recorded audio data is analyzed by software using an emotion engine to evaluate the child's emotional state. This emotion analysis technology uses algorithms that analyze the tone, pitch, and speed of the voice.

[0179] In addition, the device's camera has a built-in object recognition function that automatically captures images when it recognizes an object held by the child. At this time, a timestamp is added to the recognized image, and it is processed by an emotion engine to analyze the child's facial expressions. This allows the child's emotional state to be evaluated from the image as well.

[0180] The analyzed audio and image data is sent from the device to the server via the network. The server stores this data in a database and shares it among family members based on specific conditions. This allows users to share their child's growth and daily emotional changes with their family in real time, without any mechanical or manual operation.

[0181] For example, consider a scenario where you record the moment a child rides a bicycle outdoors for the first time, capturing a video of them smiling and moving forward. An example of a prompt using a generative AI model would be, "Analyze the video of the child riding the bicycle and pedaling with a smile, and create a message to convey that emotional state to the family." This AI model automatically generates a message for the family that reflects the emotions, based on the recorded data.

[0182] By implementing this system, users can meticulously record their child's growth and share it with their family, including changes in their emotions, thereby promoting communication within the family.

[0183] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0184] Step 1:

[0185] The device continuously monitors ambient sounds using a microphone and detects children's vocalizations. When sound is detected, the device begins recording audio data. Real-time audio signals are input to the microphone. The device converts these audio signals into digital audio data, adds a timestamp, and stores it in temporary storage. An audio file with time information is generated as output.

[0186] Step 2:

[0187] The device processes audio data stored in its storage using an emotion engine. Audio files with time information are used as input. The emotion engine performs differential calculations to analyze emotional states from audio features such as volume, pitch, and speed, and outputs data indicating a specific emotion as the analysis result.

[0188] Step 3:

[0189] The device uses the camera's object recognition function to identify objects and actions held by the child. Camera footage is used as input for real-time analysis. When an object is recognized, the camera automatically captures an image and adds a timestamp to the image data. An image file with time information is generated as output.

[0190] Step 4:

[0191] The device processes image data using an emotion engine. An image file with time information is used as input. The emotion engine analyzes facial expressions and postures in the image and estimates the emotional state using face recognition technology. The output generates data indicating the emotional state read from the image.

[0192] Step 5:

[0193] The terminal transmits analyzed audio and image data to the server over the network. Sentimental data from the audio and images is used as input. A secure protocol is used to transfer the data to the server while maintaining data integrity. As output, integrated data is generated and stored on the server.

[0194] Step 6:

[0195] The server organizes and stores the received emotional data in a database. The input is the emotional data from transmitted audio and images. The data analysis results are categorized and managed in a format easily accessible to family members. The output is a dataset of visualized emotional states.

[0196] Step 7:

[0197] The server automatically shares data among family members according to specific conditions. Organized emotional data and family structure information are used as input. Leveraging a communication platform, pre-configured prompts trigger a generative AI model to create messages, which are then sent to specific recipients. Outputs include notifications and messages to family members that reflect their emotional state.

[0198] (Application Example 2)

[0199] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as a "server" and the smart device 14 as a "terminal".

[0200] The challenge is to provide a more effective in-home observation system that not only records a child's developmental stages but also captures their emotional state in detail, allowing for rich emotional sharing with family members living far away.

[0201] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0202] In this invention, the server includes means for detecting and recording a child's voice, means for assigning a timestamp to the recorded voice and saving it, and means for using an emotion analysis engine to analyze the emotional state from the voice and image data. This makes it possible to record a child's growth along with their emotions and share it with family in real time in an emotionally rich way.

[0203] "Methods for detecting and recording children's voices" refers to a function that recognizes surrounding sounds, selects voices emitted by specific individuals, and records them.

[0204] "Means for adding a timestamp to recorded audio and saving it" refers to a function that adds the recording time to acquired audio data and stores it together with identification information.

[0205] "Means using an emotion analysis engine that analyzes emotional states from audio and image data" refers to software that estimates the emotional changes of a specific person based on audio and video information and outputs it as data.

[0206] "Means of sharing via communication" refers to a system that enables information sharing by sending and receiving data between multiple devices and users via a network.

[0207] "Methods for detecting and photographing objects held by children" refers to a function that uses a visual sensor to recognize an object, records the movements of the person holding the object, and saves it as an image.

[0208] "A means of sharing video containing the results of emotional state analysis via communication" refers to a function that transmits video data with emotional information evaluated by an emotion analysis engine to other terminals via a network, thereby sharing the information.

[0209] "Means of providing information to specific users through a communication infrastructure in accordance with predetermined conditions" refers to a communication system for distributing selected data to specific recipients based on conditions set by the user.

[0210] This invention provides a system that uses a terminal operating in the home to record a child's voice and video and analyze their emotions. The terminal is equipped with a microphone and a camera, which detect specific sounds or movements. Voice is recorded in real time, saved in digital format, and time-stamped. An emotion analysis engine is applied to the recorded voice data to identify the emotional state at that moment. Similarly, the camera recognizes objects within its field of view, captures the moment, and saves it as image data. Emotion analysis is also performed on the image data, and the results are added.

[0211] Next, the terminal sends this data to the server over the network. The server stores the received data and transmits it to a specific user based on pre-configured conditions via appropriate communication methods.

[0212] For example, if a user is a relative living far away, they can share details about an event that the relative is particularly interested in, such as the child's participation in that event. Based on a prompt like, "Do you remember the smile on your child's face the day they took their first steps?", events and emotional moments identified through sentiment analysis are shared and provided along with actual video footage. In this way, parents and relatives can use this as a means to understand the details of life, including emotional changes, and deepen communication.

[0213] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0214] Step 1:

[0215] The device continuously monitors ambient sounds and detects children's voices and specific audio signals. The input is audio data, and the output is a digital recording of the detected sounds. This recording is temporarily stored in internal memory and timestamped.

[0216] Step 2:

[0217] This process applies an emotion analysis engine to recorded audio data to analyze the emotional state. The input is the audio data obtained in step 1, and the output is data containing the analyzed emotion information. This process uses an audio pattern recognition algorithm to analyze the tone and content of the audio.

[0218] Step 3:

[0219] The device's built-in camera operates, capturing images of the child's movements and objects they are holding. The input is video data obtained from the camera sensor, and the output is an image file with a timestamp. An object recognition algorithm is applied to detect specific objects.

[0220] Step 4:

[0221] The captured image data is also analyzed using an emotion analysis engine to estimate emotional states from facial expressions and other factors. The input is the image file obtained in step 3, and the output is image data with added emotional information. The analyzed emotional state is added to the video data as metadata.

[0222] Step 5:

[0223] The terminal transmits audio and image data, including the emotion data obtained in steps 2 and 4, to the server via the network. The input is this data, and the output is the adapted data transferred to the server. The data is securely uploaded using a transmission protocol.

[0224] Step 6:

[0225] The server stores the received data and delivers it to specific users based on defined conditions. The input is the data sent in step 5, and the output is the data delivered to the user's terminal. Data filtering and transmission are performed according to the prompts set by the user.

[0226] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0227] Data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of data generation model 58 is ChatGPT (registered trademark) (Internet search).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0228] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart device 14.

[0229] [Second Embodiment]

[0230] Figure 3 shows an example of the configuration of the data processing system 210 according to the second embodiment.

[0231] As shown in Figure 3, the data processing system 210 includes a data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0232] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0233] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication interface 44. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, and camera 42 are also connected to the bus 52.

[0234] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0235] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0236] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0237] Figure 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Figure 4, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0238] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0239] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0240] In the smart glasses 214, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0241] Next, the identification processing performed by the identification processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0242] This invention provides a method for recording a child's growth process in audio and images and sharing it with distant family members via a network. This allows users, who are parents raising children, to capture precious moments and save memories, even amidst their busy daily lives, and share them with their families. The following describes how this invention is implemented.

[0243] First, let's discuss voice recording. The device is equipped with a voice detection mechanism to detect children's voices. When the user's child speaks, the device collects the sound through the microphone and starts recording. The recorded voice data is automatically time-stamped and temporarily stored in storage. This makes it easy to track what was said and when.

[0244] Next, the recorded audio data is periodically transmitted to a server via the network. The server processes the received audio using speech analysis tools, extracting and organizing important phrases and words. It is then transcribed as needed, and the analysis results are recorded in a database. This information is stored in a way that allows family members to easily search and review it later.

[0245] Regarding photography, the device's built-in camera is equipped with object recognition capabilities to recognize objects held by the child. As soon as an object is detected, the camera activates, capturing the child's movements and the object they are holding. The captured images are time-stamped and saved to storage.

[0246] The saved image data is also transmitted to a server via the network and stored in a database. The server then uses a communication platform to automatically share this data with specific users based on pre-configured conditions. For example, this allows for instant image sharing to a grandparent's account.

[0247] This system automates the recording of a child's growth without requiring specific manual operations from the user, ensuring that important moments are not missed even amidst the busyness of daily life. Furthermore, these records are always easy to share with family, fostering communication among users.

[0248] The following describes the processing flow.

[0249] Step 1:

[0250] The device constantly monitors ambient sounds using voice detection and detects children's voices. When a child's voice is detected, it starts recording using the built-in microphone.

[0251] Step 2:

[0252] The device adds a timestamp to the recorded audio data and temporarily saves it to storage.

[0253] Step 3:

[0254] The device sends the stored audio data to the server in conjunction with the timing of its network connection.

[0255] Step 4:

[0256] The server receives the transmitted audio data and processes it using audio analysis tools. It extracts important statements and phrases and transcribes them if necessary.

[0257] Step 5:

[0258] The server stores the analysis results in a database and manages that data so that users can view it.

[0259] Step 6:

[0260] The device uses its camera to detect an object the child is holding using object recognition technology and captures a photograph at that moment.

[0261] Step 7:

[0262] The device adds a timestamp to the image it takes and temporarily saves it to storage.

[0263] Step 8:

[0264] The device sends stored image data to the server via the network.

[0265] Step 9:

[0266] The server stores image data in a database and shares the data with specific users through a communication platform set by the user.

[0267] Step 10:

[0268] Users receive notifications, review data shared on social media and platforms, and provide feedback as needed.

[0269] (Example 1)

[0270] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0271] In modern times, families raising children need a quick and efficient way to record and share their children's growth moments. However, manual recording and photography, as well as managing and sharing that data, is time-consuming and burdensome for busy parents. This leads to challenges such as failing to record precious moments and difficulty in timely information sharing with family.

[0272] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0273] In this invention, the server includes an acoustic acquisition means for detecting and collecting a child's voice, a storage means for temporarily storing the collected voice with time information attached, and a data analysis means for analyzing the stored voice and selecting specific phrases or words. This makes it possible to effectively capture important moments in a child's development and quickly and automatically share that information with the family.

[0274] "Sound acquisition means" refers to a device equipped with the function of automatically detecting ambient sounds and collecting sound information.

[0275] "Time information" refers to data that indicates the specific date and time when audio or images were acquired, enabling later tracking and organization.

[0276] "Storage means" refers to technologies for temporarily or permanently storing collected audio and image data in a storage device.

[0277] "Data analysis means" refers to a program or device that has the function of processing stored audio or image data and extracting specific information.

[0278] "Information sharing means" refers to a system that uses communication technology to transfer analyzed information to other devices or users.

[0279] "Visual acquisition means" refers to a device that uses cameras or sensors to acquire image information of the environment and record a specific object.

[0280] "Information distribution means" refers to technology that transmits stored or analyzed data to other users or systems via a network.

[0281] A "selective transmission method" refers to a system equipped with the function of selecting and transmitting only appropriate data based on set criteria and conditions.

[0282] This invention provides an automated system for efficiently recording a child's growth process and easily sharing it among family members. This system combines voice acquisition means and image acquisition means, and has the function of efficiently acquiring, analyzing, storing, and sharing voice and images.

[0283] Audio acquisition and processing

[0284] The terminal is equipped with a microphone as a voice acquisition means and automatically detects the voice of the child. This voice data is given time information and temporarily stored in local storage. The collected voice data is sent to the server via the network. The server analyzes the data using advanced voice analysis software and extracts important phrases and words. This analysis result is texturized as needed and stored in a database so that it can be accessed later.

[0285] Image Acquisition and Processing

[0286] The terminal has a built-in camera as an image acquisition means and has the function of recognizing the object held by the child. When an object is detected, the camera automatically activates and takes a picture. This image data is also given time information and temporarily stored in local storage. The server receives the image data and stores it in the database. Then, the image data is automatically shared with the family based on defined conditions.

[0287] Specific Example

[0288] For example, when the user's 3-year-old child says "Mom" for the first time, this system immediately records the voice and analyzes it. The server recognizes the word "Mom" as an important phrase and converts it into text data. This data is immediately sent to the grandparents' accounts, allowing the family to share and enjoy that moment.

[0289] Example of Prompt Sentence

[0290] For example, it is possible to input the following as a prompt sentence to the generative AI model:

[0291] "Please convert the voice of the child's first words into text format, extract important words, and save them so that they can be shared with the family."

[0292] This system allows users to record important moments amidst their busy daily lives and communicate smoothly with their families.

[0293] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0294] Step 1:

[0295] The device detects the child's voice. To initiate voice detection, the device constantly monitors the surrounding sounds. When the child's voice exceeds a certain threshold, the device automatically starts recording. The input here is the voice data acquired through the microphone, and the output is the data stream of the recording that has started.

[0296] Step 2:

[0297] The device adds time information to the recorded audio data. A timestamp is added to the audio data as soon as recording begins. This allows the device to identify the date and time the recording was made. The input is the audio data being recorded, and the output is the audio data with the time information added.

[0298] Step 3:

[0299] The device temporarily stores the audio data, along with the assigned time information, in local storage. The data is stored in an efficiently accessible format. The input is audio data with time information, and the output is the temporarily stored audio file.

[0300] Step 4:

[0301] The terminal sends stored audio data to the server at regular intervals. Here, the terminal securely transfers the data over the network. The input is the audio data stored in local storage, and the output is the audio data received by the server.

[0302] Step 5:

[0303] The server analyzes the received voice data. Using voice analysis software, the server analyzes the data and extracts important phrases and words. A generative AI model is used in this process. The input is the raw voice data received by the server, and the output is the analyzed phrases and words.

[0304] Step 6:

[0305] The server saves the analysis results including the extracted phrases and words in a database. The analysis results may be converted into text format. This makes it easier for family members to access later. The input is the analyzed phrases and words, and the output is the text data recorded in the database.

[0306] Step 7:

[0307] The server shares the saved analysis results with specific users according to predetermined conditions. It sends the data to pre-set family members via a communication platform. The input is the saved analysis results, and the output is the notification received by the users.

[0308] Step 8:

[0309] The terminal recognizes the objects held by the child and detects their movements. Using a camera and sensors, when the terminal confirms the movement, it takes a picture. The input is the surrounding visual information captured by the camera sensor, and the output is the captured image data.

[0310] Step 9:

[0311] The terminal assigns time information to the captured image data. The time at the moment when the image is captured is recorded. The input is the captured image, and the output is the image data with time information added.

[0312] Step 10:

[0313] The terminal sends image data with assigned time information to the server. The data is transmitted appropriately over the network. The input is image data containing time information, and the output is image data received by the server.

[0314] Step 11:

[0315] The server stores the received image data in a database. The image data is then shared among family members based on certain criteria. The input is the received image data, and the output is the record in the database and the sharing notification to the family.

[0316] (Application Example 1)

[0317] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0318] Recording and sharing a child's development is an important form of communication within the family, but the need for manual operation makes it a burden, especially for busy parents. There is also a need to record every moment of growth to create richer memories. Furthermore, there is a desire to gain a more comprehensive understanding of a child's development by utilizing the results of voice and motion analysis.

[0319] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0320] In this invention, the server includes means for analyzing voice and notifying when a specific event has occurred, means for detecting surrounding movements and triggering specific actions, and means for inputting analysis results based on voice and actions into an AI model. This makes it possible to automatically record moments of a child's growth even amidst the busyness of daily life, and to appropriately share those records with family.

[0321] "Children's voices" refers to speech or voices of minors detected through recording devices.

[0322] "Means of recording" refers to hardware or software for saving audio as recorded data.

[0323] "Means for adding and saving timestamps" refers to a system or method for adding date and time information to recorded data and saving it.

[0324] "Means for analyzing and extracting important statements" refers to technology that identifies and extracts important information from recorded audio data based on specific criteria.

[0325] "Means of sharing via communication" refers to methods and technologies for sharing data with other terminals or systems via a network.

[0326] "A means of analyzing audio and notifying users that a specific event has occurred" refers to a system that analyzes audio data and notifies the user when a specific event is detected.

[0327] "Means of detecting surrounding movements and triggering specific actions" refers to technologies that detect the movement of the environment or objects using sensors, and automatically initiate specific actions in response.

[0328] "Means of inputting into a generative AI model" refers to an interface used to input analysis results and data into an artificial intelligence model, allowing that model to perform further processing and predictions.

[0329] The system implementing this invention is based on a terminal installed in the home. The terminal is equipped with an audio recording device and an image capture device, which are used to record the child's voice and actions in real time. A timestamp is automatically added to the audio and images, and the data is temporarily stored in the terminal's storage device.

[0330] The server receives audio data transmitted from the terminal and performs analysis using speech recognition software. This analysis extracts important statements and events based on specific criteria and notifies the user as needed. Furthermore, it applies motion recognition algorithms to image data, detecting the child's movements and behavior, and inputting the results into a generating AI model. Software such as Vosk and OpenCV can be used for this process.

[0331] Notifications and analysis results from the server are shared with specific users via the network according to pre-configured conditions. This operation utilizes cloud services and communication platforms, enabling seamless data sharing among family members.

[0332] For example, if the device records a child's first words or a drawing, these recordings are immediately analyzed and processed by the server, and the family is notified. An example of a prompt for the generating AI model might be: "Describe how the robot behaves in the home. In particular, focus on how it captures the child's voice and movements and how it sends and shares that information to the server."

[0333] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0334] Step 1:

[0335] The device uses an audio recording device to detect and record children's speech in real time within the environment. It receives an audio signal as input, adds a timestamp to the audio data, and temporarily stores it in a memory device. It generates processed audio data as output.

[0336] Step 2:

[0337] The device uses an image capture device to detect the child's movements. It takes a video feed as input and applies a motion detection algorithm. If movement is detected, an image is captured and saved along with a timestamp. The output is image data captured triggered by movement.

[0338] Step 3:

[0339] The server receives audio data transmitted from the terminal. Using the processed audio data as input, speech recognition software performs analysis to extract specific important statements. In this process, text data is generated from the audio signal and output as the analysis result.

[0340] Step 4:

[0341] The server receives image data from the terminal and applies a motion recognition algorithm. The input is the captured image data, which analyzes the child's movements and generates information about those movements, which is then input into an AI model. This results in the analysis results being output.

[0342] Step 5:

[0343] The server checks pre-configured notification conditions based on analysis results obtained from audio and image data. The input consists of analysis results and condition settings. If a notification is deemed necessary, it sends a notification to the specified user. The output is a notification message.

[0344] Step 6:

[0345] Users receive notifications from the server and review data as needed. They can view or see audio and image data on the platform and access additional information based on the generated AI model. The output includes visual and audio feedback based on user selection.

[0346] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0347] This invention provides a method for recording a child's developmental stages using audio and images, and by combining this with an emotion engine, simultaneously evaluating the child's emotional state and sharing this information with family members who live far away. This allows parents to record and share their child's daily activities along with their emotions, thereby deepening communication within the family and among relatives. The following describes how this invention is implemented.

[0348] First, let's discuss voice recording. The device is equipped with voice detection capabilities to detect children's voices. When the user's child speaks, the device collects the sound through the microphone and begins recording. The recorded voice data is time-stamped, and then the emotion engine is used to analyze the emotion from the voice. This emotion data is temporarily stored in storage along with the voice data.

[0349] Next, the saved audio and emotional data are transmitted to a server via the network. The server processes the received audio using audio analysis tools, extracting important phrases and issues, and integrating and storing emotional data from an emotional engine. The analysis results are recorded in a database and are structured to make it easy to understand the emotional state of the child.

[0350] Next, let's discuss photography. The camera built into the device is equipped with object recognition capabilities to recognize objects held by the child. As soon as an object is recognized, the camera activates and photographs the child's movements and the object they are holding. The captured images are time-stamped, and an emotion engine is used to analyze the child's emotions from their facial expressions. This emotion data is stored in storage along with the image data.

[0351] Image data and emotional data are transmitted to a server via the network. The server stores this data in a database and uses a communication platform to automatically share it with specific users based on pre-configured conditions. This allows grandparents and other family members to experience the child's growth and emotional changes in real time.

[0352] This system automates the recording of a child's growth and emotions without requiring specific manual operations from the user, ensuring that important moments are never missed. These recordings can always be shared with the family, helping to foster intimate communication.

[0353] The following describes the processing flow.

[0354] Step 1:

[0355] The device constantly monitors ambient sounds using voice detection and detects children's voices. When a child's voice is detected, it starts recording using the built-in microphone.

[0356] Step 2:

[0357] The device adds a timestamp to the recorded audio data and simultaneously uses an emotion engine to analyze the emotions in the audio data. The resulting emotional information is integrated into the audio data and temporarily stored in storage.

[0358] Step 3:

[0359] The device sends stored voice data and emotional information to the server in conjunction with the timing of its network connection.

[0360] Step 4:

[0361] The server receives the transmitted audio data and emotional information, extracts important statements and phrases using an audio analysis tool, associates them with emotional information, and stores them in a database.

[0362] Step 5:

[0363] The device's camera uses object recognition technology to detect the object the child is holding. When an object is detected, the camera activates and takes an image of that moment.

[0364] Step 6:

[0365] The device adds a timestamp to the image it captures, and then uses an emotion engine to analyze the facial expressions in the image and determine the emotion. The obtained emotion information is integrated into the image data and saved to storage.

[0366] Step 7:

[0367] The device transmits stored image data and emotional information to the server via the network.

[0368] Step 8:

[0369] The server stores image data and emotional information in a database and automatically provides the data to specific users via a communication platform.

[0370] Step 9:

[0371] Users receive notifications, review data shared on social media and other platforms, and gain real-time insights into their child's emotional state and development.

[0372] (Example 2)

[0373] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0374] This invention aims to solve the problem of difficulty for families living in remote locations to share their children's daily growth and emotional changes in real time and to engage in deep communication. Furthermore, conventional technologies lacked a mechanism to effectively combine voice and image recording with emotion analysis, so the challenge is to provide a means to efficiently achieve this.

[0375] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0376] In this invention, the server includes means for analyzing and evaluating voice using an emotion processing device, a device for analyzing images and evaluating emotional states, and a device for transmitting the analysis results to a remote recipient using communication technology. This makes it possible to automatically and instantly share a child's growth and emotional changes with family members in remote locations using voice and image data.

[0377] A "sound detection and recording device" is a device equipped with the function of detecting sounds emitted from children or the environment using a microphone and recording them in digital format.

[0378] A "device for adding and saving date and time information" is a device that has the function of adding a timestamp to recorded or photographed data and recording it as digital data along with that information.

[0379] A "device for extracting important information" is a device equipped with the function of selecting necessary sentences and words from recorded audio data based on predetermined criteria.

[0380] A "device that evaluates emotional states using an emotion processing device" is a device that analyzes voice and image data and uses an emotion engine to identify and evaluate an individual's emotional state.

[0381] A "device that identifies and photographs objects" is a device that uses a camera and artificial intelligence to automatically take a photograph when it recognizes a specific object.

[0382] A "device for sharing with remote locations via communication technology" is a device that has the function of transmitting and sharing recorded or photographed data to other locations or devices using the internet or wireless communication networks.

[0383] A "device that provides information to specific recipients via an information transmission platform" is a device that uses dedicated software or network services to provide information to specific individuals or groups according to pre-configured conditions.

[0384] This invention is a system that uses a terminal and a server to acquire and analyze a child's voice and images, evaluate their emotional state, and share it with family members in a remote location. The detailed procedure for implementing this system is shown below.

[0385] First, the device is equipped with a microphone and camera, which are used to acquire audio and images. Voice detection software on the device detects the child's vocalizations in real time, starts recording the audio, and generates audio data. Next, the recorded audio data is analyzed by software using an emotion engine to evaluate the child's emotional state. This emotion analysis technology uses algorithms that analyze the tone, pitch, and speed of the voice.

[0386] In addition, the device's camera has a built-in object recognition function that automatically captures images when it recognizes an object held by the child. At this time, a timestamp is added to the recognized image, and it is processed by an emotion engine to analyze the child's facial expressions. This allows the child's emotional state to be evaluated from the image as well.

[0387] The analyzed audio and image data is sent from the device to the server via the network. The server stores this data in a database and shares it among family members based on specific conditions. This allows users to share their child's growth and daily emotional changes with their family in real time, without any mechanical or manual operation.

[0388] For example, consider a scenario where you record the moment a child rides a bicycle outdoors for the first time, capturing a video of them smiling and moving forward. An example of a prompt using a generative AI model would be, "Analyze the video of the child riding the bicycle and pedaling with a smile, and create a message to convey that emotional state to the family." This AI model automatically generates a message for the family that reflects the emotions, based on the recorded data.

[0389] By implementing this system, users can meticulously record their child's growth and share it with their family, including changes in their emotions, thereby promoting communication within the family.

[0390] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0391] Step 1:

[0392] The device continuously monitors ambient sounds using a microphone and detects children's vocalizations. When sound is detected, the device begins recording audio data. Real-time audio signals are input to the microphone. The device converts these audio signals into digital audio data, adds a timestamp, and stores it in temporary storage. An audio file with time information is generated as output.

[0393] Step 2:

[0394] The device processes audio data stored in its storage using an emotion engine. Audio files with time information are used as input. The emotion engine performs differential calculations to analyze emotional states from audio features such as volume, pitch, and speed, and outputs data indicating a specific emotion as the analysis result.

[0395] Step 3:

[0396] The device uses the camera's object recognition function to identify objects and actions held by the child. Camera footage is used as input for real-time analysis. When an object is recognized, the camera automatically captures an image and adds a timestamp to the image data. An image file with time information is generated as output.

[0397] Step 4:

[0398] The device processes image data using an emotion engine. An image file with time information is used as input. The emotion engine analyzes facial expressions and postures in the image and estimates the emotional state using face recognition technology. The output generates data indicating the emotional state read from the image.

[0399] Step 5:

[0400] The terminal transmits analyzed audio and image data to the server over the network. Sentimental data from the audio and images is used as input. A secure protocol is used to transfer the data to the server while maintaining data integrity. As output, integrated data is generated and stored on the server.

[0401] Step 6:

[0402] The server organizes and stores the received emotional data in a database. The input is the emotional data from transmitted audio and images. The data analysis results are categorized and managed in a format easily accessible to family members. The output is a dataset of visualized emotional states.

[0403] Step 7:

[0404] The server automatically shares data among family members according to specific conditions. Organized emotional data and family structure information are used as input. Leveraging a communication platform, pre-configured prompts trigger a generative AI model to create messages, which are then sent to specific recipients. Outputs include notifications and messages to family members that reflect their emotional state.

[0405] (Application Example 2)

[0406] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0407] The challenge is to provide a more effective in-home observation system that not only records a child's developmental stages but also captures their emotional state in detail, allowing for rich emotional sharing with family members living far away.

[0408] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0409] In this invention, the server includes means for detecting and recording a child's voice, means for assigning a timestamp to the recorded voice and saving it, and means for using an emotion analysis engine to analyze the emotional state from the voice and image data. This makes it possible to record a child's growth along with their emotions and share it with family in real time in an emotionally rich way.

[0410] "Methods for detecting and recording children's voices" refers to a function that recognizes surrounding sounds, selects voices emitted by specific individuals, and records them.

[0411] "Means for adding a timestamp to recorded audio and saving it" refers to a function that adds the recording time to acquired audio data and stores it together with identification information.

[0412] "Means using an emotion analysis engine that analyzes emotional states from audio and image data" refers to software that estimates the emotional changes of a specific person based on audio and video information and outputs it as data.

[0413] "Means of sharing via communication" refers to a system that enables information sharing by sending and receiving data between multiple devices and users via a network.

[0414] "Methods for detecting and photographing objects held by children" refers to a function that uses a visual sensor to recognize an object, records the movements of the person holding the object, and saves it as an image.

[0415] "A means of sharing video containing the results of emotional state analysis via communication" refers to a function that transmits video data with emotional information evaluated by an emotion analysis engine to other terminals via a network, thereby sharing the information.

[0416] "Means of providing information to specific users through a communication infrastructure in accordance with predetermined conditions" refers to a communication system for distributing selected data to specific recipients based on conditions set by the user.

[0417] This invention provides a system that uses a terminal operating in the home to record a child's voice and video and analyze their emotions. The terminal is equipped with a microphone and a camera, which detect specific sounds or movements. Voice is recorded in real time, saved in digital format, and time-stamped. An emotion analysis engine is applied to the recorded voice data to identify the emotional state at that moment. Similarly, the camera recognizes objects within its field of view, captures the moment, and saves it as image data. Emotion analysis is also performed on the image data, and the results are added.

[0418] Next, the terminal sends this data to the server over the network. The server stores the received data and transmits it to a specific user based on pre-configured conditions via appropriate communication methods.

[0419] For example, if a user is a relative living far away, they can share details about an event that the relative is particularly interested in, such as the child's participation in that event. Based on a prompt like, "Do you remember the smile on your child's face the day they took their first steps?", events and emotional moments identified through sentiment analysis are shared and provided along with actual video footage. In this way, parents and relatives can use this as a means to understand the details of life, including emotional changes, and deepen communication.

[0420] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0421] Step 1:

[0422] The device continuously monitors ambient sounds and detects children's voices and specific audio signals. The input is audio data, and the output is a digital recording of the detected sounds. This recording is temporarily stored in internal memory and timestamped.

[0423] Step 2:

[0424] This process applies an emotion analysis engine to recorded audio data to analyze the emotional state. The input is the audio data obtained in step 1, and the output is data containing the analyzed emotion information. This process uses an audio pattern recognition algorithm to analyze the tone and content of the audio.

[0425] Step 3:

[0426] The device's built-in camera operates, capturing images of the child's movements and objects they are holding. The input is video data obtained from the camera sensor, and the output is an image file with a timestamp. An object recognition algorithm is applied to detect specific objects.

[0427] Step 4:

[0428] The captured image data is also analyzed using an emotion analysis engine to estimate emotional states from facial expressions and other factors. The input is the image file obtained in step 3, and the output is image data with added emotional information. The analyzed emotional state is added to the video data as metadata.

[0429] Step 5:

[0430] The terminal transmits audio and image data, including the emotion data obtained in steps 2 and 4, to the server via the network. The input is this data, and the output is the adapted data transferred to the server. The data is securely uploaded using a transmission protocol.

[0431] Step 6:

[0432] The server stores the received data and delivers it to specific users based on defined conditions. The input is the data sent in step 5, and the output is the data delivered to the user's terminal. Data filtering and transmission are performed according to the prompts set by the user.

[0433] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0434] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0435] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart glasses 214.

[0436] [Third Embodiment]

[0437] Figure 5 shows an example of the configuration of the data processing system 310 according to the third embodiment.

[0438] As shown in Figure 5, the data processing system 310 includes a data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0439] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0440] The headset terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a display 343. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and display 343 are also connected to the bus 52.

[0441] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0442] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0443] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0444] Figure 6 shows an example of the main functions of the data processing device 12 and the headset terminal 314. As shown in Figure 6, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0445] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0446] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0447] In the headset terminal 314, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0448] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the headset terminal 314 will be referred to as the "terminal".

[0449] This invention provides a method for recording a child's growth process in audio and images and sharing it with distant family members via a network. This allows users, who are parents raising children, to capture precious moments and save memories, even amidst their busy daily lives, and share them with their families. The following describes how this invention is implemented.

[0450] First, let's discuss voice recording. The device is equipped with a voice detection mechanism to detect children's voices. When the user's child speaks, the device collects the sound through the microphone and starts recording. The recorded voice data is automatically time-stamped and temporarily stored in storage. This makes it easy to track what was said and when.

[0451] Next, the recorded audio data is periodically transmitted to a server via the network. The server processes the received audio using speech analysis tools, extracting and organizing important phrases and words. It is then transcribed as needed, and the analysis results are recorded in a database. This information is stored in a way that allows family members to easily search and review it later.

[0452] Regarding photography, the device's built-in camera is equipped with object recognition capabilities to recognize objects held by the child. As soon as an object is detected, the camera activates, capturing the child's movements and the object they are holding. The captured images are time-stamped and saved to storage.

[0453] The saved image data is also transmitted to a server via the network and stored in a database. The server then uses a communication platform to automatically share this data with specific users based on pre-configured conditions. For example, this allows for instant image sharing to a grandparent's account.

[0454] This system automates the recording of a child's growth without requiring specific manual operations from the user, ensuring that important moments are not missed even amidst the busyness of daily life. Furthermore, these records are always easy to share with family, fostering communication among users.

[0455] The following describes the processing flow.

[0456] Step 1:

[0457] The device constantly monitors ambient sounds using voice detection and detects children's voices. When a child's voice is detected, it starts recording using the built-in microphone.

[0458] Step 2:

[0459] The device adds a timestamp to the recorded audio data and temporarily saves it to storage.

[0460] Step 3:

[0461] The device sends the stored audio data to the server in conjunction with the timing of its network connection.

[0462] Step 4:

[0463] The server receives the transmitted audio data and processes it using audio analysis tools. It extracts important statements and phrases and transcribes them if necessary.

[0464] Step 5:

[0465] The server stores the analysis results in a database and manages that data so that users can view it.

[0466] Step 6:

[0467] The device uses its camera to detect an object the child is holding using object recognition technology and captures a photograph at that moment.

[0468] Step 7:

[0469] The device adds a timestamp to the image it takes and temporarily saves it to storage.

[0470] Step 8:

[0471] The device sends stored image data to the server via the network.

[0472] Step 9:

[0473] The server stores image data in a database and shares the data with specific users through a communication platform set by the user.

[0474] Step 10:

[0475] Users receive notifications, review data shared on social media and platforms, and provide feedback as needed.

[0476] (Example 1)

[0477] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0478] In modern times, families raising children need a quick and efficient way to record and share their children's growth moments. However, manual recording and photography, as well as managing and sharing that data, is time-consuming and burdensome for busy parents. This leads to challenges such as failing to record precious moments and difficulty in timely information sharing with family.

[0479] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0480] In this invention, the server includes an acoustic acquisition means for detecting and collecting a child's voice, a storage means for temporarily storing the collected voice with time information attached, and a data analysis means for analyzing the stored voice and selecting specific phrases or words. This makes it possible to effectively capture important moments in a child's development and quickly and automatically share that information with the family.

[0481] "Sound acquisition means" refers to a device equipped with the function of automatically detecting ambient sounds and collecting sound information.

[0482] "Time information" refers to data that indicates the specific date and time when audio or images were acquired, enabling later tracking and organization.

[0483] "Storage means" refers to technologies for temporarily or permanently storing collected audio and image data in a storage device.

[0484] "Data analysis means" refers to a program or device that has the function of processing stored audio or image data and extracting specific information.

[0485] "Information sharing means" refers to a system that uses communication technology to transfer analyzed information to other devices or users.

[0486] "Visual acquisition means" refers to a device that uses cameras or sensors to acquire image information of the environment and record a specific object.

[0487] "Information distribution means" refers to technology that transmits stored or analyzed data to other users or systems via a network.

[0488] A "selective transmission method" refers to a system equipped with the function of selecting and transmitting only appropriate data based on set criteria and conditions.

[0489] This invention provides an automated system for efficiently recording a child's growth process and easily sharing it among family members. This system combines voice acquisition means and image acquisition means, and has the function of efficiently acquiring, analyzing, storing, and sharing voice and images.

[0490] Audio acquisition and processing

[0491] The device is equipped with a microphone to automatically detect children's voices. This voice data is tagged with time information and temporarily stored in local storage. The collected voice data is transmitted to a server via the network. The server uses advanced voice analysis software to analyze the data and extract important phrases and words. The analysis results are transcribed into text as needed and stored in a database for later access.

[0492] Image acquisition and processing

[0493] The device has a built-in camera for image acquisition and is equipped with a function to recognize objects held by children. When an object is detected, the camera automatically activates and takes a picture. This image data is also given time information and temporarily stored in local storage. The server receives the image data and stores it in a database. Subsequently, the image data is automatically shared with family members based on predetermined conditions.

[0494] Specific example

[0495] For example, if a user's three-year-old child says "Mama" for the first time, the system instantly records and analyzes the audio. The server recognizes "Mama" as an important phrase and converts it into text data. This data is immediately sent to the grandparents' account, allowing the family to share and enjoy that moment.

[0496] Example of a prompt

[0497] For example, the following can be entered as a prompt for the generative AI model:

[0498] "Convert your child's first words into audio format, extract important words, and save them to share with your family."

[0499] This system allows users to record important moments amidst their busy daily lives and communicate smoothly with their families.

[0500] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0501] Step 1:

[0502] The device detects the child's voice. To initiate voice detection, the device constantly monitors the surrounding sounds. When the child's voice exceeds a certain threshold, the device automatically starts recording. The input here is the voice data acquired through the microphone, and the output is the data stream of the recording that has started.

[0503] Step 2:

[0504] The device adds time information to the recorded audio data. A timestamp is added to the audio data as soon as recording begins. This allows the device to identify the date and time the recording was made. The input is the audio data being recorded, and the output is the audio data with the time information added.

[0505] Step 3:

[0506] The device temporarily stores the audio data, along with the assigned time information, in local storage. The data is stored in an efficiently accessible format. The input is audio data with time information, and the output is the temporarily stored audio file.

[0507] Step 4:

[0508] The terminal sends stored audio data to the server at regular intervals. Here, the terminal securely transfers the data over the network. The input is the audio data stored in local storage, and the output is the audio data received by the server.

[0509] Step 5:

[0510] The server analyzes the received audio data. Using speech analysis software, the server analyzes the data and extracts important phrases and words. A generative AI model is used in this process. The input is the raw audio data received by the server, and the output is the analyzed phrases and words.

[0511] Step 6:

[0512] The server stores the analysis results, including extracted phrases and words, in a database. The analysis results may be converted to text format, making them easier for family members to access later. The input is the analyzed phrases and words, and the output is the text data recorded in the database.

[0513] Step 7:

[0514] The server shares the saved analysis results with specific users according to predetermined conditions. It transmits data to pre-configured family members via a communication platform. The input is the saved analysis results, and the output is the notification received by the user.

[0515] Step 8:

[0516] The device recognizes objects held by a child and detects their movement. Using a camera and sensors, the device captures an image once it confirms the child's movement. The input is the surrounding visual information captured by the camera and sensors, and the output is the captured image data.

[0517] Step 9:

[0518] The device adds time information to the captured image data. The time at the moment the image was taken is recorded. The input is the captured image, and the output is the image data with time information added.

[0519] Step 10:

[0520] The terminal sends image data with assigned time information to the server. The data is transmitted appropriately over the network. The input is image data containing time information, and the output is image data received by the server.

[0521] Step 11:

[0522] The server stores the received image data in a database. The image data is then shared among family members based on certain criteria. The input is the received image data, and the output is the record in the database and the sharing notification to the family.

[0523] (Application Example 1)

[0524] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0525] Recording and sharing a child's development is an important form of communication within the family, but the need for manual operation makes it a burden, especially for busy parents. There is also a need to record every moment of growth to create richer memories. Furthermore, there is a desire to gain a more comprehensive understanding of a child's development by utilizing the results of voice and motion analysis.

[0526] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0527] In this invention, the server includes means for analyzing voice and notifying when a specific event has occurred, means for detecting surrounding movements and triggering specific actions, and means for inputting analysis results based on voice and actions into an AI model. This makes it possible to automatically record moments of a child's growth even amidst the busyness of daily life, and to appropriately share those records with family.

[0528] "Children's voices" refers to speech or voices of minors detected through recording devices.

[0529] "Means of recording" refers to hardware or software for saving audio as recorded data.

[0530] "Means for adding and saving timestamps" refers to a system or method for adding date and time information to recorded data and saving it.

[0531] "Means for analyzing and extracting important statements" refers to technology that identifies and extracts important information from recorded audio data based on specific criteria.

[0532] "Means of sharing via communication" refers to methods and technologies for sharing data with other terminals or systems via a network.

[0533] "A means of analyzing audio and notifying users that a specific event has occurred" refers to a system that analyzes audio data and notifies the user when a specific event is detected.

[0534] "Means of detecting surrounding movements and triggering specific actions" refers to technologies that detect the movement of the environment or objects using sensors, and automatically initiate specific actions in response.

[0535] "Means of inputting into a generative AI model" refers to an interface used to input analysis results and data into an artificial intelligence model, allowing that model to perform further processing and predictions.

[0536] The system implementing this invention is based on a terminal installed in the home. The terminal is equipped with an audio recording device and an image capture device, which are used to record the child's voice and actions in real time. A timestamp is automatically added to the audio and images, and the data is temporarily stored in the terminal's storage device.

[0537] The server receives audio data transmitted from the terminal and performs analysis using speech recognition software. This analysis extracts important statements and events based on specific criteria and notifies the user as needed. Furthermore, it applies motion recognition algorithms to image data, detecting the child's movements and behavior, and inputting the results into a generating AI model. Software such as Vosk and OpenCV can be used for this process.

[0538] Notifications and analysis results from the server are shared with specific users via the network according to pre-configured conditions. This operation utilizes cloud services and communication platforms, enabling seamless data sharing among family members.

[0539] For example, if the device records a child's first words or a drawing, these recordings are immediately analyzed and processed by the server, and the family is notified. An example of a prompt for the generating AI model might be: "Describe how the robot behaves in the home. In particular, focus on how it captures the child's voice and movements and how it sends and shares that information to the server."

[0540] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0541] Step 1:

[0542] The device uses an audio recording device to detect and record children's speech in real time within the environment. It receives an audio signal as input, adds a timestamp to the audio data, and temporarily stores it in a memory device. It generates processed audio data as output.

[0543] Step 2:

[0544] The device uses an image capture device to detect the child's movements. It takes a video feed as input and applies a motion detection algorithm. If movement is detected, an image is captured and saved along with a timestamp. The output is image data captured triggered by movement.

[0545] Step 3:

[0546] The server receives audio data transmitted from the terminal. Using the processed audio data as input, speech recognition software performs analysis to extract specific important statements. In this process, text data is generated from the audio signal and output as the analysis result.

[0547] Step 4:

[0548] The server receives image data from the terminal and applies a motion recognition algorithm. The input is the captured image data, which analyzes the child's movements and generates information about those movements, which is then input into an AI model. This results in the analysis results being output.

[0549] Step 5:

[0550] The server checks pre-configured notification conditions based on analysis results obtained from audio and image data. The input consists of analysis results and condition settings. If a notification is deemed necessary, it sends a notification to the specified user. The output is a notification message.

[0551] Step 6:

[0552] Users receive notifications from the server and review data as needed. They can view or see audio and image data on the platform and access additional information based on the generated AI model. The output includes visual and audio feedback based on user selection.

[0553] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0554] This invention provides a method for recording a child's developmental stages using audio and images, and by combining this with an emotion engine, simultaneously evaluating the child's emotional state and sharing this information with family members who live far away. This allows parents to record and share their child's daily activities along with their emotions, thereby deepening communication within the family and among relatives. The following describes how this invention is implemented.

[0555] First, let's discuss voice recording. The device is equipped with voice detection capabilities to detect children's voices. When the user's child speaks, the device collects the sound through the microphone and begins recording. The recorded voice data is time-stamped, and then the emotion engine is used to analyze the emotion from the voice. This emotion data is temporarily stored in storage along with the voice data.

[0556] Next, the saved audio and emotional data are transmitted to a server via the network. The server processes the received audio using audio analysis tools, extracting important phrases and issues, and integrating and storing emotional data from an emotional engine. The analysis results are recorded in a database and are structured to make it easy to understand the emotional state of the child.

[0557] Next, let's discuss photography. The camera built into the device is equipped with object recognition capabilities to recognize objects held by the child. As soon as an object is recognized, the camera activates and photographs the child's movements and the object they are holding. The captured images are time-stamped, and an emotion engine is used to analyze the child's emotions from their facial expressions. This emotion data is stored in storage along with the image data.

[0558] Image data and emotional data are transmitted to a server via the network. The server stores this data in a database and uses a communication platform to automatically share it with specific users based on pre-configured conditions. This allows grandparents and other family members to experience the child's growth and emotional changes in real time.

[0559] This system automates the recording of a child's growth and emotions without requiring specific manual operations from the user, ensuring that important moments are never missed. These recordings can always be shared with the family, helping to foster intimate communication.

[0560] The following describes the processing flow.

[0561] Step 1:

[0562] The device constantly monitors ambient sounds using voice detection and detects children's voices. When a child's voice is detected, it starts recording using the built-in microphone.

[0563] Step 2:

[0564] The device adds a timestamp to the recorded audio data and simultaneously uses an emotion engine to analyze the emotions in the audio data. The resulting emotional information is integrated into the audio data and temporarily stored in storage.

[0565] Step 3:

[0566] The device sends stored voice data and emotional information to the server in conjunction with the timing of its network connection.

[0567] Step 4:

[0568] The server receives the transmitted audio data and emotional information, extracts important statements and phrases using an audio analysis tool, associates them with emotional information, and stores them in a database.

[0569] Step 5:

[0570] The device's camera uses object recognition technology to detect the object the child is holding. When an object is detected, the camera activates and takes an image of that moment.

[0571] Step 6:

[0572] The device adds a timestamp to the image it captures, and then uses an emotion engine to analyze the facial expressions in the image and determine the emotion. The obtained emotion information is integrated into the image data and saved to storage.

[0573] Step 7:

[0574] The device transmits stored image data and emotional information to the server via the network.

[0575] Step 8:

[0576] The server stores image data and emotional information in a database and automatically provides the data to specific users via a communication platform.

[0577] Step 9:

[0578] Users receive notifications, review data shared on social media and other platforms, and gain real-time insights into their child's emotional state and development.

[0579] (Example 2)

[0580] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0581] This invention aims to solve the problem of difficulty for families living in remote locations to share their children's daily growth and emotional changes in real time and to engage in deep communication. Furthermore, conventional technologies lacked a mechanism to effectively combine voice and image recording with emotion analysis, so the challenge is to provide a means to efficiently achieve this.

[0582] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0583] In this invention, the server includes means for analyzing and evaluating voice using an emotion processing device, a device for analyzing images and evaluating emotional states, and a device for transmitting the analysis results to a remote recipient using communication technology. This makes it possible to automatically and instantly share a child's growth and emotional changes with family members in remote locations using voice and image data.

[0584] A "sound detection and recording device" is a device equipped with the function of detecting sounds emitted from children or the environment using a microphone and recording them in digital format.

[0585] A "device for adding and saving date and time information" is a device that has the function of adding a timestamp to recorded or photographed data and recording it as digital data along with that information.

[0586] A "device for extracting important information" is a device equipped with the function of selecting necessary sentences and words from recorded audio data based on predetermined criteria.

[0587] A "device that evaluates emotional states using an emotion processing device" is a device that analyzes voice and image data and uses an emotion engine to identify and evaluate an individual's emotional state.

[0588] A "device that identifies and photographs objects" is a device that uses a camera and artificial intelligence to automatically take a photograph when it recognizes a specific object.

[0589] A "device for sharing with remote locations via communication technology" is a device that has the function of transmitting and sharing recorded or photographed data to other locations or devices using the internet or wireless communication networks.

[0590] A "device that provides information to specific recipients via an information transmission platform" is a device that uses dedicated software or network services to provide information to specific individuals or groups according to pre-configured conditions.

[0591] This invention is a system that uses a terminal and a server to acquire and analyze a child's voice and images, evaluate their emotional state, and share it with family members in a remote location. The detailed procedure for implementing this system is shown below.

[0592] First, the device is equipped with a microphone and camera, which are used to acquire audio and images. Voice detection software on the device detects the child's vocalizations in real time, starts recording the audio, and generates audio data. Next, the recorded audio data is analyzed by software using an emotion engine to evaluate the child's emotional state. This emotion analysis technology uses algorithms that analyze the tone, pitch, and speed of the voice.

[0593] In addition, the device's camera has a built-in object recognition function that automatically captures images when it recognizes an object held by the child. At this time, a timestamp is added to the recognized image, and it is processed by an emotion engine to analyze the child's facial expressions. This allows the child's emotional state to be evaluated from the image as well.

[0594] The analyzed audio and image data is sent from the device to the server via the network. The server stores this data in a database and shares it among family members based on specific conditions. This allows users to share their child's growth and daily emotional changes with their family in real time, without any mechanical or manual operation.

[0595] For example, consider a scenario where you record the moment a child rides a bicycle outdoors for the first time, capturing a video of them smiling and moving forward. An example of a prompt using a generative AI model would be, "Analyze the video of the child riding the bicycle and pedaling with a smile, and create a message to convey that emotional state to the family." This AI model automatically generates a message for the family that reflects the emotions, based on the recorded data.

[0596] By implementing this system, users can meticulously record their child's growth and share it with their family, including changes in their emotions, thereby promoting communication within the family.

[0597] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0598] Step 1:

[0599] The device continuously monitors ambient sounds using a microphone and detects children's vocalizations. When sound is detected, the device begins recording audio data. Real-time audio signals are input to the microphone. The device converts these audio signals into digital audio data, adds a timestamp, and stores it in temporary storage. An audio file with time information is generated as output.

[0600] Step 2:

[0601] The device processes audio data stored in its storage using an emotion engine. Audio files with time information are used as input. The emotion engine performs differential calculations to analyze emotional states from audio features such as volume, pitch, and speed, and outputs data indicating a specific emotion as the analysis result.

[0602] Step 3:

[0603] The device uses the camera's object recognition function to identify objects and actions held by the child. Camera footage is used as input for real-time analysis. When an object is recognized, the camera automatically captures an image and adds a timestamp to the image data. An image file with time information is generated as output.

[0604] Step 4:

[0605] The device processes image data using an emotion engine. An image file with time information is used as input. The emotion engine analyzes facial expressions and postures in the image and estimates the emotional state using face recognition technology. The output generates data indicating the emotional state read from the image.

[0606] Step 5:

[0607] The terminal transmits analyzed audio and image data to the server over the network. Sentimental data from the audio and images is used as input. A secure protocol is used to transfer the data to the server while maintaining data integrity. As output, integrated data is generated and stored on the server.

[0608] Step 6:

[0609] The server organizes and stores the received emotional data in a database. The input is the emotional data from transmitted audio and images. The data analysis results are categorized and managed in a format easily accessible to family members. The output is a dataset of visualized emotional states.

[0610] Step 7:

[0611] The server automatically shares data among family members according to specific conditions. Organized emotional data and family structure information are used as input. Leveraging a communication platform, pre-configured prompts trigger a generative AI model to create messages, which are then sent to specific recipients. Outputs include notifications and messages to family members that reflect their emotional state.

[0612] (Application Example 2)

[0613] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0614] The challenge is to provide a more effective in-home observation system that not only records a child's developmental stages but also captures their emotional state in detail, allowing for rich emotional sharing with family members living far away.

[0615] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0616] In this invention, the server includes means for detecting and recording a child's voice, means for assigning a timestamp to the recorded voice and saving it, and means for using an emotion analysis engine to analyze the emotional state from the voice and image data. This makes it possible to record a child's growth along with their emotions and share it with family in real time in an emotionally rich way.

[0617] "Methods for detecting and recording children's voices" refers to a function that recognizes surrounding sounds, selects voices emitted by specific individuals, and records them.

[0618] "Means for adding a timestamp to recorded audio and saving it" refers to a function that adds the recording time to acquired audio data and stores it together with identification information.

[0619] "Means using an emotion analysis engine that analyzes emotional states from audio and image data" refers to software that estimates the emotional changes of a specific person based on audio and video information and outputs it as data.

[0620] "Means of sharing via communication" refers to a system that enables information sharing by sending and receiving data between multiple devices and users via a network.

[0621] "Methods for detecting and photographing objects held by children" refers to a function that uses a visual sensor to recognize an object, records the movements of the person holding the object, and saves it as an image.

[0622] "A means of sharing video containing the results of emotional state analysis via communication" refers to a function that transmits video data with emotional information evaluated by an emotion analysis engine to other terminals via a network, thereby sharing the information.

[0623] "Means of providing information to specific users through a communication infrastructure in accordance with predetermined conditions" refers to a communication system for distributing selected data to specific recipients based on conditions set by the user.

[0624] This invention provides a system that uses a terminal operating in the home to record a child's voice and video and analyze their emotions. The terminal is equipped with a microphone and a camera, which detect specific sounds or movements. Voice is recorded in real time, saved in digital format, and time-stamped. An emotion analysis engine is applied to the recorded voice data to identify the emotional state at that moment. Similarly, the camera recognizes objects within its field of view, captures the moment, and saves it as image data. Emotion analysis is also performed on the image data, and the results are added.

[0625] Next, the terminal sends this data to the server over the network. The server stores the received data and transmits it to a specific user based on pre-configured conditions via appropriate communication methods.

[0626] For example, if a user is a relative living far away, they can share details about an event that the relative is particularly interested in, such as the child's participation in that event. Based on a prompt like, "Do you remember the smile on your child's face the day they took their first steps?", events and emotional moments identified through sentiment analysis are shared and provided along with actual video footage. In this way, parents and relatives can use this as a means to understand the details of life, including emotional changes, and deepen communication.

[0627] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0628] Step 1:

[0629] The device continuously monitors ambient sounds and detects children's voices and specific audio signals. The input is audio data, and the output is a digital recording of the detected sounds. This recording is temporarily stored in internal memory and timestamped.

[0630] Step 2:

[0631] This process applies an emotion analysis engine to recorded audio data to analyze the emotional state. The input is the audio data obtained in step 1, and the output is data containing the analyzed emotion information. This process uses an audio pattern recognition algorithm to analyze the tone and content of the audio.

[0632] Step 3:

[0633] The device's built-in camera operates, capturing images of the child's movements and objects they are holding. The input is video data obtained from the camera sensor, and the output is an image file with a timestamp. An object recognition algorithm is applied to detect specific objects.

[0634] Step 4:

[0635] The captured image data is also analyzed using an emotion analysis engine to estimate emotional states from facial expressions and other factors. The input is the image file obtained in step 3, and the output is image data with added emotional information. The analyzed emotional state is added to the video data as metadata.

[0636] Step 5:

[0637] The terminal transmits audio and image data, including the emotion data obtained in steps 2 and 4, to the server via the network. The input is this data, and the output is the adapted data transferred to the server. The data is securely uploaded using a transmission protocol.

[0638] Step 6:

[0639] The server stores the received data and delivers it to specific users based on defined conditions. The input is the data sent in step 5, and the output is the data delivered to the user's terminal. Data filtering and transmission are performed according to the prompts set by the user.

[0640] The specific processing unit 290 transmits the result of the specific processing to the headset terminal 314. In the headset terminal 314, the control unit 46A causes the speaker 240 and display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0641] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0642] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and specific processing may also be performed by the headset terminal 314.

[0643] [Fourth Embodiment]

[0644] Figure 7 shows an example of the configuration of the data processing system 410 according to the fourth embodiment.

[0645] As shown in Figure 7, the data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[0646] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0647] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a controlled object 443. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and controlled object 443 are also connected to the bus 52.

[0648] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0649] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0650] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0651] The controlled object 443 includes a display device, LEDs in the eyes, and motors that drive the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the robot 414's emotions can be expressed by controlling these motors. Furthermore, the robot 414's facial expressions can also be expressed by controlling the illumination state of the LEDs in its eyes.

[0652] Figure 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Figure 8, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0653] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0654] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0655] In robot 414, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0656] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0657] This invention provides a method for recording a child's growth process in audio and images and sharing it with distant family members via a network. This allows users, who are parents raising children, to capture precious moments and save memories, even amidst their busy daily lives, and share them with their families. The following describes how this invention is implemented.

[0658] First, let's discuss voice recording. The device is equipped with a voice detection mechanism to detect children's voices. When the user's child speaks, the device collects the sound through the microphone and starts recording. The recorded voice data is automatically time-stamped and temporarily stored in storage. This makes it easy to track what was said and when.

[0659] Next, the recorded audio data is periodically transmitted to a server via the network. The server processes the received audio using speech analysis tools, extracting and organizing important phrases and words. It is then transcribed as needed, and the analysis results are recorded in a database. This information is stored in a way that allows family members to easily search and review it later.

[0660] Regarding photography, the device's built-in camera is equipped with object recognition capabilities to recognize objects held by the child. As soon as an object is detected, the camera activates, capturing the child's movements and the object they are holding. The captured images are time-stamped and saved to storage.

[0661] The saved image data is also transmitted to a server via the network and stored in a database. The server then uses a communication platform to automatically share this data with specific users based on pre-configured conditions. For example, this allows for instant image sharing to a grandparent's account.

[0662] This system automates the recording of a child's growth without requiring specific manual operations from the user, ensuring that important moments are not missed even amidst the busyness of daily life. Furthermore, these records are always easy to share with family, fostering communication among users.

[0663] The following describes the processing flow.

[0664] Step 1:

[0665] The device constantly monitors ambient sounds using voice detection and detects children's voices. When a child's voice is detected, it starts recording using the built-in microphone.

[0666] Step 2:

[0667] The device adds a timestamp to the recorded audio data and temporarily saves it to storage.

[0668] Step 3:

[0669] The device sends the stored audio data to the server in conjunction with the timing of its network connection.

[0670] Step 4:

[0671] The server receives the transmitted audio data and processes it using audio analysis tools. It extracts important statements and phrases and transcribes them if necessary.

[0672] Step 5:

[0673] The server stores the analysis results in a database and manages that data so that users can view it.

[0674] Step 6:

[0675] The device uses its camera to detect an object the child is holding using object recognition technology and captures a photograph at that moment.

[0676] Step 7:

[0677] The device adds a timestamp to the image it takes and temporarily saves it to storage.

[0678] Step 8:

[0679] The device sends stored image data to the server via the network.

[0680] Step 9:

[0681] The server stores image data in a database and shares the data with specific users through a communication platform set by the user.

[0682] Step 10:

[0683] Users receive notifications, review data shared on social media and platforms, and provide feedback as needed.

[0684] (Example 1)

[0685] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0686] In modern times, families raising children need a quick and efficient way to record and share their children's growth moments. However, manual recording and photography, as well as managing and sharing that data, is time-consuming and burdensome for busy parents. This leads to challenges such as failing to record precious moments and difficulty in timely information sharing with family.

[0687] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0688] In this invention, the server includes an acoustic acquisition means for detecting and collecting a child's voice, a storage means for temporarily storing the collected voice with time information attached, and a data analysis means for analyzing the stored voice and selecting specific phrases or words. This makes it possible to effectively capture important moments in a child's development and quickly and automatically share that information with the family.

[0689] "Sound acquisition means" refers to a device equipped with the function of automatically detecting ambient sounds and collecting sound information.

[0690] "Time information" refers to data that indicates the specific date and time when audio or images were acquired, enabling later tracking and organization.

[0691] "Storage means" refers to technologies for temporarily or permanently storing collected audio and image data in a storage device.

[0692] "Data analysis means" refers to a program or device that has the function of processing stored audio or image data and extracting specific information.

[0693] "Information sharing means" refers to a system that uses communication technology to transfer analyzed information to other devices or users.

[0694] "Visual acquisition means" refers to a device that uses cameras or sensors to acquire image information of the environment and record a specific object.

[0695] "Information distribution means" refers to technology that transmits stored or analyzed data to other users or systems via a network.

[0696] A "selective transmission method" refers to a system equipped with the function of selecting and transmitting only appropriate data based on set criteria and conditions.

[0697] This invention provides an automated system for efficiently recording a child's growth process and easily sharing it among family members. This system combines voice acquisition means and image acquisition means, and has the function of efficiently acquiring, analyzing, storing, and sharing voice and images.

[0698] Audio acquisition and processing

[0699] The device is equipped with a microphone to automatically detect children's voices. This voice data is tagged with time information and temporarily stored in local storage. The collected voice data is transmitted to a server via the network. The server uses advanced voice analysis software to analyze the data and extract important phrases and words. The analysis results are transcribed into text as needed and stored in a database for later access.

[0700] Image acquisition and processing

[0701] The device has a built-in camera for image acquisition and is equipped with a function to recognize objects held by children. When an object is detected, the camera automatically activates and takes a picture. This image data is also given time information and temporarily stored in local storage. The server receives the image data and stores it in a database. Subsequently, the image data is automatically shared with family members based on predetermined conditions.

[0702] Specific example

[0703] For example, if a user's three-year-old child says "Mama" for the first time, the system instantly records and analyzes the audio. The server recognizes "Mama" as an important phrase and converts it into text data. This data is immediately sent to the grandparents' account, allowing the family to share and enjoy that moment.

[0704] Example of a prompt

[0705] For example, the following can be entered as a prompt for the generative AI model:

[0706] "Convert your child's first words into audio format, extract important words, and save them to share with your family."

[0707] This system allows users to record important moments amidst their busy daily lives and communicate smoothly with their families.

[0708] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0709] Step 1:

[0710] The device detects the child's voice. To initiate voice detection, the device constantly monitors the surrounding sounds. When the child's voice exceeds a certain threshold, the device automatically starts recording. The input here is the voice data acquired through the microphone, and the output is the data stream of the recording that has started.

[0711] Step 2:

[0712] The device adds time information to the recorded audio data. A timestamp is added to the audio data as soon as recording begins. This allows the device to identify the date and time the recording was made. The input is the audio data being recorded, and the output is the audio data with the time information added.

[0713] Step 3:

[0714] The device temporarily stores the audio data, along with the assigned time information, in local storage. The data is stored in an efficiently accessible format. The input is audio data with time information, and the output is the temporarily stored audio file.

[0715] Step 4:

[0716] The terminal sends stored audio data to the server at regular intervals. Here, the terminal securely transfers the data over the network. The input is the audio data stored in local storage, and the output is the audio data received by the server.

[0717] Step 5:

[0718] The server analyzes the received audio data. Using speech analysis software, the server analyzes the data and extracts important phrases and words. A generative AI model is used in this process. The input is the raw audio data received by the server, and the output is the analyzed phrases and words.

[0719] Step 6:

[0720] The server stores the analysis results, including extracted phrases and words, in a database. The analysis results may be converted to text format, making them easier for family members to access later. The input is the analyzed phrases and words, and the output is the text data recorded in the database.

[0721] Step 7:

[0722] The server shares the saved analysis results with specific users according to predetermined conditions. It transmits data to pre-configured family members via a communication platform. The input is the saved analysis results, and the output is the notification received by the user.

[0723] Step 8:

[0724] The device recognizes objects held by a child and detects their movement. Using a camera and sensors, the device captures an image once it confirms the child's movement. The input is the surrounding visual information captured by the camera and sensors, and the output is the captured image data.

[0725] Step 9:

[0726] The device adds time information to the captured image data. The time at the moment the image was taken is recorded. The input is the captured image, and the output is the image data with time information added.

[0727] Step 10:

[0728] The terminal sends image data with assigned time information to the server. The data is transmitted appropriately over the network. The input is image data containing time information, and the output is image data received by the server.

[0729] Step 11:

[0730] The server stores the received image data in a database. The image data is then shared among family members based on certain criteria. The input is the received image data, and the output is the record in the database and the sharing notification to the family.

[0731] (Application Example 1)

[0732] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0733] Recording and sharing a child's development is an important form of communication within the family, but the need for manual operation makes it a burden, especially for busy parents. There is also a need to record every moment of growth to create richer memories. Furthermore, there is a desire to gain a more comprehensive understanding of a child's development by utilizing the results of voice and motion analysis.

[0734] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0735] In this invention, the server includes means for analyzing voice and notifying when a specific event has occurred, means for detecting surrounding movements and triggering specific actions, and means for inputting analysis results based on voice and actions into an AI model. This makes it possible to automatically record moments of a child's growth even amidst the busyness of daily life, and to appropriately share those records with family.

[0736] "Children's voices" refers to speech or voices of minors detected through recording devices.

[0737] "Means of recording" refers to hardware or software for saving audio as recorded data.

[0738] "Means for adding and saving timestamps" refers to a system or method for adding date and time information to recorded data and saving it.

[0739] "Means for analyzing and extracting important statements" refers to technology that identifies and extracts important information from recorded audio data based on specific criteria.

[0740] "Means of sharing via communication" refers to methods and technologies for sharing data with other terminals or systems via a network.

[0741] "A means of analyzing audio and notifying users that a specific event has occurred" refers to a system that analyzes audio data and notifies the user when a specific event is detected.

[0742] "Means of detecting surrounding movements and triggering specific actions" refers to technologies that detect the movement of the environment or objects using sensors, and automatically initiate specific actions in response.

[0743] "Means of inputting into a generative AI model" refers to an interface used to input analysis results and data into an artificial intelligence model, allowing that model to perform further processing and predictions.

[0744] The system implementing this invention is based on a terminal installed in the home. The terminal is equipped with an audio recording device and an image capture device, which are used to record the child's voice and actions in real time. A timestamp is automatically added to the audio and images, and the data is temporarily stored in the terminal's storage device.

[0745] The server receives audio data transmitted from the terminal and performs analysis using speech recognition software. This analysis extracts important statements and events based on specific criteria and notifies the user as needed. Furthermore, it applies motion recognition algorithms to image data, detecting the child's movements and behavior, and inputting the results into a generating AI model. Software such as Vosk and OpenCV can be used for this process.

[0746] Notifications and analysis results from the server are shared with specific users via the network according to pre-configured conditions. This operation utilizes cloud services and communication platforms, enabling seamless data sharing among family members.

[0747] For example, if the device records a child's first words or a drawing, these recordings are immediately analyzed and processed by the server, and the family is notified. An example of a prompt for the generating AI model might be: "Describe how the robot behaves in the home. In particular, focus on how it captures the child's voice and movements and how it sends and shares that information to the server."

[0748] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0749] Step 1:

[0750] The device uses an audio recording device to detect and record children's speech in real time within the environment. It receives an audio signal as input, adds a timestamp to the audio data, and temporarily stores it in a memory device. It generates processed audio data as output.

[0751] Step 2:

[0752] The device uses an image capture device to detect the child's movements. It takes a video feed as input and applies a motion detection algorithm. If movement is detected, an image is captured and saved along with a timestamp. The output is image data captured triggered by movement.

[0753] Step 3:

[0754] The server receives audio data transmitted from the terminal. Using the processed audio data as input, speech recognition software performs analysis to extract specific important statements. In this process, text data is generated from the audio signal and output as the analysis result.

[0755] Step 4:

[0756] The server receives image data from the terminal and applies a motion recognition algorithm. The input is the captured image data, which analyzes the child's movements and generates information about those movements, which is then input into an AI model. This results in the analysis results being output.

[0757] Step 5:

[0758] The server checks pre-configured notification conditions based on analysis results obtained from audio and image data. The input consists of analysis results and condition settings. If a notification is deemed necessary, it sends a notification to the specified user. The output is a notification message.

[0759] Step 6:

[0760] Users receive notifications from the server and review data as needed. They can view or see audio and image data on the platform and access additional information based on the generated AI model. The output includes visual and audio feedback based on user selection.

[0761] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0762] This invention provides a method for recording a child's developmental stages using audio and images, and by combining this with an emotion engine, simultaneously evaluating the child's emotional state and sharing this information with family members who live far away. This allows parents to record and share their child's daily activities along with their emotions, thereby deepening communication within the family and among relatives. The following describes how this invention is implemented.

[0763] First, let's discuss voice recording. The device is equipped with voice detection capabilities to detect children's voices. When the user's child speaks, the device collects the sound through the microphone and begins recording. The recorded voice data is time-stamped, and then the emotion engine is used to analyze the emotion from the voice. This emotion data is temporarily stored in storage along with the voice data.

[0764] Next, the saved audio and emotional data are transmitted to a server via the network. The server processes the received audio using audio analysis tools, extracting important phrases and issues, and integrating and storing emotional data from an emotional engine. The analysis results are recorded in a database and are structured to make it easy to understand the emotional state of the child.

[0765] Next, let's discuss photography. The camera built into the device is equipped with object recognition capabilities to recognize objects held by the child. As soon as an object is recognized, the camera activates and photographs the child's movements and the object they are holding. The captured images are time-stamped, and an emotion engine is used to analyze the child's emotions from their facial expressions. This emotion data is stored in storage along with the image data.

[0766] Image data and emotional data are transmitted to a server via the network. The server stores this data in a database and uses a communication platform to automatically share it with specific users based on pre-configured conditions. This allows grandparents and other family members to experience the child's growth and emotional changes in real time.

[0767] This system automates the recording of a child's growth and emotions without requiring specific manual operations from the user, ensuring that important moments are never missed. These recordings can always be shared with the family, helping to foster intimate communication.

[0768] The following describes the processing flow.

[0769] Step 1:

[0770] The device constantly monitors ambient sounds using voice detection and detects children's voices. When a child's voice is detected, it starts recording using the built-in microphone.

[0771] Step 2:

[0772] The device adds a timestamp to the recorded audio data and simultaneously uses an emotion engine to analyze the emotions in the audio data. The resulting emotional information is integrated into the audio data and temporarily stored in storage.

[0773] Step 3:

[0774] The device sends stored voice data and emotional information to the server in conjunction with the timing of its network connection.

[0775] Step 4:

[0776] The server receives the transmitted audio data and emotional information, extracts important statements and phrases using an audio analysis tool, associates them with emotional information, and stores them in a database.

[0777] Step 5:

[0778] The device's camera uses object recognition technology to detect the object the child is holding. When an object is detected, the camera activates and takes an image of that moment.

[0779] Step 6:

[0780] The device adds a timestamp to the image it captures, and then uses an emotion engine to analyze the facial expressions in the image and determine the emotion. The obtained emotion information is integrated into the image data and saved to storage.

[0781] Step 7:

[0782] The device transmits stored image data and emotional information to the server via the network.

[0783] Step 8:

[0784] The server stores image data and emotional information in a database and automatically provides the data to specific users via a communication platform.

[0785] Step 9:

[0786] Users receive notifications, review data shared on social media and other platforms, and gain real-time insights into their child's emotional state and development.

[0787] (Example 2)

[0788] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0789] This invention aims to solve the problem of difficulty for families living in remote locations to share their children's daily growth and emotional changes in real time and to engage in deep communication. Furthermore, conventional technologies lacked a mechanism to effectively combine voice and image recording with emotion analysis, so the challenge is to provide a means to efficiently achieve this.

[0790] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0791] In this invention, the server includes means for analyzing and evaluating voice using an emotion processing device, a device for analyzing images and evaluating emotional states, and a device for transmitting the analysis results to a remote recipient using communication technology. This makes it possible to automatically and instantly share a child's growth and emotional changes with family members in remote locations using voice and image data.

[0792] A "sound detection and recording device" is a device equipped with the function of detecting sounds emitted from children or the environment using a microphone and recording them in digital format.

[0793] A "device for adding and saving date and time information" is a device that has the function of adding a timestamp to recorded or photographed data and recording it as digital data along with that information.

[0794] A "device for extracting important information" is a device equipped with the function of selecting necessary sentences and words from recorded audio data based on predetermined criteria.

[0795] A "device that evaluates emotional states using an emotion processing device" is a device that analyzes voice and image data and uses an emotion engine to identify and evaluate an individual's emotional state.

[0796] A "device that identifies and photographs objects" is a device that uses a camera and artificial intelligence to automatically take a photograph when it recognizes a specific object.

[0797] A "device for sharing with remote locations via communication technology" is a device that has the function of transmitting and sharing recorded or photographed data to other locations or devices using the internet or wireless communication networks.

[0798] A "device that provides information to specific recipients via an information transmission platform" is a device that uses dedicated software or network services to provide information to specific individuals or groups according to pre-configured conditions.

[0799] This invention is a system that uses a terminal and a server to acquire and analyze a child's voice and images, evaluate their emotional state, and share it with family members in a remote location. The detailed procedure for implementing this system is shown below.

[0800] First, the device is equipped with a microphone and camera, which are used to acquire audio and images. Voice detection software on the device detects the child's vocalizations in real time, starts recording the audio, and generates audio data. Next, the recorded audio data is analyzed by software using an emotion engine to evaluate the child's emotional state. This emotion analysis technology uses algorithms that analyze the tone, pitch, and speed of the voice.

[0801] In addition, the device's camera has a built-in object recognition function that automatically captures images when it recognizes an object held by the child. At this time, a timestamp is added to the recognized image, and it is processed by an emotion engine to analyze the child's facial expressions. This allows the child's emotional state to be evaluated from the image as well.

[0802] The analyzed audio and image data is sent from the device to the server via the network. The server stores this data in a database and shares it among family members based on specific conditions. This allows users to share their child's growth and daily emotional changes with their family in real time, without any mechanical or manual operation.

[0803] For example, consider a scenario where you record the moment a child rides a bicycle outdoors for the first time, capturing a video of them smiling and moving forward. An example of a prompt using a generative AI model would be, "Analyze the video of the child riding the bicycle and pedaling with a smile, and create a message to convey that emotional state to the family." This AI model automatically generates a message for the family that reflects the emotions, based on the recorded data.

[0804] By implementing this system, users can meticulously record their child's growth and share it with their family, including changes in their emotions, thereby promoting communication within the family.

[0805] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0806] Step 1:

[0807] The device continuously monitors ambient sounds using a microphone and detects children's vocalizations. When sound is detected, the device begins recording audio data. Real-time audio signals are input to the microphone. The device converts these audio signals into digital audio data, adds a timestamp, and stores it in temporary storage. An audio file with time information is generated as output.

[0808] Step 2:

[0809] The device processes audio data stored in its storage using an emotion engine. Audio files with time information are used as input. The emotion engine performs differential calculations to analyze emotional states from audio features such as volume, pitch, and speed, and outputs data indicating a specific emotion as the analysis result.

[0810] Step 3:

[0811] The device uses the camera's object recognition function to identify objects and actions held by the child. Camera footage is used as input for real-time analysis. When an object is recognized, the camera automatically captures an image and adds a timestamp to the image data. An image file with time information is generated as output.

[0812] Step 4:

[0813] The device processes image data using an emotion engine. An image file with time information is used as input. The emotion engine analyzes facial expressions and postures in the image and estimates the emotional state using face recognition technology. The output generates data indicating the emotional state read from the image.

[0814] Step 5:

[0815] The terminal transmits analyzed audio and image data to the server over the network. Sentimental data from the audio and images is used as input. A secure protocol is used to transfer the data to the server while maintaining data integrity. As output, integrated data is generated and stored on the server.

[0816] Step 6:

[0817] The server organizes and stores the received emotional data in a database. The input is the emotional data from transmitted audio and images. The data analysis results are categorized and managed in a format easily accessible to family members. The output is a dataset of visualized emotional states.

[0818] Step 7:

[0819] The server automatically shares data among family members according to specific conditions. Organized emotional data and family structure information are used as input. Leveraging a communication platform, pre-configured prompts trigger a generative AI model to create messages, which are then sent to specific recipients. Outputs include notifications and messages to family members that reflect their emotional state.

[0820] (Application Example 2)

[0821] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0822] The challenge is to provide a more effective in-home observation system that not only records a child's developmental stages but also captures their emotional state in detail, allowing for rich emotional sharing with family members living far away.

[0823] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0824] In this invention, the server includes means for detecting and recording a child's voice, means for assigning a timestamp to the recorded voice and saving it, and means for using an emotion analysis engine to analyze the emotional state from the voice and image data. This makes it possible to record a child's growth along with their emotions and share it with family in real time in an emotionally rich way.

[0825] "Methods for detecting and recording children's voices" refers to a function that recognizes surrounding sounds, selects voices emitted by specific individuals, and records them.

[0826] "Means for adding a timestamp to recorded audio and saving it" refers to a function that adds the recording time to acquired audio data and stores it together with identification information.

[0827] "Means using an emotion analysis engine that analyzes emotional states from audio and image data" refers to software that estimates the emotional changes of a specific person based on audio and video information and outputs it as data.

[0828] "Means of sharing via communication" refers to a system that enables information sharing by sending and receiving data between multiple devices and users via a network.

[0829] "Methods for detecting and photographing objects held by children" refers to a function that uses a visual sensor to recognize an object, records the movements of the person holding the object, and saves it as an image.

[0830] "A means of sharing video containing the results of emotional state analysis via communication" refers to a function that transmits video data with emotional information evaluated by an emotion analysis engine to other terminals via a network, thereby sharing the information.

[0831] "Means of providing information to specific users through a communication infrastructure in accordance with predetermined conditions" refers to a communication system for distributing selected data to specific recipients based on conditions set by the user.

[0832] This invention provides a system that uses a terminal operating in the home to record a child's voice and video and analyze their emotions. The terminal is equipped with a microphone and a camera, which detect specific sounds or movements. Voice is recorded in real time, saved in digital format, and time-stamped. An emotion analysis engine is applied to the recorded voice data to identify the emotional state at that moment. Similarly, the camera recognizes objects within its field of view, captures the moment, and saves it as image data. Emotion analysis is also performed on the image data, and the results are added.

[0833] Next, the terminal sends this data to the server over the network. The server stores the received data and transmits it to a specific user based on pre-configured conditions via appropriate communication methods.

[0834] For example, if a user is a relative living far away, they can share details about an event that the relative is particularly interested in, such as the child's participation in that event. Based on a prompt like, "Do you remember the smile on your child's face the day they took their first steps?", events and emotional moments identified through sentiment analysis are shared and provided along with actual video footage. In this way, parents and relatives can use this as a means to understand the details of life, including emotional changes, and deepen communication.

[0835] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0836] Step 1:

[0837] The device continuously monitors ambient sounds and detects children's voices and specific audio signals. The input is audio data, and the output is a digital recording of the detected sounds. This recording is temporarily stored in internal memory and timestamped.

[0838] Step 2:

[0839] This process applies an emotion analysis engine to recorded audio data to analyze the emotional state. The input is the audio data obtained in step 1, and the output is data containing the analyzed emotion information. This process uses an audio pattern recognition algorithm to analyze the tone and content of the audio.

[0840] Step 3:

[0841] The device's built-in camera operates, capturing images of the child's movements and objects they are holding. The input is video data obtained from the camera sensor, and the output is an image file with a timestamp. An object recognition algorithm is applied to detect specific objects.

[0842] Step 4:

[0843] The captured image data is also analyzed using an emotion analysis engine to estimate emotional states from facial expressions and other factors. The input is the image file obtained in step 3, and the output is image data with added emotional information. The analyzed emotional state is added to the video data as metadata.

[0844] Step 5:

[0845] The terminal transmits audio and image data, including the emotion data obtained in steps 2 and 4, to the server via the network. The input is this data, and the output is the adapted data transferred to the server. The data is securely uploaded using a transmission protocol.

[0846] Step 6:

[0847] The server stores the received data and delivers it to specific users based on defined conditions. The input is the data sent in step 5, and the output is the data delivered to the user's terminal. Data filtering and transmission are performed according to the prompts set by the user.

[0848] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the controlled object 443 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0849] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0850] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the robot 414.

[0851] Furthermore, the emotion identification model 59, acting as an emotion engine, may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to a specific mapping, which is an emotion map (see Figure 9). Similarly, the emotion identification model 59 may also determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[0852] Figure 9 shows an emotion map 400 in which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. The closer to the center of the concentric circles, the more primitive the emotions are located. Further out of the concentric circles, emotions representing states and actions arising from mental states are located. Emotion is a concept that includes feelings and mental states. On the left side of the concentric circles, emotions that are generally generated from reactions occurring in the brain are located. On the right side of the concentric circles, emotions that are generally induced by situational judgment are located. Above and below the concentric circles, emotions that are generally generated from reactions occurring in the brain and induced by situational judgment are located. In addition, the emotion of "pleasure" is located on the upper side of the concentric circles, and the emotion of "displeasure" is located on the lower side. Thus, in the emotion map 400, multiple emotions are mapped based on the structure in which emotions arise, and emotions that are likely to occur simultaneously are mapped close together.

[0853] These emotions are distributed at the 3 o'clock position on the Emotion Map 400, and usually fluctuate between feelings of security and anxiety. In the right half of the Emotion Map 400, situational awareness takes precedence over internal feelings, resulting in a calm impression.

[0854] The inside of the Emotion Map 400 represents inner thoughts, while the outside represents actions. Therefore, the further you go from the outside of the Emotion Map 400, the more visible (expressed in actions) your emotions become.

[0855] Here, human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. Similarly, in robots, cars, motorcycles, etc., emotions can be created based on various balances, such as posture and battery level. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. The emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on a system for analyzing brain physiological signals of speech emotion recognition and emotion, Tokushima University, doctoral dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map contains emotions belonging to a region called "response," where sensation is dominant. The right half of the emotion map contains emotions belonging to a region called "situation," where situational awareness is dominant.

[0856] The emotion map defines two emotions that promote learning. One is the emotion around the middle of the negative "repentance" and "reflection" on the situation side. In other words, it is when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is the emotion around the positive "desire" on the reaction side. In other words, it is when the robot has positive feelings such as "I want more" or "I want to know more."

[0857] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​representing each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple training data sets, which are combinations of user input and emotion values ​​representing each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions located close together have similar values, as shown in the emotion map 900 in Figure 10. Figure 10 shows an example where multiple emotions such as "reassured," "calm," and "confident" have similar emotion values.

[0858] The above description primarily focuses on the functions of the data processing device 12 in relation to this disclosure. However, the system related to this disclosure is not necessarily implemented on a server. The system related to this disclosure may be implemented as a general information processing system. This disclosure may be implemented, for example, as a software program that runs on a personal computer or as an application that runs on a smartphone. The method related to this disclosure may be provided to users in SaaS (Software as a Service) format.

[0859] In the above embodiment, an example was given in which a specific process is performed by a single computer 22. However, the technology of this disclosure is not limited thereto, and a distributed processing of the specific process may be performed by multiple computers, including computer 22. For example, a data generation model 58 may be provided in an external device of the data processing device 12, and the external device may generate data according to the input data.

[0860] In the above embodiment, an example was given in which the specific processing program 56 is stored in the storage 32, but the technology of this disclosure is not limited thereto. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-temporary storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-temporary storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes specific processing according to the specific processing program 56.

[0861] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[0862] Furthermore, it is not necessary to store the entirety of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store the entirety of the specific processing program 56 in the storage 32; it is acceptable to store only a portion of the specific processing program 56.

[0863] The following types of processors can be used as hardware resources to perform specific processing. Examples of processors include a CPU, a general-purpose processor that functions as a hardware resource to perform specific processing by executing software, i.e., a program. Other examples of processors include dedicated electrical circuits, such as FPGAs (Field-Programmable Gate Arrays), PLDs (Programmable Logic Devices), or ASICs (Application Specific Integrated Circuits), which have circuit configurations specifically designed to perform specific processing. All of these processors have built-in or connected memory, and all of them perform specific processing by using memory.

[0864] The hardware resource that performs a specific process may consist of one of these various processors, or it may consist of a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Alternatively, the hardware resource that performs a specific process may consist of a single processor.

[0865] Examples of configurations using a single processor include, firstly, a configuration in which one or more CPUs and software are combined to form a single processor, and this processor functions as a hardware resource that performs a specific process. Secondly, there is a configuration using a processor that realizes the functions of the entire system, including multiple hardware resources that perform a specific process, on a single IC chip, as exemplified by SoCs (System-on-a-chip). In this way, a specific process is realized using one or more of the above types of processors as hardware resources.

[0866] Furthermore, the hardware structure of these various processors can more specifically utilize electrical circuits that combine circuit elements such as semiconductor devices. Also, the specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps can be deleted, new steps added, or the processing order rearranged, as long as it does not deviate from the main purpose.

[0867] The descriptions and illustrations presented above are detailed explanations of the technical aspects of this disclosure and are merely examples of the technical aspects. For example, the above descriptions of the structure, function, operation, and effect are examples of the structure, function, operation, and effect of the technical aspects of this disclosure. Therefore, it goes without saying that you may delete unnecessary parts, add new elements, or replace elements in the descriptions and illustrations presented above, as long as you do not deviate from the essence of the technical aspects of this disclosure. Furthermore, in order to avoid confusion and facilitate understanding of the technical aspects of this disclosure, explanations of common technical knowledge and the like that do not require special explanation to enable the implementation of the technical aspects of this disclosure have been omitted from the descriptions and illustrations presented above.

[0868] All documents, patent applications, and technical standards described herein are incorporated by reference to the same extent as if each individual document, patent application, and technical standard were specifically and individually noted to be incorporated by reference.

[0869] The following is further disclosed regarding the embodiments described above.

[0870] (Claim 1)

[0871] A means of detecting and recording children's voices,

[0872] A means of adding a timestamp to recorded audio and saving it,

[0873] A means of analyzing saved audio and extracting important statements,

[0874] A means of sharing analysis results via communication,

[0875] A system that includes this.

[0876] (Claim 2)

[0877] A method for recognizing and photographing an object held in a child's hand,

[0878] A means of adding a timestamp to a captured image and saving it,

[0879] A means of sharing saved images via communication,

[0880] The system according to claim 1, including the following:

[0881] (Claim 3)

[0882] A means of providing recorded audio and captured images to specific users via a communication platform based on pre-set conditions,

[0883] The system according to claim 1, including the following:

[0884] "Example 1"

[0885] (Claim 1)

[0886] A means for acquiring sound to detect and collect children's voices,

[0887] A storage method for temporarily storing collected audio with time information added,

[0888] A data analysis method for analyzing stored audio and selecting specific phrases or words,

[0889] An information sharing means for storing the analyzed results on a recording medium and sharing them with other users by communicating under predetermined conditions,

[0890] A system that includes this.

[0891] (Claim 2)

[0892] A visual acquisition method for recognizing and photographing an object held by a child,

[0893] A means for saving images with time information attached,

[0894] Information distribution means for sharing saved images with other users by communicating under predetermined conditions,

[0895] The system according to claim 1, including the following:

[0896] (Claim 3)

[0897] A selective transmission means for providing analyzed audio and captured images to a specific user via a communication device based on pre-set conditions,

[0898] The system according to claim 1, including the following:

[0899] "Application Example 1"

[0900] (Claim 1)

[0901] A means of detecting and recording children's voices,

[0902] A means of adding a timestamp to recorded audio and saving it,

[0903] A means of analyzing saved audio and extracting important statements,

[0904] A means of sharing analysis results via communication,

[0905] A means of analyzing audio and notifying that a specific event has occurred,

[0906] A means of detecting surrounding movements and triggering specific actions,

[0907] A system that includes this.

[0908] (Claim 2)

[0909] A method for recognizing and photographing an object held in a child's hand,

[0910] A means of adding a timestamp to a captured image and saving it,

[0911] A means of sharing saved images via communication,

[0912] A means of detecting surrounding movement and triggering a shooting action,

[0913] The system according to claim 1, including the following:

[0914] (Claim 3)

[0915] A means of providing recorded audio and captured images to specific users via a communication platform based on pre-set conditions,

[0916] A means of inputting analysis results based on voice and actions into an AI model,

[0917] The system according to claim 1, including the following:

[0918] "Example 2 of combining an emotion engine"

[0919] (Claim 1)

[0920] A device that detects and records sound,

[0921] A device that adds date and time information to recorded audio and saves it,

[0922] A device that analyzes stored audio and extracts important information,

[0923] A device that evaluates the emotional state of analyzed speech using an emotion processing device,

[0924] A device that analyzes captured images and evaluates emotional states,

[0925] A device that transmits analysis results to a remote recipient via communication technology,

[0926] A system that includes this.

[0927] (Claim 2)

[0928] A device that identifies and photographs objects,

[0929] A device that adds date and time information to captured images and saves them,

[0930] A device that shares stored images with a remote location via communication technology,

[0931] The system according to claim 1, including the following:

[0932] (Claim 3)

[0933] A device that provides recorded audio and captured images to specific recipients via an information transmission platform according to pre-set criteria,

[0934] The system according to claim 1, including the following:

[0935] "Application example 2 when combining with an emotional engine"

[0936] (Claim 1)

[0937] A means of detecting and recording children's voices,

[0938] A means of adding a timestamp to recorded audio and saving it,

[0939] A means of analyzing stored audio and extracting important expressions,

[0940] A means of using an emotion analysis engine that analyzes emotional states from audio and image data,

[0941] A means of sharing information, including analyzed emotional data, via communication,

[0942] A system that includes this.

[0943] (Claim 2)

[0944] A method for detecting and photographing objects held in a child's hand,

[0945] A means of adding a timestamp to the captured video and saving it,

[0946] A means of sharing video containing the results of an analysis of emotional state via communication,

[0947] The system according to claim 1, including the following:

[0948] (Claim 3)

[0949] A means of providing recorded audio and captured video to specific users via a communication infrastructure, in accordance with predetermined conditions,

[0950] The system according to claim 1, including the following: [Explanation of Symbols]

[0951] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Devices 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robots< / url:> < / url:> < / url:> < / url:>

Claims

1. A means of detecting and recording children's voices, A means of adding a timestamp to recorded audio and saving it, A means of analyzing saved audio and extracting important statements, A means of sharing analysis results via communication, A system that includes this.

2. A method for recognizing and photographing an object held in a child's hand, A means of adding a timestamp to a captured image and saving it, A means of sharing saved images via communication, The system according to claim 1, including the following:

3. A means of providing recorded audio and captured images to specific users via a communication platform based on pre-set conditions, The system according to claim 1, including the following:

Images