system

Abstract

A system is provided.SOLUTION: A system comprising: means for recognizing an utterance of a specific person, assigning a time stamp, and recording voice data; means for recognizing an object held in a hand of the specific person, assigning a time stamp, and recording image data; and means for storing the recorded voice data and image data in a database and automatically sharing the voice data and the image data through an SNS.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] Parenting generations are busy, making it difficult to record the moment their child speaks for the first time or brings home something new. Furthermore, families living far away are less able to keep track of their children's growth in real time. To solve these issues, there is a need for a system that can automatically record a child's speech and belongings, and share them via social media with a timestamp. [Means for solving the problem]

[0005] The present invention solves the above-mentioned problems with a system that includes the following means: It provides a means for recognizing the speech of a specific person, recording the audio data with a timestamp, and a means for recognizing an object held by a specific person, recording image data with a timestamp. It also provides a means for storing the recorded audio and image data in a database and automatically sharing it via social media, allowing users to record every moment of their child's growth and share it with their family. It also provides an even more versatile recording method by using generative AI to create diary-style documents based on the stored audio and image data, which can be displayed and shared via social media or dedicated applications.

[0006] "Specific person" refers to a pre-defined individual who is the subject of voice or image recognition.

[0007] "Speech" means human vocalizations, especially those that are recognized as words or words.

[0008] A "timestamp" refers to adding information to data (audio or image) that indicates the date and time it was acquired.

[0009] "Voice data" refers to data that is a digital recording of human speech.

[0010] The "item in hand" is an item held in the hand of a specific person, and is the subject of image recognition.

[0011] "Image data" refers to data that records images captured by a camera or sensor in digital format.

[0012] "Database" refers to a system that can efficiently manage and search stored audio and image data.

[0013] "SNS" is an abbreviation for social networking service and refers to a platform for sharing information over the Internet.

[0014] "Generative AI" refers to artificial intelligence technology that automatically generates new documents and information based on stored data.

[0015] A "diary-style document" refers to a document that compiles events and records over a certain period of time in the form of a diary.

[0016] "Dedicated application" refers to software specifically designed for this system, and is an application for displaying and sharing data. [Brief explanation of the drawings]

[0017] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11]FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0018] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0019] First, the terms used in the following description will be explained.

[0020] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0021] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0022] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0023] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0024] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0025] [First embodiment]

[0026] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0027] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0028] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0029] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0030] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0031] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0032] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0033] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0034] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0035] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0036] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0037] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0038] The present invention is a system that automatically records what a specific person is saying and what they are holding, and shares that information via social networking services. A specific embodiment of this system will be described below.

[0039] This system consists of a device (e.g., a smartphone or home camera) and a server. It recognizes the user's (child's) speech and the objects they are holding, records them with a timestamp, and automatically shares them via social media.

[0040] Audio recording

[0041] 1. Audio monitoring and recognition

[0042] The device is equipped with a voice recognition function that constantly monitors the sounds around the user (child). The voice recognition algorithm detects specific utterances (e.g., "Mom," "Dad," etc.).

[0043] 2. Recording audio data

[0044] When the device recognizes a user's speech, it records that portion of the audio and adds a timestamp to it, which clearly indicates the date and time of the recording.

[0045] 3. Send to the server

[0046] The device sends the recorded voice data to a server, which stores the received voice data in a database and checks whether the same utterance has been recorded in the past.

[0047] 4. Sharing on social media

[0048] If the server recognizes the utterance as new, it will generate a message to share on SNS (e.g., LINE) and automatically send it. The shared content will include a timestamp, indicating the date and time of the utterance.

[0049] Image recording

[0050] 1. Image monitoring and recognition

[0051] The device (e.g., a home camera) monitors the user's (child's) movements and uses image recognition to identify the object being held.

[0052] 2. Recording image data

[0053] When the device recognizes a new item, it takes a picture and adds a timestamp to it. For example, if a user brings home a new toy, it will record an image of that exact moment.

[0054] 3. Send to the server

[0055] The device sends the captured image data to a server, which stores the received image data in a database and checks whether the same object has been recorded previously.

[0056] 4. Sharing on social media

[0057] If the server recognizes the item as new, it will automatically generate and send a message to share on social media, including a timestamp to indicate when the item was picked up.

[0058] Diary creation using generative AI

[0059] 1. Data Collection

[0060] The server extracts audio data and image data for a certain period (e.g., one week) from the database.

[0061] 2. Diary-style document generation

[0062] The server uses generative AI to create diary-style documents based on the extracted data, transcribes audio data, and provides explanatory text for image data.

[0063] 3. Share your diary

[0064] The created diary can be displayed and shared with users and their families via social media or a dedicated application, allowing users to check their child's growth record in diary format.

[0065] Specific examples

[0066] Audio recording example

[0067] When a user (child) says "Mom," the device recognizes this and records it. The recorded voice data is given a timestamp, such as "October 1, 2023, 10:30." The server receives this data, compares it with its existing database, and if it determines that it has not been recorded, shares the content on a social networking site (e.g., Line) with the message, "[October 1, 2023, 10:30] My child said 'Mom.'"

[0068] Image recording example

[0069] When a user (child) brings home a new toy, the device (home camera) recognizes this and takes a picture. The image data is given a timestamp, such as "October 1, 2023, 14:30." The server receives this data, compares it with its existing database, and if it determines that it has not been recorded, it shares the message on social media, saying, "[October 1, 2023, 14:30] Your child has brought home a new toy."

[0070] This system allows for real-time recording of a child's growth and automatically sharing it. It also uses generative AI to create diary-style documents based on the saved data, which can be shared with family members, providing a more detailed record of a child's growth.

[0071] The processing flow will be explained below.

[0072] Audio recording

[0073] Step 1:

[0074] The device (smartphone app) activates its voice recognition module and constantly monitors the sounds around the user (child), acquiring the voice data stream and preparing to analyze the voice patterns in real time.

[0075] Step 2:

[0076] The device detects a specific speech pattern, for example, "Mama," and the speech recognition module trims that portion of the audio and recognizes it as speech.

[0077] Step 3:

[0078] The device temporarily stores the recognized speech in memory and timestamps the recording with the current date and time, ensuring an accurate recording of the speech.

[0079] Step 4:

[0080] The device sends time-stamped audio data to the server, which then encrypts it and transfers it securely over the network to the server.

[0081] Step 5:

[0082] The server stores the received voice data in a database, and an algorithm runs to check whether the same utterance exists in the database by comparing it with previous records.

[0083] Step 6:

[0084] When the server recognizes a new utterance, it converts the information into a message for use on social media. For example, it generates a message in the format: "[October 1, 2023, 10:30] My child said 'Mom.'"

[0085] Step 7:

[0086] The server automatically sends the generated message to an SNS (e.g., Line), and the recipient can receive the message in the specified account.

[0087] Image recording

[0088] Step 1:

[0089] The device (home camera) activates the image recognition module and constantly monitors the user's (child's) movements, preparing to analyze the camera footage in real time to detect objects being held.

[0090] Step 2:

[0091] The device recognizes that the user (child) is holding a new object, and the image recognition module identifies the object and captures an image.

[0092] Step 3:

[0093] The device temporarily stores the captured image in memory and timestamps the image data with the current date and time, ensuring an accurate record of the photo.

[0094] Step 4:

[0095] The device sends image data with a timestamp to the server, where it is encrypted and securely transferred to the server over the network.

[0096] Step 5:

[0097] The server stores the received image data in a database, where an algorithm runs to check against previous databases to see if the same object has been recorded before.

[0098] Step 6:

[0099] When the server recognizes a new item, it converts that information into a message for social media, such as "My child brought home a new toy [October 1, 2023, 14:30]."

[0100] Step 7:

[0101] The server automatically sends the generated message to an SNS (e.g., Line), and the recipient can receive the message in the specified account.

[0102] Diary creation using generative AI

[0103] Step 1:

[0104] The server extracts audio and image data for a certain period (e.g., one week) from the database, and the extracted data is processed by the AI ​​module.

[0105] Step 2:

[0106] The server uses generative AI to create diary-style documents based on the extracted data, automatically transcribes audio data, and provides explanatory text for image data.

[0107] Step 3:

[0108] The server uploads the created diary to social networking sites or dedicated applications, allowing users and their families to view their child's growth record in diary format.

[0109] Step 4:

[0110] Users can view and share the diary entries they create with their families via social media or a dedicated application. This information is organized with timestamps and can be used as an accurate record of their child's growth.

[0111] Example 1

[0112] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0113] In recent years, there has been a growing need to record children's developmental progress at home and share it with family members. However, conventional methods require time-consuming manual recording and sharing, and often lack real-time performance and accuracy. Furthermore, managing images and audio individually is cumbersome, creating a need for a unified recording method. Furthermore, recorded data needs to be displayed in an easily understandable and organized format when reviewing it later.

[0114] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0115] In this invention, the server includes means for recognizing the speech of a specific person using voice recognition technology, recording the voice data with a timestamp, means for recognizing an object held by the specific person using image recognition technology, recording image data with a timestamp, and means for saving the recorded voice data and image data in a database and automatically sharing them via SNS. This makes it possible to record a child's growth process in real time and automatically share it.

[0116] "Voice recognition technology" is a technology that identifies the speech of a specific person in real time and converts it into text format.

[0117] A "timestamp" is information indicating the date and time when data was recorded, and includes the date and time.

[0118] "Audio data" refers to recorded audio information stored in digital format.

[0119] "Image recognition technology" is a technology that uses computer vision algorithms to automatically identify objects and features in images.

[0120] "Image data" refers to photographs or videos taken by a device such as a camera and stored in digital format.

[0121] A "database" is a system that structures and stores information such as audio data and image data, and allows for efficient management and retrieval.

[0122] "SNS" is an abbreviation for social networking service, a platform that supports communication and information sharing between users via the Internet.

[0123] "Generative AI technology" is an artificial intelligence technology that learns from large amounts of data and generates new data and content.

[0124] A "diary-style document" is a document that describes events that occurred during a specific period in chronological order.

[0125] A "specialized application" is software designed for a specific function or purpose, in this case, to display and share growth records.

[0126] This invention is a system that automatically records what a specific person is saying and what they are holding, and shares that information via social networking services. This system consists of a device (e.g., a smartphone or home camera) and a server. It recognizes what the user (child) is saying and what they are holding, records it with a timestamp, and automatically shares it on social networking services.

[0127] Audio recording

[0128] 1. Audio monitoring and recognition

[0129] The device uses voice recognition technology to constantly monitor the sounds around the user (child). Specifically, it uses the microphone on a smartphone or home camera and uses a voice recognition algorithm such as "Google Cloud Speech-to-Text" to detect specific utterances (e.g., "Mom" or "Dad") in real time.

[0130] 2. Recording audio data

[0131] When the device recognizes a specific utterance, it records that portion of the audio and adds a timestamp using Python's datetime module. For example, the recording date and time is added to the data in the format "October 1, 2023, 10:30." This data is temporarily stored on the device.

[0132] 3. Send to the server

[0133] The device sends the recorded voice data to the server via an HTTP POST request. When the server receives this data, it stores it in a database (e.g., MySQL) and compares it with past data to determine whether the utterance is new.

[0134] 4. Sharing on social media

[0135] If the server recognizes the utterance as new, it uses the API of a social networking service (e.g., Line) to generate and send a message stating, "[October 1, 2023, 10:30 AM] My child said 'Mom.'" This allows the content of the utterance to be shared with family members in real time.

[0136] Image recording

[0137] 1. Image monitoring and recognition

[0138] The device (e.g., a home camera) uses image recognition technology (e.g., OpenCV, TensorFlow) to constantly monitor the user's (child's) movements, thereby identifying the object being held in real time.

[0139] 2. Recording image data

[0140] When the device recognizes a new item, it takes a photo of that moment and adds a timestamp (e.g., "October 1, 2023, 14:30"), which is also temporarily stored on the device.

[0141] 3. Send to the server

[0142] The device sends the captured image data to the server using an HTTP POST request, which stores the data in a database and similarly compares it with past data to determine whether the item is new.

[0143] 4. Sharing on social media

[0144] If the server recognizes the item as new, it generates and sends a message to the social networking site saying, "[October 1, 2023, 14:30] Your child has brought home a new toy."

[0145] Diary creation using generative AI

[0146] 1. Data Collection

[0147] The server extracts audio and image data for a certain period (e.g., one week) from the database, and stores this data temporarily for diary generation.

[0148] 2. Diary-style document generation

[0149] The server uses generative AI technology (e.g., OpenAI GPT-4) to generate diary-style documents based on the extracted data. Audio data is transcribed and image data is annotated with explanatory text. This process is performed by inputting prompts to the generative AI model.

[0150] Example prompt sentence:

[0151] Create a diary of your child's growth based on one week's worth of audio and video data. Here is an example of the data:

[0152] Audio data:

[0153] 1. [October 1, 2023 10:30] "Mom"

[0154] 2. [October 2, 2023 11:00] "Dad"

[0155] Image data:

[0156] 1. [October 1, 2023 14:30] A user has come home with a new toy.

[0157] 2. [October 2, 2023, 15:00] A photo of a user playing in a park

[0158] Based on this data, please write a detailed diary of your child's growth.

[0159] 3. Share your diary

[0160] The created diary can be displayed and shared via social media or a dedicated application. The server converts the created diary into PDF or HTML format and generates a sharing link to provide to the user or their family.

[0161] As described above, the embodiments of the present invention have been specifically described, which allow users to seamlessly manage and share growth records, thereby stimulating communication within the home.

[0162] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0163] Processing Steps

[0164] Audio recording

[0165] Step 1: Audio monitoring and recognition

[0166] The device uses the microphone of a smartphone or home camera to constantly monitor the sounds around the user (child). It uses voice recognition technology (e.g., Google Cloud Speech-to-Text) to detect specific utterances (e.g., "Mom" or "Dad") in real time.

[0167] Input: Ambient sound captured by the microphone

[0168] Processing: Speech recognition algorithms detect specific utterances in real time

[0169] Output: A trigger event that a specific utterance was detected

[0170] Step 2: Recording audio data

[0171] When the device recognizes a specific utterance, it records that portion of the audio and adds a timestamp. Using the Python datetime module, it adds the recording date and time to the data in the format "October 1, 2023, 10:30." The audio data is temporarily stored on the device.

[0172] Input: Trigger event, audio data

[0173] Processing: Recording and time stamping audio data

[0174] Output: Audio data with timestamps

[0175] Step 3: Send to the server

[0176] The device sends the recorded voice data to the server via an HTTP POST request. The data is converted to JSON format and sent to the server's API endpoint. The server receives this data and stores it in a database (e.g., MySQL). A Python script is used to compare it with existing data and determine whether it is a new utterance.

[0177] Input: Time-stamped audio data

[0178] Processing: Convert to JSON format, send via HTTP POST request, save to database, and collate data

[0179] Output: New utterance determination result

[0180] Step 4: Share on social media

[0181] If the server determines that the utterance is new, it uses the API of the social networking service (e.g., Line) to generate and send a shared message. For example, it might send a message to the social networking service saying, "[October 1, 2023, 10:30] My child said 'Mom.'"

[0182] Input: New utterance determination result

[0183] Processing: Message creation and sending via SNS API

[0184] Output: Message sent to SNS

[0185] Image recording

[0186] Step 1: Image monitoring and recognition

[0187] The device (e.g., a home camera) constantly monitors the user's (child's) movements and uses image recognition technology (e.g., OpenCV, TensorFlow) to identify the object being held in real time.

[0188] Input: Image data obtained from the camera

[0189] Processing: Object recognition using image recognition algorithms

[0190] Output: Trigger event that new possession is recognized

[0191] Step 2: Recording image data

[0192] When the device recognizes a new item, it takes an image of that moment and uses the Python datetime module to add a timestamp (e.g., "October 1, 2023, 14:30"), which is also temporarily stored on the device.

[0193] Input: Trigger event, image data

[0194] Processing: Taking an image and adding a timestamp

[0195] Output: Image data with time stamp

[0196] Step 3: Send to the server

[0197] The device sends the captured image data to the server using an HTTP POST request. The data is converted to JSON format and sent to the server's API endpoint. The server receives this data and stores it in a database. It compares it with existing image data to determine if it is a new item.

[0198] Input: Image data with timestamp

[0199] Processing: Convert to JSON format, send via HTTP POST request, save to database, and collate data

[0200] Output: New possession determination result

[0201] Step 4: Share on social media

[0202] If the server determines that the item is new, it uses the SNS API to generate and send a sharing message. For example, it might send a message to the SNS saying, "[October 1, 2023, 14:30] Your child has brought home a new toy."

[0203] Input: New possession determination result

[0204] Processing: Message creation and sending via SNS API

[0205] Output: Message sent to SNS

[0206] Diary creation using generative AI

[0207] Step 1: Data collection

[0208] The server extracts audio and image data for a certain period (e.g., one week) from the database, and stores this data temporarily for diary generation.

[0209] Input: Audio data and image data in the database

[0210] Processing: Data extraction, temporary storage

[0211] Output: Data for diary generation

[0212] Step 2: Creating a diary-style document

[0213] The server uses generative AI technology (e.g., OpenAI GPT-4) to generate diary-style documents based on the extracted data. Audio data is transcribed and image data is annotated with explanatory text. This process is performed by inputting prompts into the generative AI model.

[0214] Input: Data for diary generation, prompt text

[0215] Processing: Generative AI models generate text, transcribe audio data, and add descriptions to image data

[0216] Output: Diary-style document

[0217] Step 3: Share your diary

[0218] The created diary can be displayed and shared via social media or a dedicated application. The server converts the created diary into PDF or HTML format and generates a sharing link to provide to the user or their family.

[0219] Input: diary-style document

[0220] Processing: Convert to PDF or HTML format, generate a link for sharing, send via social media or dedicated applications

[0221] Output: Shareable link, diary display on social media or dedicated application

[0222] This is the specific processing flow of the program for this system, which allows users to automatically manage and share their child's growth record in real time through audio and images.

[0223] (Application example 1)

[0224] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0225] In traditional brick-and-mortar stores, it is possible for customers to talk about products and check the products they have picked up, but there is a lack of a way to properly record customer behavior and the content of their conversations and provide personalized information in real time. It is also difficult to store customer consumption behavior in a database and use it for effective marketing or individualized support. For these reasons, a new system was needed to improve the customer experience and streamline store operations.

[0226] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0227] In this invention, the server includes means for recognizing the speech of a specific person, assigning a timestamp and recording the speech data, means for recognizing an object held by a specific person, assigning a timestamp and recording image data, means for saving the recorded speech data and image data in a database and automatically sharing them via social networking services, means for displaying the speech data and image data on an information display device in real time, and means for generating diary-style documents using a generative AI model based on past records of speech data and image data. This makes it possible to record customer speech and behavior in real time and provide personalized information, thereby improving the efficiency of store operations and the customer experience.

[0228] "Specified Person" refers to the person who will be monitored and recorded by the system.

[0229] "Utterance" refers to the sound produced by a particular person.

[0230] A "timestamp" refers to information indicating the date and time when audio or image data was recorded.

[0231] "Audio data" refers to an audio file that records the speech of a specific person.

[0232] "Handheld object" refers to an object held in the hand of a particular person.

[0233] "Image data" refers to an image file that captures an object being held by a specific person.

[0234] "Database" refers to a system for storing and managing audio and image data.

[0235] "SNS" is an abbreviation for social networking service, and refers to an online platform where users can share information with each other.

[0236] "Automatically sharing" means that the system automatically sends data to social media sites, etc., without the need for human operation.

[0237] "Information display device" refers to a device for visually displaying audio data and image data.

[0238] A "generative AI model" refers to an algorithm that uses machine learning techniques to generate sentences or documents from input data.

[0239] A "diary-style document" refers to a file that records events in chronological order and is organized as text.

[0240] "Personalized information" refers to information and services that are customized for a specific individual.

[0241] "Recording" refers to saving audio data or image data.

[0242] "Displayed in real time" means that data is displayed as soon as it is generated, with little delay.

[0243] The system that realizes this application example has the functions of recognizing the speech of a specific person, recording the voice data with a timestamp, recognizing the object held by the specific person, recording the image data with a timestamp, and storing the recorded voice and image data in a database, which can then be automatically shared via social networking sites.

[0244] Hardware and software used

[0245] This system uses the following hardware and software:

[0246] Microphone: A device for collecting speech from a specific person. It is built into smart glasses.

[0247] Camera: A device that takes pictures of objects held by a specific person. It is built into smart glasses.

[0248] Smart glasses: Information display devices that display speech and image data in real time.

[0249] Server: A computer system that processes and stores data and manages sharing to social media.

[0250] For example, we use the Python speech_recognition library for speech recognition, the cv2 (OpenCV) and pytesseract (OCR) libraries for image recognition, and the requests library for data processing and transmission.

[0251] Data processing and calculation

[0252] 1. Speech Recognition:

[0253] A user wears smart glasses and the microphone collects the speech of a specific person.

[0254] The audio data collected by the microphone is converted to text using the speech_recognition library, which is then time-stamped and the resulting audio data is generated.

[0255] The generated voice data is immediately sent to the server and stored in a database.

[0256] 2. Image Recognition:

[0257] A camera captures an image of an object being held by a particular person.

[0258] The captured image data is then extracted using the cv2 and pytesseract libraries, which then timestamp the image and generate the resulting image data.

[0259] The generated image data is immediately sent to a server and stored in a database.

[0260] 3. Save to database and share on social media:

[0261] The server stores the received audio data and image data in a database.

[0262] New data stored in the database is automatically shared via social media. For example, if a voice utterance such as "What do you think of this product?" is registered, that information will be posted to social media in real time.

[0263] 4. Generative AI diary creation:

[0264] The server extracts past audio and image data for a certain period of time (e.g., one week) and generates a diary-style document using a generative AI model.

[0265] The created diary can be shared with users via social media or dedicated applications.

[0266] Specific examples

[0267] For example, if a customer wearing smart glasses walks through a store and says, "What are the characteristics of this wine?", the microphone collects the voice and the voice recognition software converts it into text. This voice data is time-stamped and sent to the server. In addition, the camera also captures information about the wine bottle the customer picks up, and the image recognition software analyzes the content and sends it to the server.

[0268] The collected data is processed on the server and automatically shared on social media in the form of, for example, "[Date and time] A customer said, 'What are the features of this wine?' Wine information: Red wine, 750ml." At the end of the week, the generative AI model generates a diary-style document based on past data, which is shared via social media and dedicated applications with content such as, "This week's new customer interest product was red wine."

[0269] Prompt Sentence Examples

[0270] What is special about this wine?

[0271] "Tell me more about this product."

[0272] - "What are the new customer interest products this week?"

[0273] This process allows the system to record customer utterances and behavior in real time and provide personalized information, making it a powerful tool for improving customer experience and streamlining store operations.

[0274] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0275] Step 1:

[0276] The microphone in the smart glasses collects the user's speech. The microphone constantly monitors the surrounding sound of a specific person and captures voice data. The input is voice waveform data, and the output is an audio file.

[0277] Step 2:

[0278] Convert audio files to text using the speech_recognition library. Audio data is input and text data is output. Specifically, the speech recognition algorithm analyzes the waveform of the audio and converts it into meaningful words.

[0279] Step 3:

[0280] A timestamp is added to the converted text data. The input is text data obtained by speech recognition, and the current date and time are added as a timestamp. The output is text data with a timestamp.

[0281] Step 4:

[0282] Sends text data with a timestamp to the server. Audio data and a timestamp are input and sent to the server as an HTTP request. The output is the data received on the server side.

[0283] Step 5:

[0284] The server stores the received audio data in a database. The input is time-stamped audio data that is inserted into a database table. The output is an audio recording stored in the database.

[0285] Step 6:

[0286] The camera in the smart glasses takes a picture of the object held by the user. The camera is constantly capturing images of the surroundings, and the input is image data. The output is an image file.

[0287] Step 7:

[0288] Extract text information from an image using the cv2 and pytesseract libraries. The input is image data, and the output is extracted text information. Specifically, OCR technology analyzes the text in the image and outputs it as a string of characters.

[0289] Step 8:

[0290] A timestamp is added to the converted image text data. The input is text data obtained by OCR, and the current date and time are added as a timestamp. The output is text data with a timestamp.

[0291] Step 9:

[0292] Send image text data with a timestamp to the server. Image data and a timestamp are input and sent to the server as an HTTP request. The output is the data received on the server side.

[0293] Step 10:

[0294] The server stores the received image text data in a database. The input is time-stamped image data that is inserted into a database table. The output is an image record stored in the database.

[0295] Step 11:

[0296] The server automatically posts new data to SNS. The input is new data stored in the database, and the server calls the SNS API to generate a message to post. The output is the message posted to SNS.

[0297] Step 12:

[0298] Periodically, the server extracts historical audio and image data. The input is the past week's worth of data from the database, and the output is an extracted data set.

[0299] Step 13:

[0300] The server sends the extracted data to a generative AI model to generate a diary-style document. The input is data from the past week, and the AI ​​model generates sentences. The output is a diary-style document.

[0301] Step 14:

[0302] The server shares the created diary through social networking sites or dedicated applications. The input is the created diary document, which is sent to the social networking site API or application API. The output is the diary data that can be accessed by users.

[0303] In this way, it is possible to record the user's speech and actions in real time and provide personalized information.

[0304] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0305] The present invention combines an emotion engine with a system that automatically records the speech and objects held by a specific person and shares them via social networking services. A specific embodiment of this system is described below.

[0306] This system consists of a device (e.g., smartphone, home camera) and a server. It recognizes the user's (child's) speech and the objects they are holding, records them with a timestamp, and automatically shares them via social media. It also uses an emotion engine to recognize the user's emotional state, and records and shares it.

[0307] Audio recording

[0308] 1. Audio monitoring and recognition

[0309] The device is equipped with a voice recognition function that constantly monitors the sounds around the user (child). A voice recognition algorithm detects specific speech.

[0310] 2. Recording audio data

[0311] When the device recognizes the user's speech, it records that part of the audio, adds a timestamp, and temporarily saves the recorded data.

[0312] 3. Send to the server

[0313] The device sends the recorded voice data to a server, which stores the received voice data in a database and checks whether the voice is new by comparing it with past data.

[0314] 4. Analysis by Emotion Engine

[0315] The server analyzes the voice data with an emotion engine to identify the user's emotional state (e.g., joy, sadness, anger, etc.), and this emotion data is also stored along with the voice data.

[0316] 5. Sharing on social media

[0317] If the server recognizes the utterance as new, it generates a message containing emotional data for the social networking site. For example, it generates a message in the format "[October 1, 2023, 10:30] My child said 'Mom' (joy)" and automatically sends it to the social networking site.

[0318] Image recording

[0319] 1. Image monitoring and recognition

[0320] The device (home camera) activates image recognition and monitors the user's (child's) movements. It analyzes the camera footage to detect what is being held in the user's hand.

[0321] 2. Recording image data

[0322] When the device detects a new object, it takes a photo of that moment, adds a timestamp to it, and then temporarily stores this data.

[0323] 3. Send to the server

[0324] The device sends the captured image data to a server, which stores the received image data in a database and checks it against previous databases to see if it is a new item.

[0325] 4. Analysis by Emotion Engine

[0326] The server analyzes the image data using an emotion engine and identifies the user's emotional state from their facial expressions and movements. The emotion data is saved together with the image data.

[0327] 5. Sharing on social media

[0328] If the server recognizes the item as new, it generates a message containing emotional data for the social networking site. For example, it could generate a message in the format "[October 1, 2023, 14:30] My child brought home a new toy. (Surprised)" and automatically send it to the social networking site.

[0329] Diary creation using generative AI

[0330] 1. Data Collection

[0331] The server extracts audio data and image data for a certain period of time (e.g., one week) from the database.

[0332] 2. Diary-style document generation

[0333] The server uses generative AI to create a diary-style document based on the extracted data. Audio data is transcribed, image data is provided with explanatory text, and emotional data is also reflected in the diary.

[0334] 3. Share your diary

[0335] The created diary can be displayed and shared with the user and their family via social media or a dedicated application, allowing the user to check their growth record in diary format.

[0336] Specific examples

[0337] Audio recording example

[0338] When a user (child) says "Mom," the device recognizes this and records it. The recorded voice data is given a timestamp, such as "October 1, 2023, 10:30." When the server receives this data and analyzes it with its emotion engine, the emotion of joy is detected. The server then shares this information in the form of a message on social media. For example, a message could be sent that reads, "[October 1, 2023, 10:30] Child said 'Mom' (Joy)."

[0339] Image recording example

[0340] When a user (child) brings home a new toy, the device (home camera) recognizes this and takes a picture. The image data is given a timestamp, such as "October 1, 2023, 14:30." When the server receives this data and analyzes it with its emotion engine, the emotion of surprise is detected. The server then shares this information in the form of a message on social media. For example, a message could be sent saying, "[October 1, 2023, 14:30] Your child brought home a new toy. (Surprised)"

[0341] This system allows users to record both their growth and emotions in real time and share them automatically. Generative AI also creates diary-style documents based on the saved data, which can be shared with family members, providing a more detailed and emotional record of their growth.

[0342] The processing flow will be explained below.

[0343] Audio recording

[0344] Step 1:

[0345] The device (smartphone app) activates its voice recognition module and constantly monitors the sounds around the user (child), acquiring the voice data stream and preparing to analyze the voice patterns in real time.

[0346] Step 2:

[0347] The device detects speech. For example, if a voice pattern of "mama" is detected, the voice recognition module trims that part of the audio and recognizes it as speech.

[0348] Step 3:

[0349] The device temporarily stores the recognized speech in memory and adds the current date and time as a timestamp to the recorded data.

[0350] Step 4:

[0351] The device sends time-stamped audio data to the server, which then encrypts it and transfers it securely over the network to the server.

[0352] Step 5:

[0353] The server stores the received voice data in a database, and an algorithm runs to check whether the same utterance exists in the database by comparing it with previous records.

[0354] Step 6:

[0355] The server analyzes the voice data with an emotion engine to identify the user's emotional state (e.g., joy, sadness, anger, etc.). The emotion data is stored together with the voice data.

[0356] Step 7:

[0357] If the server recognizes the utterance as new, it generates a message containing emotional data for the social networking site. For example, it generates a message in the format "[October 1, 2023, 10:30] My child said 'Mom' (joy)" and automatically sends it to the social networking site.

[0358] Image recording

[0359] Step 1:

[0360] The device (home camera) activates the image recognition module to monitor the user's (child's) movements, preparing to analyze the camera footage in real time to detect objects being held in the hands.

[0361] Step 2:

[0362] The device recognizes that the user (child) is holding a new object, and the image recognition module identifies the object and captures an image.

[0363] Step 3:

[0364] The terminal temporarily stores the captured image in memory, and adds the current date and time as a timestamp to the image data.

[0365] Step 4:

[0366] The device sends image data with a timestamp to the server, where it is encrypted and securely transferred to the server over the network.

[0367] Step 5:

[0368] The server stores the received image data in a database, where an algorithm runs to check against previous databases to see if the same object has been recorded before.

[0369] Step 6:

[0370] The server analyzes the image data using an emotion engine and identifies the user's emotional state from their facial expressions and movements. The emotion data is saved together with the image data.

[0371] Step 7:

[0372] If the server recognizes the item as new, it generates a message containing emotional data for the social networking site. For example, it could generate a message in the format "[October 1, 2023, 14:30] My child brought home a new toy. (Surprised)" and automatically send it to the social networking site.

[0373] Diary creation using generative AI

[0374] Step 1:

[0375] The server extracts audio data and image data for a certain period of time (e.g., one week) from the database.

[0376] Step 2:

[0377] The server uses generative AI to create a diary-style document based on the extracted data. Audio data is transcribed, and image data is provided with explanatory text. Emotional data is also reflected in the diary.

[0378] Step 3:

[0379] The server uploads the created diary to social media or a dedicated application, allowing users and their families to view the child's growth record in diary format.

[0380] Step 4:

[0381] Users can view and share the diary entries they create with their families via social media or a dedicated application. The entries are organized with timestamps and include emotional records, making them an accurate and emotional record of their child's growth.

[0382] Example 2

[0383] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0384] Currently, there is a lack of systems that can accurately reflect a user's emotional state and appropriately record and share what a specific person is saying or holding. Furthermore, there is a lack of functionality to automatically generate diary-style documents based on recorded data and share them with the user and their family. This makes it difficult to record a user's growth in a comprehensive and emotional way.

[0385] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes: means for monitoring the voice of a specific person, recognizing the speech, assigning a timestamp, and recording the voice data; means for monitoring an object held by the specific person, recognizing the speech, assigning a timestamp, and recording image data; means for transmitting the recorded voice data and image data to the server and storing them in a database; means for analyzing the emotional state contained in the stored voice data and image data; and means for generating a message including the analyzed emotional data and automatically sharing it via an SNS. This makes it possible to share messages reflecting the emotional state of the user on an SNS by recording the user's speech and belongings. In addition, a diary-style document automatically generated based on the collected data allows the user to emotionally record their growth.

[0386] "Voice data" refers to information on voice that has been recorded by recognizing the speech of a specific person and adding a timestamp.

[0387] "Image data" is image information that is recorded by recognizing an object held by a specific person and adding a timestamp.

[0388] A "timestamp" is information that is added to audio data and image data and indicates the date and time of recording.

[0389] A "server" is a computer system that receives recorded audio and image data, stores it in a database, and performs further processing.

[0390] "Database" refers to an information system for storing and managing recorded audio and video data, as well as related emotional data.

[0391] An "emotion engine" is an algorithm and software that analyzes audio and image data to identify the emotional state of a particular person.

[0392] "SNS" is a social networking service that allows users to share recorded audio data, image data, and emotional data with other users.

[0393] "Message generation" is the process of automatically creating messages to be shared on social media based on analyzed emotional data.

[0394] "Artificial intelligence" refers to algorithms and software technology that analyzes large amounts of data and automatically learns and infers.

[0395] A "diary-style document" is written information that summarizes events and emotions over a specific period of time, generated based on saved audio and image data.

[0396] "Specialized Application" means a specific software application for viewing and sharing the generated diary-style documents.

[0397] This invention combines an emotion engine with a system that automatically records what a specific person is saying and what they are holding, and shares that information via social networking services. Specifically, the system records audio and image data, stores it in a database, analyzes emotions, and generates shared messages that are then automatically posted to social networking services.

[0398] This system mainly consists of a terminal and a server. The terminal can be a smartphone or a home camera. The server is a cloud-based computer system that works in conjunction with a database for processing and storing large amounts of data.

[0399] First, the device has a voice recognition function to constantly monitor the user's voice. For example, it uses voice recognition technology such as Google Speech-to-Text to analyze the user's speech in real time. When it detects a specific utterance (e.g., "Mom"), it records the voice data, adds a timestamp, and temporarily stores it.

[0400] Next, the device uses the home camera to monitor what the user is holding in their hand. Using image recognition technology (e.g., OpenCV), it analyzes what the user is holding in real time, and when it recognizes a new object, it takes a picture of that moment, adds a timestamp to it, and temporarily stores it.

[0401] The voice and image data recorded on the device are periodically sent to a server. The server stores the received data in a database and checks whether the information is new or not from the past database. The server also uses an emotion engine (e.g., IBM Watson Tone Analyzer or Affectiva) to identify the user's emotional state from the voice and image data. This emotion data is also stored along with the data.

[0402] When a new utterance or item is recognized, the server generates a message containing emotion data. For example, it generates a message in the format "[October 1, 2023, 10:30] My child said 'Mom' (joy)" and automatically posts it using a social networking API (e.g., Twitter API or Facebook API).

[0403] Furthermore, the server uses generative AI (e.g., OpenAI GPT-4) to create a diary-style document based on data from a certain period of time (e.g., one week). The voice data is transcribed, and explanatory text is added to the image data. The diary thus created can be shared with the user and their family via a dedicated application or social media. A specific prompt could be, "Please transcribe one week's worth of the user's voice data and compile it into a diary-style document along with emotional data."

[0404] This system allows users to record their growth and emotional state in real time, automatically generating and sharing emotionally rich growth records.

[0405] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0406] Step 1: Audio monitoring and recognition

[0407] The device constantly monitors the sounds around the user. A speech recognition algorithm (e.g., Google Speech-to-Text) is installed on the device, and it analyzes the user's speech in real time. It takes in the surrounding audio data as input and searches for specific utterances (e.g., "Mom"). If recognized, it extracts the utterance and outputs it as audio data.

[0408] Step 2: Recording audio data

[0409] The device records the voice data recognized in step 1 and assigns a timestamp. This voice data is temporarily stored in the device's local storage. Specifically, a timestamp such as "October 1, 2023, 10:30" is assigned to the voice data. The input is the recognized voice data, and the output is voice data with the timestamp.

[0410] Step 3: Send to the server

[0411] The device periodically transmits time-stamped voice data to the server. The transmitted voice data is stored in a database on the server. Specifically, the device uploads data to the server via the Internet. The input is the time-stamped voice data, and the output is the voice data stored in the database on the server.

[0412] Step 4: Emotion Engine Analysis

[0413] The server analyzes the stored voice data using an emotion engine (e.g., IBM Watson Tone Analyzer). It identifies the user's emotional state (e.g., joy, sadness, anger, etc.) from the voice data and generates emotion data. The input is the voice data stored on the server, and the output is emotion data. Specifically, the voice data is passed through an algorithm to analyze the emotional state.

[0414] Step 5: Share on social media

[0415] The server generates a message for social media based on the data recognized as new speech. A message containing emotion data is created in the following format, for example: "[October 1, 2023, 10:30] My child said 'Mom' (joy)." This message is automatically posted using a social media API (e.g., Twitter API). The input is voice data with emotion data, and the output is a post on social media. The specific operation is to post the message via the API.

[0416] Step 6: Image monitoring and recognition

[0417] The device (home camera) monitors the user's movements. It analyzes the camera image in real time and uses an image recognition algorithm (e.g., OpenCV) to detect what is being held in the user's hand. The input is live video data, and the output is recognized image data. As a concrete example, it detects the moment when the user picks up a new toy.

[0418] Step 7: Recording image data

[0419] The device assigns a timestamp to the recognized image data and records it. This data is temporarily stored in local storage. The recognized image data is used as input, and image data with a timestamp is generated as output. Specifically, the device assigns a timestamp such as "October 1, 2023, 14:30" to the image.

[0420] Step 8: Send to the server

[0421] The terminal sends the image data with the timestamp to the server. The sent data is stored in a database on the server. The input is the image data with the timestamp, and the output is the image data stored on the server. In concrete terms, the data is uploaded to the server via the Internet.

[0422] Step 9: Analysis by Emotion Engine (Image)

[0423] The server analyzes the stored image data using an emotion engine (e.g., Affectiva). This identifies the user's emotional state from their facial expressions and movements. The input is the image data stored on the server, and the output is emotion data. Specifically, the image data is passed through an algorithm to analyze the emotional state.

[0424] Step 10: Share on social media (images)

[0425] The server generates a message containing emotional data based on the data recognized as a new item. For example, it creates a message in the format "[October 1, 2023, 14:30] My child brought home a new toy. (Surprised)" and automatically posts it using a Facebook API or similar. The input is image data with emotional data, and the output is a post on social media. The specific operation is to post the message via the API.

[0426] Step 11: Data collection

[0427] The server extracts audio and image data for a certain period (e.g., one week) from a database. The input is a request specifying a specific period, and the output is the data collected during that period. Specifically, the data is extracted through a database query.

[0428] Step 12: Diary-style document generation

[0429] The server uses a generative AI (e.g., OpenAI GPT-4) to create a diary-style document based on the extracted data. As a specific example, the prompt is "Please transcribe one week's worth of user voice data and compile it into a diary-style document along with emotional data." The input is the collected voice data and image data, and the output is a diary-style document.

[0430] Step 13: Share your diary

[0431] The generated diary can be displayed and shared with the user or family members through a dedicated application or social networking site. The input is a diary-format document, and the output is a display on the application or a post on the social networking site. The specific operation involves posting the generated document to the application or social networking site.

[0432] (Application example 2)

[0433] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0434] To strengthen home security, it is important to be able to detect suspicious behavior and abnormal emotional states in real time and respond quickly to them. However, current home security systems do not adequately identify abnormal behavior and emotional states, which can lead to delayed responses in emergencies. In addition, there is a lack of means to quickly communicate detected information to family members or administrators, which delays information sharing and makes it difficult to take appropriate action.

[0435] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for recognizing the speech of a specific person, recording the voice data with a timestamp, recognizing an object held by the specific person, recording image data with a timestamp, saving the recorded voice data and image data in a database and automatically sharing them via SNS, and means for monitoring abnormal behavior and emotional states in specific locations in real time and issuing an alarm. This makes it possible to quickly detect abnormal behavior and emotional states within the home and share information in real time.

[0436] "Means for recognizing the speech of a specific person, adding a timestamp and recording the voice data" is a function that uses voice recognition technology to detect the speech of a selected person and saves the time at which the speech was made, along with a record of the time at which it was made.

[0437] "Means for recognizing an object held by a specific person, assigning a timestamp and recording image data" is a function that uses image analysis technology to detect an object held by a selected subject and saves a record of the time the object was held.

[0438] "Means for storing recorded audio and image data in a database and automatically sharing it through social networking services" refers to a function for building a database for systematically storing collected audio and image data and automatically sharing that data through social networking services.

[0439] "Means for monitoring abnormal behavior and emotional states in a specific location in real time and issuing an alarm" refers to a function that constantly monitors abnormal behavior and emotional states occurring within a designated space and issues an alarm based on the results.

[0440] "A means of using generative AI to create diary-style documents based on saved voice and image data" is a function that uses a machine learning model to generate natural-looking sentences based on collected voice and image data, and compiles them in diary format.

[0441] "Means for displaying and sharing the created diary through social networking services and dedicated applications" refers to a function for displaying and sharing the created diary using social networking services and specific mobile applications.

[0442] This invention is a system that automatically records what specific people say and what they hold in their hands, and shares that data in real time to enhance home security. The system mainly consists of a terminal, a server, and a generative AI model.

[0443] Terminal part

[0444] The device can be a smartphone or a home camera, and has the following main functions:

[0445] 1. Audio Recording:

[0446] It has a voice recognition function that constantly monitors the user's speech.

[0447] When a specific utterance is detected, the audio for that part is recorded and a timestamp is added.

[0448] The recorded data is temporarily stored and later sent to the server.

[0449] 2. Taking an image:

[0450] Cameras with image recognition capabilities for monitoring activity within the home.

[0451] When it detects that a specific object is being held in your hand, it takes an image of that moment and adds a timestamp.

[0452] The captured image data is also temporarily stored and later sent to the server.

[0453] Server part

[0454] The server processes the received audio and image data and generates the diary using generation AI.

[0455] 1. Data storage and collation:

[0456] The server stores the audio data and image data transmitted from the terminal in a database.

[0457] It is compared with past data to check whether the utterance or the item is new.

[0458] 2. Emotion engine analysis:

[0459] The audio and image data is analyzed to determine the user's emotional state.

[0460] This emotion data is also stored in the database.

[0461] 3. Share on social media:

[0462] When new data is recognized, a message containing the emotion data is generated for the SNS.

[0463] For example, it could automatically send a message like, "[October 1, 2023, 10:30] The sound of 'joy' was detected in your home."

[0464] Generative AI Models

[0465] The generative AI model automatically generates diaries from data stored on the server.

[0466] 1. Data Collection:

[0467] The server extracts audio data and image data for a certain period of time (for example, one week).

[0468] 2. Diary generation:

[0469] Generative AI is used to create diary-style documents based on collected data.

[0470] The audio data is transcribed and the image data is accompanied by explanatory text.

[0471] Emotional data is also reflected, providing a more detailed growth record.

[0472] 3. Share your diary:

[0473] The created diary can be displayed and shared via social media or dedicated applications.

[0474] For example, if suspicious activity is detected in the middle of the night in a home, the system will immediately record and analyze the change, and then send a message to social media saying, "October 1, 2023, 2:30 AM: A 'frightening' sound has been detected in the home," alerting the entire family.

[0475] Example prompts to input to a generative AI model:

[0476] "Please generate a sample program for an application that uses the emotion recognition system to monitor and share abnormal behavior and emotional states within the home in real time."

[0477] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0478] Step 1:

[0479] The terminal recognizes the user's speech and records the voice data.

[0480] Input: User utterance

[0481] How it works: The device's voice recognition function constantly monitors the user's speech, and when it detects a specific utterance, it records that part of the audio and adds a timestamp.

[0482] Output: Audio data with timestamps

[0483] Step 2:

[0484] The device temporarily stores the recorded audio data and sends it to the server.

[0485] Input: Audio data with timestamps

[0486] Operation: The audio data temporarily stored on the device is sent to the server.

[0487] Output: Audio data sent to the server

[0488] Step 3:

[0489] The server receives the audio data and stores it in a database.

[0490] Input: Audio data sent to the server

[0491] How it works: The server stores the received voice data in a database and checks it against past databases to see if it is a new utterance.

[0492] Output: Audio data stored in a database

[0493] Step 4:

[0494] The server analyzes the voice data using an emotion engine to identify the user's emotional state.

[0495] Input: Audio data stored in the database

[0496] How it works: The server analyzes the voice data with an emotion engine to determine the user's emotional state.

[0497] Output: Emotion data

[0498] Step 5:

[0499] The server generates a message for SNS based on the voice data and emotion data and automatically sends it.

[0500] Input: Voice data, emotion data

[0501] Behavior: If the server recognizes a new utterance, it generates a message for SNS containing emotion data. Example: "[October 1, 2023, 10:30] The voice of 'joy' was detected in the home."

[0502] Output: Message sent to SNS

[0503] Step 6:

[0504] The device recognizes what the user is holding in their hand and takes a picture.

[0505] Input: User action

[0506] How it works: When the device's camera recognizes an object being held by the user, it takes a picture of that moment and adds a timestamp.

[0507] Output: Image data with timestamp

[0508] Step 7:

[0509] The device temporarily stores the captured image data and sends it to the server.

[0510] Input: Image data with timestamp

[0511] Operation: Image data temporarily stored on the device is sent to the server.

[0512] Output: Image data sent to the server

[0513] Step 8:

[0514] The server receives the image data and stores it in a database.

[0515] Input: Image data sent to the server

[0516] How it works: The server stores the image data it receives in a database and checks it against previous databases to see if it is new.

[0517] Output: Image data stored in a database

[0518] Step 9:

[0519] The server analyzes the image data using an emotion engine to identify the user's emotional state.

[0520] Input: Image data stored in a database

[0521] How it works: The server analyzes the image data with an emotion engine to determine the user's emotional state.

[0522] Output: Emotion data

[0523] Step 10:

[0524] The server generates a message for SNS based on the image data and emotional data and automatically sends it.

[0525] Input: image data, emotion data

[0526] Behavior: If the server recognizes a new possession, it generates a message for social media containing emotion data. Example: "[October 1, 2023, 14:30] 'Surprising' images detected in the home."

[0527] Output: Message sent to SNS

[0528] Step 11:

[0529] The server analyzes a certain period of audio and image data using a generative AI model and creates a diary-style document.

[0530] Input: Audio data and image data for a certain period of time

[0531] How it works: The server uses a generative AI model to create a diary-style document based on the extracted data. Audio data is transcribed, image data is annotated with explanatory text, and emotion data is also reflected in the diary.

[0532] Output: Diary-style document

[0533] Step 12:

[0534] The server displays and shares the created diary document through social networking sites or dedicated applications.

[0535] Input: diary-style document

[0536] How it works: The server displays and shares the created diary with the user and their family via social media or a dedicated application.

[0537] Output: View and share diary entries

[0538] Example prompts to input to a generative AI model:

[0539] "Please generate a sample program for an application that uses the emotion recognition system to monitor and share abnormal behavior and emotional states within the home in real time."

[0540] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0541] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0542] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0543] [Second embodiment]

[0544] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0545] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0546] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0547] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0548] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0549] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0550] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0551] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0552] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0553] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0554] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0555] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0556] The present invention is a system that automatically records what a specific person is saying and what they are holding, and shares that information via social networking services. A specific embodiment of this system will be described below.

[0557] This system consists of a device (e.g., a smartphone or home camera) and a server. It recognizes the user's (child's) speech and the objects they are holding, records them with a timestamp, and automatically shares them via social media.

[0558] Audio recording

[0559] 1. Audio monitoring and recognition

[0560] The device is equipped with a voice recognition function that constantly monitors the sounds around the user (child). The voice recognition algorithm detects specific utterances (e.g., "Mom," "Dad," etc.).

[0561] 2. Recording audio data

[0562] When the device recognizes a user's speech, it records that portion of the audio and adds a timestamp to it, which clearly indicates the date and time of the recording.

[0563] 3. Send to the server

[0564] The device sends the recorded voice data to a server, which stores the received voice data in a database and checks whether the same utterance has been recorded in the past.

[0565] 4. Sharing on social media

[0566] If the server recognizes the utterance as new, it will generate a message to share on SNS (e.g., LINE) and automatically send it. The shared content will include a timestamp, indicating the date and time of the utterance.

[0567] Image recording

[0568] 1. Image monitoring and recognition

[0569] The device (e.g., a home camera) monitors the user's (child's) movements and uses image recognition to identify the object being held.

[0570] 2. Recording image data

[0571] When the device recognizes a new item, it takes a picture and adds a timestamp to it. For example, if a user brings home a new toy, it will record an image of that exact moment.

[0572] 3. Send to the server

[0573] The device sends the captured image data to a server, which stores the received image data in a database and checks whether the same object has been recorded previously.

[0574] 4. Sharing on social media

[0575] If the server recognizes the item as new, it will automatically generate and send a message to share on social media, including a timestamp to indicate when the item was picked up.

[0576] Diary creation using generative AI

[0577] 1. Data Collection

[0578] The server extracts audio data and image data for a certain period (e.g., one week) from the database.

[0579] 2. Diary-style document generation

[0580] The server uses generative AI to create diary-style documents based on the extracted data, transcribes audio data, and provides explanatory text for image data.

[0581] 3. Share your diary

[0582] The created diary can be displayed and shared with users and their families via social media or a dedicated application, allowing users to check their child's growth record in diary format.

[0583] Specific examples

[0584] Audio recording example

[0585] When a user (child) says "Mom," the device recognizes this and records it. The recorded voice data is given a timestamp, such as "October 1, 2023, 10:30." The server receives this data, compares it with its existing database, and if it determines that it has not been recorded, shares the content on a social networking site (e.g., Line) with the message, "[October 1, 2023, 10:30] My child said 'Mom.'"

[0586] Image recording example

[0587] When a user (child) brings home a new toy, the device (home camera) recognizes this and takes a picture. The image data is given a timestamp, such as "October 1, 2023, 14:30." The server receives this data, compares it with its existing database, and if it determines that it has not been recorded, it shares the message on social media, saying, "[October 1, 2023, 14:30] Your child has brought home a new toy."

[0588] This system allows for real-time recording of a child's growth and automatically sharing it. It also uses generative AI to create diary-style documents based on the saved data, which can be shared with family members, providing a more detailed record of a child's growth.

[0589] The processing flow will be explained below.

[0590] Audio recording

[0591] Step 1:

[0592] The device (smartphone app) activates its voice recognition module and constantly monitors the sounds around the user (child), acquiring the voice data stream and preparing to analyze the voice patterns in real time.

[0593] Step 2:

[0594] The device detects a specific speech pattern, for example, "Mama," and the speech recognition module trims that portion of the audio and recognizes it as speech.

[0595] Step 3:

[0596] The device temporarily stores the recognized speech in memory and timestamps the recording with the current date and time, ensuring an accurate recording of the speech.

[0597] Step 4:

[0598] The device sends time-stamped audio data to the server, which then encrypts it and transfers it securely over the network to the server.

[0599] Step 5:

[0600] The server stores the received voice data in a database, and an algorithm runs to check whether the same utterance exists in the database by comparing it with previous records.

[0601] Step 6:

[0602] When the server recognizes a new utterance, it converts the information into a message for use on social media. For example, it generates a message in the format: "[October 1, 2023, 10:30] My child said 'Mom.'"

[0603] Step 7:

[0604] The server automatically sends the generated message to an SNS (e.g., Line), and the recipient can receive the message in the specified account.

[0605] Image recording

[0606] Step 1:

[0607] The device (home camera) activates the image recognition module and constantly monitors the user's (child's) movements, preparing to analyze the camera footage in real time to detect objects being held.

[0608] Step 2:

[0609] The device recognizes that the user (child) is holding a new object, and the image recognition module identifies the object and captures an image.

[0610] Step 3:

[0611] The device temporarily stores the captured image in memory and timestamps the image data with the current date and time, ensuring an accurate record of the photo.

[0612] Step 4:

[0613] The device sends image data with a timestamp to the server, where it is encrypted and securely transferred to the server over the network.

[0614] Step 5:

[0615] The server stores the received image data in a database, where an algorithm runs to check against previous databases to see if the same object has been recorded before.

[0616] Step 6:

[0617] When the server recognizes a new item, it converts that information into a message for social media, such as "My child brought home a new toy [October 1, 2023, 14:30]."

[0618] Step 7:

[0619] The server automatically sends the generated message to an SNS (e.g., Line), and the recipient can receive the message in the specified account.

[0620] Diary creation using generative AI

[0621] Step 1:

[0622] The server extracts audio and image data for a certain period (e.g., one week) from the database, and the extracted data is processed by the AI ​​module.

[0623] Step 2:

[0624] The server uses generative AI to create diary-style documents based on the extracted data, automatically transcribes audio data, and provides explanatory text for image data.

[0625] Step 3:

[0626] The server uploads the created diary to social networking sites or dedicated applications, allowing users and their families to view their child's growth record in diary format.

[0627] Step 4:

[0628] Users can view and share the diary entries they create with their families via social media or a dedicated application. This information is organized with timestamps and can be used as an accurate record of their child's growth.

[0629] Example 1

[0630] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0631] In recent years, there has been a growing need to record children's developmental progress at home and share it with family members. However, conventional methods require time-consuming manual recording and sharing, and often lack real-time performance and accuracy. Furthermore, managing images and audio individually is cumbersome, creating a need for a unified recording method. Furthermore, recorded data needs to be displayed in an easily understandable and organized format when reviewing it later.

[0632] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0633] In this invention, the server includes means for recognizing the speech of a specific person using voice recognition technology, recording the voice data with a timestamp, means for recognizing an object held by the specific person using image recognition technology, recording image data with a timestamp, and means for saving the recorded voice data and image data in a database and automatically sharing them via SNS. This makes it possible to record a child's growth process in real time and automatically share it.

[0634] "Voice recognition technology" is a technology that identifies the speech of a specific person in real time and converts it into text format.

[0635] A "timestamp" is information indicating the date and time when data was recorded, and includes the date and time.

[0636] "Audio data" refers to recorded audio information stored in digital format.

[0637] "Image recognition technology" is a technology that uses computer vision algorithms to automatically identify objects and features in images.

[0638] "Image data" refers to photographs or videos taken by a device such as a camera and stored in digital format.

[0639] A "database" is a system that structures and stores information such as audio data and image data, and allows for efficient management and retrieval.

[0640] "SNS" is an abbreviation for social networking service, a platform that supports communication and information sharing between users via the Internet.

[0641] "Generative AI technology" is an artificial intelligence technology that learns from large amounts of data and generates new data and content.

[0642] A "diary-style document" is a document that describes events that occurred during a specific period in chronological order.

[0643] A "specialized application" is software designed for a specific function or purpose, in this case, to display and share growth records.

[0644] This invention is a system that automatically records what a specific person is saying and what they are holding, and shares that information via social networking services. This system consists of a device (e.g., a smartphone or home camera) and a server. It recognizes what the user (child) is saying and what they are holding, records it with a timestamp, and automatically shares it on social networking services.

[0645] Audio recording

[0646] 1. Audio monitoring and recognition

[0647] The device uses voice recognition technology to constantly monitor the sounds around the user (child). Specifically, it uses the microphone on a smartphone or home camera and uses a voice recognition algorithm such as "Google Cloud Speech-to-Text" to detect specific utterances (e.g., "Mom" or "Dad") in real time.

[0648] 2. Recording audio data

[0649] When the device recognizes a specific utterance, it records that portion of the audio and adds a timestamp using Python's datetime module. For example, the recording date and time is added to the data in the format "October 1, 2023, 10:30." This data is temporarily stored on the device.

[0650] 3. Send to the server

[0651] The device sends the recorded voice data to the server via an HTTP POST request. When the server receives this data, it stores it in a database (e.g., MySQL) and compares it with past data to determine whether the utterance is new.

[0652] 4. Sharing on social media

[0653] If the server recognizes the utterance as new, it uses the API of a social networking service (e.g., Line) to generate and send a message stating, "[October 1, 2023, 10:30 AM] My child said 'Mom.'" This allows the content of the utterance to be shared with family members in real time.

[0654] Image recording

[0655] 1. Image monitoring and recognition

[0656] The device (e.g., a home camera) uses image recognition technology (e.g., OpenCV, TensorFlow) to constantly monitor the user's (child's) movements, thereby identifying the object being held in real time.

[0657] 2. Recording image data

[0658] When the device recognizes a new item, it takes a photo of that moment and adds a timestamp (e.g., "October 1, 2023, 14:30"), which is also temporarily stored on the device.

[0659] 3. Send to the server

[0660] The device sends the captured image data to the server using an HTTP POST request, which stores the data in a database and similarly compares it with past data to determine whether the item is new.

[0661] 4. Sharing on social media

[0662] If the server recognizes the item as new, it generates and sends a message to the social networking site saying, "[October 1, 2023, 14:30] Your child has brought home a new toy."

[0663] Diary creation using generative AI

[0664] 1. Data Collection

[0665] The server extracts audio and image data for a certain period (e.g., one week) from the database, and stores this data temporarily for diary generation.

[0666] 2. Diary-style document generation

[0667] The server uses generative AI technology (e.g., OpenAI GPT-4) to generate diary-style documents based on the extracted data. Audio data is transcribed and image data is annotated with explanatory text. This process is performed by inputting prompts to the generative AI model.

[0668] Example prompt sentence:

[0669] Create a diary of your child's growth based on one week's worth of audio and video data. Here is an example of the data:

[0670] Audio data:

[0671] 1. [October 1, 2023 10:30] "Mom"

[0672] 2. [October 2, 2023 11:00] "Dad"

[0673] Image data:

[0674] 1. [October 1, 2023 14:30] A user has come home with a new toy.

[0675] 2. [October 2, 2023, 15:00] A photo of a user playing in a park

[0676] Based on this data, please write a detailed diary of your child's growth.

[0677] 3. Share your diary

[0678] The created diary can be displayed and shared via social media or a dedicated application. The server converts the created diary into PDF or HTML format and generates a sharing link to provide to the user or their family.

[0679] As described above, the embodiments of the present invention have been specifically described, which allow users to seamlessly manage and share growth records, thereby stimulating communication within the home.

[0680] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0681] Processing Steps

[0682] Audio recording

[0683] Step 1: Audio monitoring and recognition

[0684] The device uses the microphone of a smartphone or home camera to constantly monitor the sounds around the user (child). It uses voice recognition technology (e.g., Google Cloud Speech-to-Text) to detect specific utterances (e.g., "Mom" or "Dad") in real time.

[0685] Input: Ambient sound captured by the microphone

[0686] Processing: Speech recognition algorithms detect specific utterances in real time

[0687] Output: A trigger event that a specific utterance was detected

[0688] Step 2: Recording audio data

[0689] When the device recognizes a specific utterance, it records that portion of the audio and adds a timestamp. Using the Python datetime module, it adds the recording date and time to the data in the format "October 1, 2023, 10:30." The audio data is temporarily stored on the device.

[0690] Input: Trigger event, audio data

[0691] Processing: Recording and time stamping audio data

[0692] Output: Audio data with timestamps

[0693] Step 3: Send to the server

[0694] The device sends the recorded voice data to the server via an HTTP POST request. The data is converted to JSON format and sent to the server's API endpoint. The server receives this data and stores it in a database (e.g., MySQL). A Python script is used to compare it with existing data and determine whether it is a new utterance.

[0695] Input: Time-stamped audio data

[0696] Processing: Convert to JSON format, send via HTTP POST request, save to database, and collate data

[0697] Output: New utterance determination result

[0698] Step 4: Share on social media

[0699] If the server determines that the utterance is new, it uses the API of the social networking service (e.g., Line) to generate and send a shared message. For example, it might send a message to the social networking service saying, "[October 1, 2023, 10:30] My child said 'Mom.'"

[0700] Input: New utterance determination result

[0701] Processing: Message creation and sending via SNS API

[0702] Output: Message sent to SNS

[0703] Image recording

[0704] Step 1: Image monitoring and recognition

[0705] The device (e.g., a home camera) constantly monitors the user's (child's) movements and uses image recognition technology (e.g., OpenCV, TensorFlow) to identify the object being held in real time.

[0706] Input: Image data obtained from the camera

[0707] Processing: Object recognition using image recognition algorithms

[0708] Output: Trigger event that new possession is recognized

[0709] Step 2: Recording image data

[0710] When the device recognizes a new item, it takes an image of that moment and uses the Python datetime module to add a timestamp (e.g., "October 1, 2023, 14:30"), which is also temporarily stored on the device.

[0711] Input: Trigger event, image data

[0712] Processing: Taking an image and adding a timestamp

[0713] Output: Image data with time stamp

[0714] Step 3: Send to the server

[0715] The device sends the captured image data to the server using an HTTP POST request. The data is converted to JSON format and sent to the server's API endpoint. The server receives this data and stores it in a database. It compares it with existing image data to determine if it is a new item.

[0716] Input: Image data with timestamp

[0717] Processing: Convert to JSON format, send via HTTP POST request, save to database, and collate data

[0718] Output: New possession determination result

[0719] Step 4: Share on social media

[0720] If the server determines that the item is new, it uses the SNS API to generate and send a sharing message. For example, it might send a message to the SNS saying, "[October 1, 2023, 14:30] Your child has brought home a new toy."

[0721] Input: New possession determination result

[0722] Processing: Message creation and sending via SNS API

[0723] Output: Message sent to SNS

[0724] Diary creation using generative AI

[0725] Step 1: Data collection

[0726] The server extracts audio and image data for a certain period (e.g., one week) from the database, and stores this data temporarily for diary generation.

[0727] Input: Audio data and image data in the database

[0728] Processing: Data extraction, temporary storage

[0729] Output: Data for diary generation

[0730] Step 2: Creating a diary-style document

[0731] The server uses generative AI technology (e.g., OpenAI GPT-4) to generate diary-style documents based on the extracted data. Audio data is transcribed and image data is annotated with explanatory text. This process is performed by inputting prompts into the generative AI model.

[0732] Input: Data for diary generation, prompt text

[0733] Processing: Generative AI models generate text, transcribe audio data, and add descriptions to image data

[0734] Output: Diary-style document

[0735] Step 3: Share your diary

[0736] The created diary can be displayed and shared via social media or a dedicated application. The server converts the created diary into PDF or HTML format and generates a sharing link to provide to the user or their family.

[0737] Input: diary-style document

[0738] Processing: Convert to PDF or HTML format, generate a link for sharing, send via social media or dedicated applications

[0739] Output: Shareable link, diary display on social media or dedicated application

[0740] This is the specific processing flow of the program for this system, which allows users to automatically manage and share their child's growth record in real time through audio and images.

[0741] (Application example 1)

[0742] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0743] In traditional brick-and-mortar stores, it is possible for customers to talk about products and check the products they have picked up, but there is a lack of a way to properly record customer behavior and the content of their conversations and provide personalized information in real time. It is also difficult to store customer consumption behavior in a database and use it for effective marketing or individualized support. For these reasons, a new system was needed to improve the customer experience and streamline store operations.

[0744] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0745] In this invention, the server includes means for recognizing the speech of a specific person, assigning a timestamp and recording the speech data, means for recognizing an object held by a specific person, assigning a timestamp and recording image data, means for saving the recorded speech data and image data in a database and automatically sharing them via social networking services, means for displaying the speech data and image data on an information display device in real time, and means for generating diary-style documents using a generative AI model based on past records of speech data and image data. This makes it possible to record customer speech and behavior in real time and provide personalized information, thereby improving the efficiency of store operations and the customer experience.

[0746] "Specified Person" refers to the person who will be monitored and recorded by the system.

[0747] "Utterance" refers to the sound produced by a particular person.

[0748] A "timestamp" refers to information indicating the date and time when audio or image data was recorded.

[0749] "Audio data" refers to an audio file that records the speech of a specific person.

[0750] "Handheld object" refers to an object held in the hand of a particular person.

[0751] "Image data" refers to an image file that captures an object being held by a specific person.

[0752] "Database" refers to a system for storing and managing audio and image data.

[0753] "SNS" is an abbreviation for social networking service, and refers to an online platform where users can share information with each other.

[0754] "Automatically sharing" means that the system automatically sends data to social media sites, etc., without the need for human operation.

[0755] "Information display device" refers to a device for visually displaying audio data and image data.

[0756] A "generative AI model" refers to an algorithm that uses machine learning techniques to generate sentences or documents from input data.

[0757] A "diary-style document" refers to a file that records events in chronological order and is organized as text.

[0758] "Personalized information" refers to information and services that are customized for a specific individual.

[0759] "Recording" refers to saving audio data or image data.

[0760] "Displayed in real time" means that data is displayed as soon as it is generated, with little delay.

[0761] The system that realizes this application example has the functions of recognizing the speech of a specific person, recording the voice data with a timestamp, recognizing the object held by the specific person, recording the image data with a timestamp, and storing the recorded voice and image data in a database, which can then be automatically shared via social networking sites.

[0762] Hardware and software used

[0763] This system uses the following hardware and software:

[0764] Microphone: A device for collecting speech from a specific person. It is built into smart glasses.

[0765] Camera: A device that takes pictures of objects held by a specific person. It is built into smart glasses.

[0766] Smart glasses: Information display devices that display speech and image data in real time.

[0767] Server: A computer system that processes and stores data and manages sharing to social media.

[0768] For example, we use the Python speech_recognition library for speech recognition, the cv2 (OpenCV) and pytesseract (OCR) libraries for image recognition, and the requests library for data processing and transmission.

[0769] Data processing and calculation

[0770] 1. Speech Recognition:

[0771] A user wears smart glasses and the microphone collects the speech of a specific person.

[0772] The audio data collected by the microphone is converted to text using the speech_recognition library, which is then time-stamped and the resulting audio data is generated.

[0773] The generated voice data is immediately sent to the server and stored in a database.

[0774] 2. Image Recognition:

[0775] A camera captures an image of an object being held by a particular person.

[0776] The captured image data is then extracted using the cv2 and pytesseract libraries, which then timestamp the image and generate the resulting image data.

[0777] The generated image data is immediately sent to a server and stored in a database.

[0778] 3. Save to database and share on social media:

[0779] The server stores the received audio data and image data in a database.

[0780] New data stored in the database is automatically shared via social media. For example, if a voice utterance such as "What do you think of this product?" is registered, that information will be posted to social media in real time.

[0781] 4. Generative AI diary creation:

[0782] The server extracts past audio and image data for a certain period of time (e.g., one week) and generates a diary-style document using a generative AI model.

[0783] The created diary can be shared with users via social media or dedicated applications.

[0784] Specific examples

[0785] For example, if a customer wearing smart glasses walks through a store and says, "What are the characteristics of this wine?", the microphone collects the voice and the voice recognition software converts it into text. This voice data is time-stamped and sent to the server. In addition, the camera also captures information about the wine bottle the customer picks up, and the image recognition software analyzes the content and sends it to the server.

[0786] The collected data is processed on the server and automatically shared on social media in the form of, for example, "[Date and time] A customer said, 'What are the features of this wine?' Wine information: Red wine, 750ml." At the end of the week, the generative AI model generates a diary-style document based on past data, which is shared via social media and dedicated applications with content such as, "This week's new customer interest product was red wine."

[0787] Prompt Sentence Examples

[0788] What is special about this wine?

[0789] "Tell me more about this product."

[0790] - "What are the new customer interest products this week?"

[0791] This process allows the system to record customer utterances and behavior in real time and provide personalized information, making it a powerful tool for improving customer experience and streamlining store operations.

[0792] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0793] Step 1:

[0794] The microphone in the smart glasses collects the user's speech. The microphone constantly monitors the surrounding sound of a specific person and captures voice data. The input is voice waveform data, and the output is an audio file.

[0795] Step 2:

[0796] Convert audio files to text using the speech_recognition library. Audio data is input and text data is output. Specifically, the speech recognition algorithm analyzes the waveform of the audio and converts it into meaningful words.

[0797] Step 3:

[0798] A timestamp is added to the converted text data. The input is text data obtained by speech recognition, and the current date and time are added as a timestamp. The output is text data with a timestamp.

[0799] Step 4:

[0800] Sends text data with a timestamp to the server. Audio data and a timestamp are input and sent to the server as an HTTP request. The output is the data received on the server side.

[0801] Step 5:

[0802] The server stores the received audio data in a database. The input is time-stamped audio data that is inserted into a database table. The output is an audio recording stored in the database.

[0803] Step 6:

[0804] The camera in the smart glasses takes a picture of the object held by the user. The camera is constantly capturing images of the surroundings, and the input is image data. The output is an image file.

[0805] Step 7:

[0806] Extract text information from an image using the cv2 and pytesseract libraries. The input is image data, and the output is extracted text information. Specifically, OCR technology analyzes the text in the image and outputs it as a string of characters.

[0807] Step 8:

[0808] A timestamp is added to the converted image text data. The input is text data obtained by OCR, and the current date and time are added as a timestamp. The output is text data with a timestamp.

[0809] Step 9:

[0810] Send image text data with a timestamp to the server. Image data and a timestamp are input and sent to the server as an HTTP request. The output is the data received on the server side.

[0811] Step 10:

[0812] The server stores the received image text data in a database. The input is time-stamped image data that is inserted into a database table. The output is an image record stored in the database.

[0813] Step 11:

[0814] The server automatically posts new data to SNS. The input is new data stored in the database, and the server calls the SNS API to generate a message to post. The output is the message posted to SNS.

[0815] Step 12:

[0816] Periodically, the server extracts historical audio and image data. The input is the past week's worth of data from the database, and the output is an extracted data set.

[0817] Step 13:

[0818] The server sends the extracted data to a generative AI model to generate a diary-style document. The input is data from the past week, and the AI ​​model generates sentences. The output is a diary-style document.

[0819] Step 14:

[0820] The server shares the created diary through social networking sites or dedicated applications. The input is the created diary document, which is sent to the social networking site API or application API. The output is the diary data that can be accessed by users.

[0821] In this way, it is possible to record the user's speech and actions in real time and provide personalized information.

[0822] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0823] The present invention combines an emotion engine with a system that automatically records the speech and objects held by a specific person and shares them via social networking services. A specific embodiment of this system is described below.

[0824] This system consists of a device (e.g., smartphone, home camera) and a server. It recognizes the user's (child's) speech and the objects they are holding, records them with a timestamp, and automatically shares them via social media. It also uses an emotion engine to recognize the user's emotional state, and records and shares it.

[0825] Audio recording

[0826] 1. Audio monitoring and recognition

[0827] The device is equipped with a voice recognition function that constantly monitors the sounds around the user (child). A voice recognition algorithm detects specific speech.

[0828] 2. Recording audio data

[0829] When the device recognizes the user's speech, it records that part of the audio, adds a timestamp, and temporarily saves the recorded data.

[0830] 3. Send to the server

[0831] The device sends the recorded voice data to a server, which stores the received voice data in a database and checks whether the voice is new by comparing it with past data.

[0832] 4. Analysis by Emotion Engine

[0833] The server analyzes the voice data with an emotion engine to identify the user's emotional state (e.g., joy, sadness, anger, etc.), and this emotion data is also stored along with the voice data.

[0834] 5. Sharing on social media

[0835] If the server recognizes the utterance as new, it generates a message containing emotional data for the social networking site. For example, it generates a message in the format "[October 1, 2023, 10:30] My child said 'Mom' (joy)" and automatically sends it to the social networking site.

[0836] Image recording

[0837] 1. Image monitoring and recognition

[0838] The device (home camera) activates image recognition and monitors the user's (child's) movements. It analyzes the camera footage to detect what is being held in the user's hand.

[0839] 2. Recording image data

[0840] When the device detects a new object, it takes a photo of that moment, adds a timestamp to it, and then temporarily stores this data.

[0841] 3. Send to the server

[0842] The device sends the captured image data to a server, which stores the received image data in a database and checks it against previous databases to see if it is a new item.

[0843] 4. Analysis by Emotion Engine

[0844] The server analyzes the image data using an emotion engine and identifies the user's emotional state from their facial expressions and movements. The emotion data is saved together with the image data.

[0845] 5. Sharing on social media

[0846] If the server recognizes the item as new, it generates a message containing emotional data for the social networking site. For example, it could generate a message in the format "[October 1, 2023, 14:30] My child brought home a new toy. (Surprised)" and automatically send it to the social networking site.

[0847] Diary creation using generative AI

[0848] 1. Data Collection

[0849] The server extracts audio data and image data for a certain period of time (e.g., one week) from the database.

[0850] 2. Diary-style document generation

[0851] The server uses generative AI to create a diary-style document based on the extracted data. Audio data is transcribed, image data is provided with explanatory text, and emotional data is also reflected in the diary.

[0852] 3. Share your diary

[0853] The created diary can be displayed and shared with the user and their family via social media or a dedicated application, allowing the user to check their growth record in diary format.

[0854] Specific examples

[0855] Audio recording example

[0856] When a user (child) says "Mom," the device recognizes this and records it. The recorded voice data is given a timestamp, such as "October 1, 2023, 10:30." When the server receives this data and analyzes it with its emotion engine, the emotion of joy is detected. The server then shares this information in the form of a message on social media. For example, a message could be sent that reads, "[October 1, 2023, 10:30] Child said 'Mom' (Joy)."

[0857] Image recording example

[0858] When a user (child) brings home a new toy, the device (home camera) recognizes this and takes a picture. The image data is given a timestamp, such as "October 1, 2023, 14:30." When the server receives this data and analyzes it with its emotion engine, the emotion of surprise is detected. The server then shares this information in the form of a message on social media. For example, a message could be sent saying, "[October 1, 2023, 14:30] Your child brought home a new toy. (Surprised)"

[0859] This system allows users to record both their growth and emotions in real time and share them automatically. Generative AI also creates diary-style documents based on the saved data, which can be shared with family members, providing a more detailed and emotional record of their growth.

[0860] The processing flow will be explained below.

[0861] Audio recording

[0862] Step 1:

[0863] The device (smartphone app) activates its voice recognition module and constantly monitors the sounds around the user (child), acquiring the voice data stream and preparing to analyze the voice patterns in real time.

[0864] Step 2:

[0865] The device detects speech. For example, if a voice pattern of "mama" is detected, the voice recognition module trims that part of the audio and recognizes it as speech.

[0866] Step 3:

[0867] The device temporarily stores the recognized speech in memory and adds the current date and time as a timestamp to the recorded data.

[0868] Step 4:

[0869] The device sends time-stamped audio data to the server, which then encrypts it and transfers it securely over the network to the server.

[0870] Step 5:

[0871] The server stores the received voice data in a database, and an algorithm runs to check whether the same utterance exists in the database by comparing it with previous records.

[0872] Step 6:

[0873] The server analyzes the voice data with an emotion engine to identify the user's emotional state (e.g., joy, sadness, anger, etc.). The emotion data is stored together with the voice data.

[0874] Step 7:

[0875] If the server recognizes the utterance as new, it generates a message containing emotional data for the social networking site. For example, it generates a message in the format "[October 1, 2023, 10:30] My child said 'Mom' (joy)" and automatically sends it to the social networking site.

[0876] Image recording

[0877] Step 1:

[0878] The device (home camera) activates the image recognition module to monitor the user's (child's) movements, preparing to analyze the camera footage in real time to detect objects being held in the hands.

[0879] Step 2:

[0880] The device recognizes that the user (child) is holding a new object, and the image recognition module identifies the object and captures an image.

[0881] Step 3:

[0882] The terminal temporarily stores the captured image in memory, and adds the current date and time as a timestamp to the image data.

[0883] Step 4:

[0884] The device sends image data with a timestamp to the server, where it is encrypted and securely transferred to the server over the network.

[0885] Step 5:

[0886] The server stores the received image data in a database, where an algorithm runs to check against previous databases to see if the same object has been recorded before.

[0887] Step 6:

[0888] The server analyzes the image data using an emotion engine and identifies the user's emotional state from their facial expressions and movements. The emotion data is saved together with the image data.

[0889] Step 7:

[0890] If the server recognizes the item as new, it generates a message containing emotional data for the social networking site. For example, it could generate a message in the format "[October 1, 2023, 14:30] My child brought home a new toy. (Surprised)" and automatically send it to the social networking site.

[0891] Diary creation using generative AI

[0892] Step 1:

[0893] The server extracts audio data and image data for a certain period of time (e.g., one week) from the database.

[0894] Step 2:

[0895] The server uses generative AI to create a diary-style document based on the extracted data. Audio data is transcribed, and image data is provided with explanatory text. Emotional data is also reflected in the diary.

[0896] Step 3:

[0897] The server uploads the created diary to social media or a dedicated application, allowing users and their families to view the child's growth record in diary format.

[0898] Step 4:

[0899] Users can view and share the diary entries they create with their families via social media or a dedicated application. The entries are organized with timestamps and include emotional records, making them an accurate and emotional record of their child's growth.

[0900] Example 2

[0901] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0902] Currently, there is a lack of systems that can accurately reflect a user's emotional state and appropriately record and share what a specific person is saying or holding. Furthermore, there is a lack of functionality to automatically generate diary-style documents based on recorded data and share them with the user and their family. This makes it difficult to record a user's growth in a comprehensive and emotional way.

[0903] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes: means for monitoring the voice of a specific person, recognizing the speech, assigning a timestamp, and recording the voice data; means for monitoring an object held by the specific person, recognizing the speech, assigning a timestamp, and recording image data; means for transmitting the recorded voice data and image data to the server and storing them in a database; means for analyzing the emotional state contained in the stored voice data and image data; and means for generating a message including the analyzed emotional data and automatically sharing it via an SNS. This makes it possible to share messages reflecting the emotional state of the user on an SNS by recording the user's speech and belongings. In addition, a diary-style document automatically generated based on the collected data allows the user to emotionally record their growth.

[0904] "Voice data" refers to information on voice that has been recorded by recognizing the speech of a specific person and adding a timestamp.

[0905] "Image data" is image information that is recorded by recognizing an object held by a specific person and adding a timestamp.

[0906] A "timestamp" is information that is added to audio data and image data and indicates the date and time of recording.

[0907] A "server" is a computer system that receives recorded audio and image data, stores it in a database, and performs further processing.

[0908] "Database" refers to an information system for storing and managing recorded audio and video data, as well as related emotional data.

[0909] An "emotion engine" is an algorithm and software that analyzes audio and image data to identify the emotional state of a particular person.

[0910] "SNS" is a social networking service that allows users to share recorded audio data, image data, and emotional data with other users.

[0911] "Message generation" is the process of automatically creating messages to be shared on social media based on analyzed emotional data.

[0912] "Artificial intelligence" refers to algorithms and software technology that analyzes large amounts of data and automatically learns and infers.

[0913] A "diary-style document" is written information that summarizes events and emotions over a specific period of time, generated based on saved audio and image data.

[0914] "Specialized Application" means a specific software application for viewing and sharing the generated diary-style documents.

[0915] This invention combines an emotion engine with a system that automatically records what a specific person is saying and what they are holding, and shares that information via social networking services. Specifically, the system records audio and image data, stores it in a database, analyzes emotions, and generates shared messages that are then automatically posted to social networking services.

[0916] This system mainly consists of a terminal and a server. The terminal can be a smartphone or a home camera. The server is a cloud-based computer system that works in conjunction with a database for processing and storing large amounts of data.

[0917] First, the device has a voice recognition function to constantly monitor the user's voice. For example, it uses voice recognition technology such as Google Speech-to-Text to analyze the user's speech in real time. When it detects a specific utterance (e.g., "Mom"), it records the voice data, adds a timestamp, and temporarily stores it.

[0918] Next, the device uses the home camera to monitor what the user is holding in their hand. Using image recognition technology (e.g., OpenCV), it analyzes what the user is holding in real time, and when it recognizes a new object, it takes a picture of that moment, adds a timestamp to it, and temporarily stores it.

[0919] The voice and image data recorded on the device are periodically sent to a server. The server stores the received data in a database and checks whether the information is new or not from the past database. The server also uses an emotion engine (e.g., IBM Watson Tone Analyzer or Affectiva) to identify the user's emotional state from the voice and image data. This emotion data is also stored along with the data.

[0920] When a new utterance or item is recognized, the server generates a message containing emotion data. For example, it generates a message in the format "[October 1, 2023, 10:30] My child said 'Mom' (joy)" and automatically posts it using a social networking API (e.g., Twitter API or Facebook API).

[0921] Furthermore, the server uses generative AI (e.g., OpenAI GPT-4) to create a diary-style document based on data from a certain period of time (e.g., one week). The voice data is transcribed, and explanatory text is added to the image data. The diary thus created can be shared with the user and their family via a dedicated application or social media. A specific prompt could be, "Please transcribe one week's worth of the user's voice data and compile it into a diary-style document along with emotional data."

[0922] This system allows users to record their growth and emotional state in real time, automatically generating and sharing emotionally rich growth records.

[0923] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0924] Step 1: Audio monitoring and recognition

[0925] The device constantly monitors the sounds around the user. A speech recognition algorithm (e.g., Google Speech-to-Text) is installed on the device, and it analyzes the user's speech in real time. It takes in the surrounding audio data as input and searches for specific utterances (e.g., "Mom"). If recognized, it extracts the utterance and outputs it as audio data.

[0926] Step 2: Recording audio data

[0927] The device records the voice data recognized in step 1 and assigns a timestamp. This voice data is temporarily stored in the device's local storage. Specifically, a timestamp such as "October 1, 2023, 10:30" is assigned to the voice data. The input is the recognized voice data, and the output is voice data with the timestamp.

[0928] Step 3: Send to the server

[0929] The device periodically transmits time-stamped voice data to the server. The transmitted voice data is stored in a database on the server. Specifically, the device uploads data to the server via the Internet. The input is the time-stamped voice data, and the output is the voice data stored in the database on the server.

[0930] Step 4: Emotion Engine Analysis

[0931] The server analyzes the stored voice data using an emotion engine (e.g., IBM Watson Tone Analyzer). It identifies the user's emotional state (e.g., joy, sadness, anger, etc.) from the voice data and generates emotion data. The input is the voice data stored on the server, and the output is emotion data. Specifically, the voice data is passed through an algorithm to analyze the emotional state.

[0932] Step 5: Share on social media

[0933] The server generates a message for social media based on the data recognized as new speech. A message containing emotion data is created in the following format, for example: "[October 1, 2023, 10:30] My child said 'Mom' (joy)." This message is automatically posted using a social media API (e.g., Twitter API). The input is voice data with emotion data, and the output is a post on social media. The specific operation is to post the message via the API.

[0934] Step 6: Image monitoring and recognition

[0935] The device (home camera) monitors the user's movements. It analyzes the camera image in real time and uses an image recognition algorithm (e.g., OpenCV) to detect what is being held in the user's hand. The input is live video data, and the output is recognized image data. As a concrete example, it detects the moment when the user picks up a new toy.

[0936] Step 7: Recording image data

[0937] The device assigns a timestamp to the recognized image data and records it. This data is temporarily stored in local storage. The recognized image data is used as input, and image data with a timestamp is generated as output. Specifically, the device assigns a timestamp such as "October 1, 2023, 14:30" to the image.

[0938] Step 8: Send to the server

[0939] The terminal sends the image data with the timestamp to the server. The sent data is stored in a database on the server. The input is the image data with the timestamp, and the output is the image data stored on the server. In concrete terms, the data is uploaded to the server via the Internet.

[0940] Step 9: Analysis by Emotion Engine (Image)

[0941] The server analyzes the stored image data using an emotion engine (e.g., Affectiva). This identifies the user's emotional state from their facial expressions and movements. The input is the image data stored on the server, and the output is emotion data. Specifically, the image data is passed through an algorithm to analyze the emotional state.

[0942] Step 10: Share on social media (images)

[0943] The server generates a message containing emotional data based on the data recognized as a new item. For example, it creates a message in the format "[October 1, 2023, 14:30] My child brought home a new toy. (Surprised)" and automatically posts it using a Facebook API or similar. The input is image data with emotional data, and the output is a post on social media. The specific operation is to post the message via the API.

[0944] Step 11: Data collection

[0945] The server extracts audio and image data for a certain period (e.g., one week) from a database. The input is a request specifying a specific period, and the output is the data collected during that period. Specifically, the data is extracted through a database query.

[0946] Step 12: Diary-style document generation

[0947] The server uses a generative AI (e.g., OpenAI GPT-4) to create a diary-style document based on the extracted data. As a specific example, the prompt is "Please transcribe one week's worth of user voice data and compile it into a diary-style document along with emotional data." The input is the collected voice data and image data, and the output is a diary-style document.

[0948] Step 13: Share your diary

[0949] The generated diary can be displayed and shared with the user or family members through a dedicated application or social networking site. The input is a diary-format document, and the output is a display on the application or a post on the social networking site. The specific operation involves posting the generated document to the application or social networking site.

[0950] (Application example 2)

[0951] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0952] To strengthen home security, it is important to be able to detect suspicious behavior and abnormal emotional states in real time and respond quickly to them. However, current home security systems do not adequately identify abnormal behavior and emotional states, which can lead to delayed responses in emergencies. In addition, there is a lack of means to quickly communicate detected information to family members or administrators, which delays information sharing and makes it difficult to take appropriate action.

[0953] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for recognizing the speech of a specific person, recording the voice data with a timestamp, recognizing an object held by the specific person, recording image data with a timestamp, saving the recorded voice data and image data in a database and automatically sharing them via SNS, and means for monitoring abnormal behavior and emotional states in specific locations in real time and issuing an alarm. This makes it possible to quickly detect abnormal behavior and emotional states within the home and share information in real time.

[0954] "Means for recognizing the speech of a specific person, adding a timestamp and recording the voice data" is a function that uses voice recognition technology to detect the speech of a selected person and saves the time at which the speech was made, along with a record of the time at which it was made.

[0955] "Means for recognizing an object held by a specific person, assigning a timestamp and recording image data" is a function that uses image analysis technology to detect an object held by a selected subject and saves a record of the time the object was held.

[0956] "Means for storing recorded audio and image data in a database and automatically sharing it through social networking services" refers to a function for building a database for systematically storing collected audio and image data and automatically sharing that data through social networking services.

[0957] "Means for monitoring abnormal behavior and emotional states in a specific location in real time and issuing an alarm" refers to a function that constantly monitors abnormal behavior and emotional states occurring within a designated space and issues an alarm based on the results.

[0958] "A means of using generative AI to create diary-style documents based on saved voice and image data" is a function that uses a machine learning model to generate natural-looking sentences based on collected voice and image data, and compiles them in diary format.

[0959] "Means for displaying and sharing the created diary through social networking services and dedicated applications" refers to a function for displaying and sharing the created diary using social networking services and specific mobile applications.

[0960] This invention is a system that automatically records what specific people say and what they hold in their hands, and shares that data in real time to enhance home security. The system mainly consists of a terminal, a server, and a generative AI model.

[0961] Terminal part

[0962] The device can be a smartphone or a home camera, and has the following main functions:

[0963] 1. Audio Recording:

[0964] It has a voice recognition function that constantly monitors the user's speech.

[0965] When a specific utterance is detected, the audio for that part is recorded and a timestamp is added.

[0966] The recorded data is temporarily stored and later sent to the server.

[0967] 2. Taking an image:

[0968] Cameras with image recognition capabilities for monitoring activity within the home.

[0969] When it detects that a specific object is being held in your hand, it takes an image of that moment and adds a timestamp.

[0970] The captured image data is also temporarily stored and later sent to the server.

[0971] Server part

[0972] The server processes the received audio and image data and generates the diary using generation AI.

[0973] 1. Data storage and collation:

[0974] The server stores the audio data and image data transmitted from the terminal in a database.

[0975] It is compared with past data to check whether the utterance or the item is new.

[0976] 2. Emotion engine analysis:

[0977] The audio and image data is analyzed to determine the user's emotional state.

[0978] This emotion data is also stored in the database.

[0979] 3. Share on social media:

[0980] When new data is recognized, a message containing the emotion data is generated for the SNS.

[0981] For example, it could automatically send a message like, "[October 1, 2023, 10:30] The sound of 'joy' was detected in your home."

[0982] Generative AI Models

[0983] The generative AI model automatically generates diaries from data stored on the server.

[0984] 1. Data Collection:

[0985] The server extracts audio data and image data for a certain period of time (for example, one week).

[0986] 2. Diary generation:

[0987] Generative AI is used to create diary-style documents based on collected data.

[0988] The audio data is transcribed and the image data is accompanied by explanatory text.

[0989] Emotional data is also reflected, providing a more detailed growth record.

[0990] 3. Share your diary:

[0991] The created diary can be displayed and shared via social media or dedicated applications.

[0992] For example, if suspicious activity is detected in the middle of the night in a home, the system will immediately record and analyze the change, and then send a message to social media saying, "October 1, 2023, 2:30 AM: A 'frightening' sound has been detected in the home," alerting the entire family.

[0993] Example prompts to input to a generative AI model:

[0994] "Please generate a sample program for an application that uses the emotion recognition system to monitor and share abnormal behavior and emotional states within the home in real time."

[0995] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0996] Step 1:

[0997] The terminal recognizes the user's speech and records the voice data.

[0998] Input: User utterance

[0999] How it works: The device's voice recognition function constantly monitors the user's speech, and when it detects a specific utterance, it records that part of the audio and adds a timestamp.

[1000] Output: Audio data with timestamps

[1001] Step 2:

[1002] The device temporarily stores the recorded audio data and sends it to the server.

[1003] Input: Audio data with timestamps

[1004] Operation: The audio data temporarily stored on the device is sent to the server.

[1005] Output: Audio data sent to the server

[1006] Step 3:

[1007] The server receives the audio data and stores it in a database.

[1008] Input: Audio data sent to the server

[1009] How it works: The server stores the received voice data in a database and checks it against past databases to see if it is a new utterance.

[1010] Output: Audio data stored in a database

[1011] Step 4:

[1012] The server analyzes the voice data using an emotion engine to identify the user's emotional state.

[1013] Input: Audio data stored in the database

[1014] How it works: The server analyzes the voice data with an emotion engine to determine the user's emotional state.

[1015] Output: Emotion data

[1016] Step 5:

[1017] The server generates a message for SNS based on the voice data and emotion data and automatically sends it.

[1018] Input: Voice data, emotion data

[1019] Behavior: If the server recognizes a new utterance, it generates a message for SNS containing emotion data. Example: "[October 1, 2023, 10:30] The voice of 'joy' was detected in the home."

[1020] Output: Message sent to SNS

[1021] Step 6:

[1022] The device recognizes what the user is holding in their hand and takes a picture.

[1023] Input: User action

[1024] How it works: When the device's camera recognizes an object being held by the user, it takes a picture of that moment and adds a timestamp.

[1025] Output: Image data with timestamp

[1026] Step 7:

[1027] The device temporarily stores the captured image data and sends it to the server.

[1028] Input: Image data with timestamp

[1029] Operation: Image data temporarily stored on the device is sent to the server.

[1030] Output: Image data sent to the server

[1031] Step 8:

[1032] The server receives the image data and stores it in a database.

[1033] Input: Image data sent to the server

[1034] How it works: The server stores the image data it receives in a database and checks it against previous databases to see if it is new.

[1035] Output: Image data stored in a database

[1036] Step 9:

[1037] The server analyzes the image data using an emotion engine to identify the user's emotional state.

[1038] Input: Image data stored in a database

[1039] How it works: The server analyzes the image data with an emotion engine to determine the user's emotional state.

[1040] Output: Emotion data

[1041] Step 10:

[1042] The server generates a message for SNS based on the image data and emotional data and automatically sends it.

[1043] Input: image data, emotion data

[1044] Behavior: If the server recognizes a new possession, it generates a message for social media containing emotion data. Example: "[October 1, 2023, 14:30] 'Surprising' images detected in the home."

[1045] Output: Message sent to SNS

[1046] Step 11:

[1047] The server analyzes a certain period of audio and image data using a generative AI model and creates a diary-style document.

[1048] Input: Audio data and image data for a certain period of time

[1049] How it works: The server uses a generative AI model to create a diary-style document based on the extracted data. Audio data is transcribed, image data is annotated with explanatory text, and emotion data is also reflected in the diary.

[1050] Output: Diary-style document

[1051] Step 12:

[1052] The server displays and shares the created diary document through social networking sites or dedicated applications.

[1053] Input: diary-style document

[1054] How it works: The server displays and shares the created diary with the user and their family via social media or a dedicated application.

[1055] Output: View and share diary entries

[1056] Example prompts to input to a generative AI model:

[1057] "Please generate a sample program for an application that uses the emotion recognition system to monitor and share abnormal behavior and emotional states within the home in real time."

[1058] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1059] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1060] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[1061] [Third embodiment]

[1062] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[1063] 5, the data processing system 310 includes the data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[1064] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1065] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[1066] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1067] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1068] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1069] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1070] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1071] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1072] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1073] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[1074] The present invention is a system that automatically records what a specific person is saying and what they are holding, and shares that information via social networking services. A specific embodiment of this system will be described below.

[1075] This system consists of a device (e.g., a smartphone or home camera) and a server. It recognizes the user's (child's) speech and the objects they are holding, records them with a timestamp, and automatically shares them via social media.

[1076] Audio recording

[1077] 1. Audio monitoring and recognition

[1078] The device is equipped with a voice recognition function that constantly monitors the sounds around the user (child). The voice recognition algorithm detects specific utterances (e.g., "Mom," "Dad," etc.).

[1079] 2. Recording audio data

[1080] When the device recognizes a user's speech, it records that portion of the audio and adds a timestamp to it, which clearly indicates the date and time of the recording.

[1081] 3. Send to the server

[1082] The device sends the recorded voice data to a server, which stores the received voice data in a database and checks whether the same utterance has been recorded in the past.

[1083] 4. Sharing on social media

[1084] If the server recognizes the utterance as new, it will generate a message to share on SNS (e.g., LINE) and automatically send it. The shared content will include a timestamp, indicating the date and time of the utterance.

[1085] Image recording

[1086] 1. Image monitoring and recognition

[1087] The device (e.g., a home camera) monitors the user's (child's) movements and uses image recognition to identify the object being held.

[1088] 2. Recording image data

[1089] When the device recognizes a new item, it takes a picture and adds a timestamp to it. For example, if a user brings home a new toy, it will record an image of that exact moment.

[1090] 3. Send to the server

[1091] The device sends the captured image data to a server, which stores the received image data in a database and checks whether the same object has been recorded previously.

[1092] 4. Sharing on social media

[1093] If the server recognizes the item as new, it will automatically generate and send a message to share on social media, including a timestamp to indicate when the item was picked up.

[1094] Diary creation using generative AI

[1095] 1. Data Collection

[1096] The server extracts audio data and image data for a certain period (e.g., one week) from the database.

[1097] 2. Diary-style document generation

[1098] The server uses generative AI to create diary-style documents based on the extracted data, transcribes audio data, and provides explanatory text for image data.

[1099] 3. Share your diary

[1100] The created diary can be displayed and shared with users and their families via social media or a dedicated application, allowing users to check their child's growth record in diary format.

[1101] Specific examples

[1102] Audio recording example

[1103] When a user (child) says "Mom," the device recognizes this and records it. The recorded voice data is given a timestamp, such as "October 1, 2023, 10:30." The server receives this data, compares it with its existing database, and if it determines that it has not been recorded, shares the content on a social networking site (e.g., Line) with the message, "[October 1, 2023, 10:30] My child said 'Mom.'"

[1104] Image recording example

[1105] When a user (child) brings home a new toy, the device (home camera) recognizes this and takes a picture. The image data is given a timestamp, such as "October 1, 2023, 14:30." The server receives this data, compares it with its existing database, and if it determines that it has not been recorded, it shares the message on social media, saying, "[October 1, 2023, 14:30] Your child has brought home a new toy."

[1106] This system allows for real-time recording of a child's growth and automatically sharing it. It also uses generative AI to create diary-style documents based on the saved data, which can be shared with family members, providing a more detailed record of a child's growth.

[1107] The processing flow will be explained below.

[1108] Audio recording

[1109] Step 1:

[1110] The device (smartphone app) activates its voice recognition module and constantly monitors the sounds around the user (child), acquiring the voice data stream and preparing to analyze the voice patterns in real time.

[1111] Step 2:

[1112] The device detects a specific speech pattern, for example, "Mama," and the speech recognition module trims that portion of the audio and recognizes it as speech.

[1113] Step 3:

[1114] The device temporarily stores the recognized speech in memory and timestamps the recording with the current date and time, ensuring an accurate recording of the speech.

[1115] Step 4:

[1116] The device sends time-stamped audio data to the server, which then encrypts it and transfers it securely over the network to the server.

[1117] Step 5:

[1118] The server stores the received voice data in a database, and an algorithm runs to check whether the same utterance exists in the database by comparing it with previous records.

[1119] Step 6:

[1120] When the server recognizes a new utterance, it converts the information into a message for use on social media. For example, it generates a message in the format: "[October 1, 2023, 10:30] My child said 'Mom.'"

[1121] Step 7:

[1122] The server automatically sends the generated message to an SNS (e.g., Line), and the recipient can receive the message in the specified account.

[1123] Image recording

[1124] Step 1:

[1125] The device (home camera) activates the image recognition module and constantly monitors the user's (child's) movements, preparing to analyze the camera footage in real time to detect objects being held.

[1126] Step 2:

[1127] The device recognizes that the user (child) is holding a new object, and the image recognition module identifies the object and captures an image.

[1128] Step 3:

[1129] The device temporarily stores the captured image in memory and timestamps the image data with the current date and time, ensuring an accurate record of the photo.

[1130] Step 4:

[1131] The device sends image data with a timestamp to the server, where it is encrypted and securely transferred to the server over the network.

[1132] Step 5:

[1133] The server stores the received image data in a database, where an algorithm runs to check against previous databases to see if the same object has been recorded before.

[1134] Step 6:

[1135] When the server recognizes a new item, it converts that information into a message for social media, such as "My child brought home a new toy [October 1, 2023, 14:30]."

[1136] Step 7:

[1137] The server automatically sends the generated message to an SNS (e.g., Line), and the recipient can receive the message in the specified account.

[1138] Diary creation using generative AI

[1139] Step 1:

[1140] The server extracts audio and image data for a certain period (e.g., one week) from the database, and the extracted data is processed by the AI ​​module.

[1141] Step 2:

[1142] The server uses generative AI to create diary-style documents based on the extracted data, automatically transcribes audio data, and provides explanatory text for image data.

[1143] Step 3:

[1144] The server uploads the created diary to social networking sites or dedicated applications, allowing users and their families to view their child's growth record in diary format.

[1145] Step 4:

[1146] Users can view and share the diary entries they create with their families via social media or a dedicated application. This information is organized with timestamps and can be used as an accurate record of their child's growth.

[1147] Example 1

[1148] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1149] In recent years, there has been a growing need to record children's developmental progress at home and share it with family members. However, conventional methods require time-consuming manual recording and sharing, and often lack real-time performance and accuracy. Furthermore, managing images and audio individually is cumbersome, creating a need for a unified recording method. Furthermore, recorded data needs to be displayed in an easily understandable and organized format when reviewing it later.

[1150] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1151] In this invention, the server includes means for recognizing the speech of a specific person using voice recognition technology, recording the voice data with a timestamp, means for recognizing an object held by the specific person using image recognition technology, recording image data with a timestamp, and means for saving the recorded voice data and image data in a database and automatically sharing them via SNS. This makes it possible to record a child's growth process in real time and automatically share it.

[1152] "Voice recognition technology" is a technology that identifies the speech of a specific person in real time and converts it into text format.

[1153] A "timestamp" is information indicating the date and time when data was recorded, and includes the date and time.

[1154] "Audio data" refers to recorded audio information stored in digital format.

[1155] "Image recognition technology" is a technology that uses computer vision algorithms to automatically identify objects and features in images.

[1156] "Image data" refers to photographs or videos taken by a device such as a camera and stored in digital format.

[1157] A "database" is a system that structures and stores information such as audio data and image data, and allows for efficient management and retrieval.

[1158] "SNS" is an abbreviation for social networking service, a platform that supports communication and information sharing between users via the Internet.

[1159] "Generative AI technology" is an artificial intelligence technology that learns from large amounts of data and generates new data and content.

[1160] A "diary-style document" is a document that describes events that occurred during a specific period in chronological order.

[1161] A "specialized application" is software designed for a specific function or purpose, in this case, to display and share growth records.

[1162] This invention is a system that automatically records what a specific person is saying and what they are holding, and shares that information via social networking services. This system consists of a device (e.g., a smartphone or home camera) and a server. It recognizes what the user (child) is saying and what they are holding, records it with a timestamp, and automatically shares it on social networking services.

[1163] Audio recording

[1164] 1. Audio monitoring and recognition

[1165] The device uses voice recognition technology to constantly monitor the sounds around the user (child). Specifically, it uses the microphone on a smartphone or home camera and uses a voice recognition algorithm such as "Google Cloud Speech-to-Text" to detect specific utterances (e.g., "Mom" or "Dad") in real time.

[1166] 2. Recording audio data

[1167] When the device recognizes a specific utterance, it records that portion of the audio and adds a timestamp using Python's datetime module. For example, the recording date and time is added to the data in the format "October 1, 2023, 10:30." This data is temporarily stored on the device.

[1168] 3. Send to the server

[1169] The device sends the recorded voice data to the server via an HTTP POST request. When the server receives this data, it stores it in a database (e.g., MySQL) and compares it with past data to determine whether the utterance is new.

[1170] 4. Sharing on social media

[1171] If the server recognizes the utterance as new, it uses the API of a social networking service (e.g., Line) to generate and send a message stating, "[October 1, 2023, 10:30 AM] My child said 'Mom.'" This allows the content of the utterance to be shared with family members in real time.

[1172] Image recording

[1173] 1. Image monitoring and recognition

[1174] The device (e.g., a home camera) uses image recognition technology (e.g., OpenCV, TensorFlow) to constantly monitor the user's (child's) movements, thereby identifying the object being held in real time.

[1175] 2. Recording image data

[1176] When the device recognizes a new item, it takes a photo of that moment and adds a timestamp (e.g., "October 1, 2023, 14:30"), which is also temporarily stored on the device.

[1177] 3. Send to the server

[1178] The device sends the captured image data to the server using an HTTP POST request, which stores the data in a database and similarly compares it with past data to determine whether the item is new.

[1179] 4. Sharing on social media

[1180] If the server recognizes the item as new, it generates and sends a message to the social networking site saying, "[October 1, 2023, 14:30] Your child has brought home a new toy."

[1181] Diary creation using generative AI

[1182] 1. Data Collection

[1183] The server extracts audio and image data for a certain period (e.g., one week) from the database, and stores this data temporarily for diary generation.

[1184] 2. Diary-style document generation

[1185] The server uses generative AI technology (e.g., OpenAI GPT-4) to generate diary-style documents based on the extracted data. Audio data is transcribed and image data is annotated with explanatory text. This process is performed by inputting prompts to the generative AI model.

[1186] Example prompt sentence:

[1187] Create a diary of your child's growth based on one week's worth of audio and video data. Here is an example of the data:

[1188] Audio data:

[1189] 1. [October 1, 2023 10:30] "Mom"

[1190] 2. [October 2, 2023 11:00] "Dad"

[1191] Image data:

[1192] 1. [October 1, 2023 14:30] A user has come home with a new toy.

[1193] 2. [October 2, 2023, 15:00] A photo of a user playing in a park

[1194] Based on this data, please write a detailed diary of your child's growth.

[1195] 3. Share your diary

[1196] The created diary can be displayed and shared via social media or a dedicated application. The server converts the created diary into PDF or HTML format and generates a sharing link to provide to the user or their family.

[1197] As described above, the embodiments of the present invention have been specifically described, which allow users to seamlessly manage and share growth records, thereby stimulating communication within the home.

[1198] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1199] Processing Steps

[1200] Audio recording

[1201] Step 1: Audio monitoring and recognition

[1202] The device uses the microphone of a smartphone or home camera to constantly monitor the sounds around the user (child). It uses voice recognition technology (e.g., Google Cloud Speech-to-Text) to detect specific utterances (e.g., "Mom" or "Dad") in real time.

[1203] Input: Ambient sound captured by the microphone

[1204] Processing: Speech recognition algorithms detect specific utterances in real time

[1205] Output: A trigger event that a specific utterance was detected

[1206] Step 2: Recording audio data

[1207] When the device recognizes a specific utterance, it records that portion of the audio and adds a timestamp. Using the Python datetime module, it adds the recording date and time to the data in the format "October 1, 2023, 10:30." The audio data is temporarily stored on the device.

[1208] Input: Trigger event, audio data

[1209] Processing: Recording and time stamping audio data

[1210] Output: Audio data with timestamps

[1211] Step 3: Send to the server

[1212] The device sends the recorded voice data to the server via an HTTP POST request. The data is converted to JSON format and sent to the server's API endpoint. The server receives this data and stores it in a database (e.g., MySQL). A Python script is used to compare it with existing data and determine whether it is a new utterance.

[1213] Input: Time-stamped audio data

[1214] Processing: Convert to JSON format, send via HTTP POST request, save to database, and collate data

[1215] Output: New utterance determination result

[1216] Step 4: Share on social media

[1217] If the server determines that the utterance is new, it uses the API of the social networking service (e.g., Line) to generate and send a shared message. For example, it might send a message to the social networking service saying, "[October 1, 2023, 10:30] My child said 'Mom.'"

[1218] Input: New utterance determination result

[1219] Processing: Message creation and sending via SNS API

[1220] Output: Message sent to SNS

[1221] Image recording

[1222] Step 1: Image monitoring and recognition

[1223] The device (e.g., a home camera) constantly monitors the user's (child's) movements and uses image recognition technology (e.g., OpenCV, TensorFlow) to identify the object being held in real time.

[1224] Input: Image data obtained from the camera

[1225] Processing: Object recognition using image recognition algorithms

[1226] Output: Trigger event that new possession is recognized

[1227] Step 2: Recording image data

[1228] When the device recognizes a new item, it takes an image of that moment and uses the Python datetime module to add a timestamp (e.g., "October 1, 2023, 14:30"), which is also temporarily stored on the device.

[1229] Input: Trigger event, image data

[1230] Processing: Taking an image and adding a timestamp

[1231] Output: Image data with time stamp

[1232] Step 3: Send to the server

[1233] The device sends the captured image data to the server using an HTTP POST request. The data is converted to JSON format and sent to the server's API endpoint. The server receives this data and stores it in a database. It compares it with existing image data to determine if it is a new item.

[1234] Input: Image data with timestamp

[1235] Processing: Convert to JSON format, send via HTTP POST request, save to database, and collate data

[1236] Output: New possession determination result

[1237] Step 4: Share on social media

[1238] If the server determines that the item is new, it uses the SNS API to generate and send a sharing message. For example, it might send a message to the SNS saying, "[October 1, 2023, 14:30] Your child has brought home a new toy."

[1239] Input: New possession determination result

[1240] Processing: Message creation and sending via SNS API

[1241] Output: Message sent to SNS

[1242] Diary creation using generative AI

[1243] Step 1: Data collection

[1244] The server extracts audio and image data for a certain period (e.g., one week) from the database, and stores this data temporarily for diary generation.

[1245] Input: Audio data and image data in the database

[1246] Processing: Data extraction, temporary storage

[1247] Output: Data for diary generation

[1248] Step 2: Creating a diary-style document

[1249] The server uses generative AI technology (e.g., OpenAI GPT-4) to generate diary-style documents based on the extracted data. Audio data is transcribed and image data is annotated with explanatory text. This process is performed by inputting prompts into the generative AI model.

[1250] Input: Data for diary generation, prompt text

[1251] Processing: Generative AI models generate text, transcribe audio data, and add descriptions to image data

[1252] Output: Diary-style document

[1253] Step 3: Share your diary

[1254] The created diary can be displayed and shared via social media or a dedicated application. The server converts the created diary into PDF or HTML format and generates a sharing link to provide to the user or their family.

[1255] Input: diary-style document

[1256] Processing: Convert to PDF or HTML format, generate a link for sharing, send via social media or dedicated applications

[1257] Output: Shareable link, diary display on social media or dedicated application

[1258] This is the specific processing flow of the program for this system, which allows users to automatically manage and share their child's growth record in real time through audio and images.

[1259] (Application example 1)

[1260] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1261] In traditional brick-and-mortar stores, it is possible for customers to talk about products and check the products they have picked up, but there is a lack of a way to properly record customer behavior and the content of their conversations and provide personalized information in real time. It is also difficult to store customer consumption behavior in a database and use it for effective marketing or individualized support. For these reasons, a new system was needed to improve the customer experience and streamline store operations.

[1262] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1263] In this invention, the server includes means for recognizing the speech of a specific person, assigning a timestamp and recording the speech data, means for recognizing an object held by a specific person, assigning a timestamp and recording image data, means for saving the recorded speech data and image data in a database and automatically sharing them via social networking services, means for displaying the speech data and image data on an information display device in real time, and means for generating diary-style documents using a generative AI model based on past records of speech data and image data. This makes it possible to record customer speech and behavior in real time and provide personalized information, thereby improving the efficiency of store operations and the customer experience.

[1264] "Specified Person" refers to the person who will be monitored and recorded by the system.

[1265] "Utterance" refers to the sound produced by a particular person.

[1266] A "timestamp" refers to information indicating the date and time when audio or image data was recorded.

[1267] "Audio data" refers to an audio file that records the speech of a specific person.

[1268] "Handheld object" refers to an object held in the hand of a particular person.

[1269] "Image data" refers to an image file that captures an object being held by a specific person.

[1270] "Database" refers to a system for storing and managing audio and image data.

[1271] "SNS" is an abbreviation for social networking service, and refers to an online platform where users can share information with each other.

[1272] "Automatically sharing" means that the system automatically sends data to social media sites, etc., without the need for human operation.

[1273] "Information display device" refers to a device for visually displaying audio data and image data.

[1274] A "generative AI model" refers to an algorithm that uses machine learning techniques to generate sentences or documents from input data.

[1275] A "diary-style document" refers to a file that records events in chronological order and is organized as text.

[1276] "Personalized information" refers to information and services that are customized for a specific individual.

[1277] "Recording" refers to saving audio data or image data.

[1278] "Displayed in real time" means that data is displayed as soon as it is generated, with little delay.

[1279] The system that realizes this application example has the functions of recognizing the speech of a specific person, recording the voice data with a timestamp, recognizing the object held by the specific person, recording the image data with a timestamp, and storing the recorded voice and image data in a database, which can then be automatically shared via social networking sites.

[1280] Hardware and software used

[1281] This system uses the following hardware and software:

[1282] Microphone: A device for collecting speech from a specific person. It is built into smart glasses.

[1283] Camera: A device that takes pictures of objects held by a specific person. It is built into smart glasses.

[1284] Smart glasses: Information display devices that display speech and image data in real time.

[1285] Server: A computer system that processes and stores data and manages sharing to social media.

[1286] For example, we use the Python speech_recognition library for speech recognition, the cv2 (OpenCV) and pytesseract (OCR) libraries for image recognition, and the requests library for data processing and transmission.

[1287] Data processing and calculation

[1288] 1. Speech Recognition:

[1289] A user wears smart glasses and the microphone collects the speech of a specific person.

[1290] The audio data collected by the microphone is converted to text using the speech_recognition library, which is then time-stamped and the resulting audio data is generated.

[1291] The generated voice data is immediately sent to the server and stored in a database.

[1292] 2. Image Recognition:

[1293] A camera captures an image of an object being held by a particular person.

[1294] The captured image data is then extracted using the cv2 and pytesseract libraries, which then timestamp the image and generate the resulting image data.

[1295] The generated image data is immediately sent to a server and stored in a database.

[1296] 3. Save to database and share on social media:

[1297] The server stores the received audio data and image data in a database.

[1298] New data stored in the database is automatically shared via social media. For example, if a voice utterance such as "What do you think of this product?" is registered, that information will be posted to social media in real time.

[1299] 4. Generative AI diary creation:

[1300] The server extracts past audio and image data for a certain period of time (e.g., one week) and generates a diary-style document using a generative AI model.

[1301] The created diary can be shared with users via social media or dedicated applications.

[1302] Specific examples

[1303] For example, if a customer wearing smart glasses walks through a store and says, "What are the characteristics of this wine?", the microphone collects the voice and the voice recognition software converts it into text. This voice data is time-stamped and sent to the server. In addition, the camera also captures information about the wine bottle the customer picks up, and the image recognition software analyzes the content and sends it to the server.

[1304] The collected data is processed on the server and automatically shared on social media in the form of, for example, "[Date and time] A customer said, 'What are the features of this wine?' Wine information: Red wine, 750ml." At the end of the week, the generative AI model generates a diary-style document based on past data, which is shared via social media and dedicated applications with content such as, "This week's new customer interest product was red wine."

[1305] Prompt Sentence Examples

[1306] What is special about this wine?

[1307] "Tell me more about this product."

[1308] - "What are the new customer interest products this week?"

[1309] This process allows the system to record customer utterances and behavior in real time and provide personalized information, making it a powerful tool for improving customer experience and streamlining store operations.

[1310] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1311] Step 1:

[1312] The microphone in the smart glasses collects the user's speech. The microphone constantly monitors the surrounding sound of a specific person and captures voice data. The input is voice waveform data, and the output is an audio file.

[1313] Step 2:

[1314] Convert audio files to text using the speech_recognition library. Audio data is input and text data is output. Specifically, the speech recognition algorithm analyzes the waveform of the audio and converts it into meaningful words.

[1315] Step 3:

[1316] A timestamp is added to the converted text data. The input is text data obtained by speech recognition, and the current date and time are added as a timestamp. The output is text data with a timestamp.

[1317] Step 4:

[1318] Sends text data with a timestamp to the server. Audio data and a timestamp are input and sent to the server as an HTTP request. The output is the data received on the server side.

[1319] Step 5:

[1320] The server stores the received audio data in a database. The input is time-stamped audio data that is inserted into a database table. The output is an audio recording stored in the database.

[1321] Step 6:

[1322] The camera in the smart glasses takes a picture of the object held by the user. The camera is constantly capturing images of the surroundings, and the input is image data. The output is an image file.

[1323] Step 7:

[1324] Extract text information from an image using the cv2 and pytesseract libraries. The input is image data, and the output is extracted text information. Specifically, OCR technology analyzes the text in the image and outputs it as a string of characters.

[1325] Step 8:

[1326] A timestamp is added to the converted image text data. The input is text data obtained by OCR, and the current date and time are added as a timestamp. The output is text data with a timestamp.

[1327] Step 9:

[1328] Send image text data with a timestamp to the server. Image data and a timestamp are input and sent to the server as an HTTP request. The output is the data received on the server side.

[1329] Step 10:

[1330] The server stores the received image text data in a database. The input is time-stamped image data that is inserted into a database table. The output is an image record stored in the database.

[1331] Step 11:

[1332] The server automatically posts new data to SNS. The input is new data stored in the database, and the server calls the SNS API to generate a message to post. The output is the message posted to SNS.

[1333] Step 12:

[1334] Periodically, the server extracts historical audio and image data. The input is the past week's worth of data from the database, and the output is an extracted data set.

[1335] Step 13:

[1336] The server sends the extracted data to a generative AI model to generate a diary-style document. The input is data from the past week, and the AI ​​model generates sentences. The output is a diary-style document.

[1337] Step 14:

[1338] The server shares the created diary through social networking sites or dedicated applications. The input is the created diary document, which is sent to the social networking site API or application API. The output is the diary data that can be accessed by users.

[1339] In this way, it is possible to record the user's speech and actions in real time and provide personalized information.

[1340] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1341] The present invention combines an emotion engine with a system that automatically records the speech and objects held by a specific person and shares them via social networking services. A specific embodiment of this system is described below.

[1342] This system consists of a device (e.g., smartphone, home camera) and a server. It recognizes the user's (child's) speech and the objects they are holding, records them with a timestamp, and automatically shares them via social media. It also uses an emotion engine to recognize the user's emotional state, and records and shares it.

[1343] Audio recording

[1344] 1. Audio monitoring and recognition

[1345] The device is equipped with a voice recognition function that constantly monitors the sounds around the user (child). A voice recognition algorithm detects specific speech.

[1346] 2. Recording audio data

[1347] When the device recognizes the user's speech, it records that part of the audio, adds a timestamp, and temporarily saves the recorded data.

[1348] 3. Send to the server

[1349] The device sends the recorded voice data to a server, which stores the received voice data in a database and checks whether the voice is new by comparing it with past data.

[1350] 4. Analysis by Emotion Engine

[1351] The server analyzes the voice data with an emotion engine to identify the user's emotional state (e.g., joy, sadness, anger, etc.), and this emotion data is also stored along with the voice data.

[1352] 5. Sharing on social media

[1353] If the server recognizes the utterance as new, it generates a message containing emotional data for the social networking site. For example, it generates a message in the format "[October 1, 2023, 10:30] My child said 'Mom' (joy)" and automatically sends it to the social networking site.

[1354] Image recording

[1355] 1. Image monitoring and recognition

[1356] The device (home camera) activates image recognition and monitors the user's (child's) movements. It analyzes the camera footage to detect what is being held in the user's hand.

[1357] 2. Recording image data

[1358] When the device detects a new object, it takes a photo of that moment, adds a timestamp to it, and then temporarily stores this data.

[1359] 3. Send to the server

[1360] The device sends the captured image data to a server, which stores the received image data in a database and checks it against previous databases to see if it is a new item.

[1361] 4. Analysis by Emotion Engine

[1362] The server analyzes the image data using an emotion engine and identifies the user's emotional state from their facial expressions and movements. The emotion data is saved together with the image data.

[1363] 5. Sharing on social media

[1364] If the server recognizes the item as new, it generates a message containing emotional data for the social networking site. For example, it could generate a message in the format "[October 1, 2023, 14:30] My child brought home a new toy. (Surprised)" and automatically send it to the social networking site.

[1365] Diary creation using generative AI

[1366] 1. Data Collection

[1367] The server extracts audio data and image data for a certain period of time (e.g., one week) from the database.

[1368] 2. Diary-style document generation

[1369] The server uses generative AI to create a diary-style document based on the extracted data. Audio data is transcribed, image data is provided with explanatory text, and emotional data is also reflected in the diary.

[1370] 3. Share your diary

[1371] The created diary can be displayed and shared with the user and their family via social media or a dedicated application, allowing the user to check their growth record in diary format.

[1372] Specific examples

[1373] Audio recording example

[1374] When a user (child) says "Mom," the device recognizes this and records it. The recorded voice data is given a timestamp, such as "October 1, 2023, 10:30." When the server receives this data and analyzes it with its emotion engine, the emotion of joy is detected. The server then shares this information in the form of a message on social media. For example, a message could be sent that reads, "[October 1, 2023, 10:30] Child said 'Mom' (Joy)."

[1375] Image recording example

[1376] When a user (child) brings home a new toy, the device (home camera) recognizes this and takes a picture. The image data is given a timestamp, such as "October 1, 2023, 14:30." When the server receives this data and analyzes it with its emotion engine, the emotion of surprise is detected. The server then shares this information in the form of a message on social media. For example, a message could be sent saying, "[October 1, 2023, 14:30] Your child brought home a new toy. (Surprised)"

[1377] This system allows users to record both their growth and emotions in real time and share them automatically. Generative AI also creates diary-style documents based on the saved data, which can be shared with family members, providing a more detailed and emotional record of their growth.

[1378] The processing flow will be explained below.

[1379] Audio recording

[1380] Step 1:

[1381] The device (smartphone app) activates its voice recognition module and constantly monitors the sounds around the user (child), acquiring the voice data stream and preparing to analyze the voice patterns in real time.

[1382] Step 2:

[1383] The device detects speech. For example, if a voice pattern of "mama" is detected, the voice recognition module trims that part of the audio and recognizes it as speech.

[1384] Step 3:

[1385] The device temporarily stores the recognized speech in memory and adds the current date and time as a timestamp to the recorded data.

[1386] Step 4:

[1387] The device sends time-stamped audio data to the server, which then encrypts it and transfers it securely over the network to the server.

[1388] Step 5:

[1389] The server stores the received voice data in a database, and an algorithm runs to check whether the same utterance exists in the database by comparing it with previous records.

[1390] Step 6:

[1391] The server analyzes the voice data with an emotion engine to identify the user's emotional state (e.g., joy, sadness, anger, etc.). The emotion data is stored together with the voice data.

[1392] Step 7:

[1393] If the server recognizes the utterance as new, it generates a message containing emotional data for the social networking site. For example, it generates a message in the format "[October 1, 2023, 10:30] My child said 'Mom' (joy)" and automatically sends it to the social networking site.

[1394] Image recording

[1395] Step 1:

[1396] The device (home camera) activates the image recognition module to monitor the user's (child's) movements, preparing to analyze the camera footage in real time to detect objects being held in the hands.

[1397] Step 2:

[1398] The device recognizes that the user (child) is holding a new object, and the image recognition module identifies the object and captures an image.

[1399] Step 3:

[1400] The terminal temporarily stores the captured image in memory, and adds the current date and time as a timestamp to the image data.

[1401] Step 4:

[1402] The device sends image data with a timestamp to the server, where it is encrypted and securely transferred to the server over the network.

[1403] Step 5:

[1404] The server stores the received image data in a database, where an algorithm runs to check against previous databases to see if the same object has been recorded before.

[1405] Step 6:

[1406] The server analyzes the image data using an emotion engine and identifies the user's emotional state from their facial expressions and movements. The emotion data is saved together with the image data.

[1407] Step 7:

[1408] If the server recognizes the item as new, it generates a message containing emotional data for the social networking site. For example, it could generate a message in the format "[October 1, 2023, 14:30] My child brought home a new toy. (Surprised)" and automatically send it to the social networking site.

[1409] Diary creation using generative AI

[1410] Step 1:

[1411] The server extracts audio data and image data for a certain period of time (e.g., one week) from the database.

[1412] Step 2:

[1413] The server uses generative AI to create a diary-style document based on the extracted data. Audio data is transcribed, and image data is provided with explanatory text. Emotional data is also reflected in the diary.

[1414] Step 3:

[1415] The server uploads the created diary to social media or a dedicated application, allowing users and their families to view the child's growth record in diary format.

[1416] Step 4:

[1417] Users can view and share the diary entries they create with their families via social media or a dedicated application. The entries are organized with timestamps and include emotional records, making them an accurate and emotional record of their child's growth.

[1418] Example 2

[1419] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1420] Currently, there is a lack of systems that can accurately reflect a user's emotional state and appropriately record and share what a specific person is saying or holding. Furthermore, there is a lack of functionality to automatically generate diary-style documents based on recorded data and share them with the user and their family. This makes it difficult to record a user's growth in a comprehensive and emotional way.

[1421] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes: means for monitoring the voice of a specific person, recognizing the speech, assigning a timestamp, and recording the voice data; means for monitoring an object held by the specific person, recognizing the speech, assigning a timestamp, and recording image data; means for transmitting the recorded voice data and image data to the server and storing them in a database; means for analyzing the emotional state contained in the stored voice data and image data; and means for generating a message including the analyzed emotional data and automatically sharing it via an SNS. This makes it possible to share messages reflecting the emotional state of the user on an SNS by recording the user's speech and belongings. In addition, a diary-style document automatically generated based on the collected data allows the user to emotionally record their growth.

[1422] "Voice data" refers to information on voice that has been recorded by recognizing the speech of a specific person and adding a timestamp.

[1423] "Image data" is image information that is recorded by recognizing an object held by a specific person and adding a timestamp.

[1424] A "timestamp" is information that is added to audio data and image data and indicates the date and time of recording.

[1425] A "server" is a computer system that receives recorded audio and image data, stores it in a database, and performs further processing.

[1426] "Database" refers to an information system for storing and managing recorded audio and video data, as well as related emotional data.

[1427] An "emotion engine" is an algorithm and software that analyzes audio and image data to identify the emotional state of a particular person.

[1428] "SNS" is a social networking service that allows users to share recorded audio data, image data, and emotional data with other users.

[1429] "Message generation" is the process of automatically creating messages to be shared on social media based on analyzed emotional data.

[1430] "Artificial intelligence" refers to algorithms and software technology that analyzes large amounts of data and automatically learns and infers.

[1431] A "diary-style document" is written information that summarizes events and emotions over a specific period of time, generated based on saved audio and image data.

[1432] "Specialized Application" means a specific software application for viewing and sharing the generated diary-style documents.

[1433] This invention combines an emotion engine with a system that automatically records what a specific person is saying and what they are holding, and shares that information via social networking services. Specifically, the system records audio and image data, stores it in a database, analyzes emotions, and generates shared messages that are then automatically posted to social networking services.

[1434] This system mainly consists of a terminal and a server. The terminal can be a smartphone or a home camera. The server is a cloud-based computer system that works in conjunction with a database for processing and storing large amounts of data.

[1435] First, the device has a voice recognition function to constantly monitor the user's voice. For example, it uses voice recognition technology such as Google Speech-to-Text to analyze the user's speech in real time. When it detects a specific utterance (e.g., "Mom"), it records the voice data, adds a timestamp, and temporarily stores it.

[1436] Next, the device uses the home camera to monitor what the user is holding in their hand. Using image recognition technology (e.g., OpenCV), it analyzes what the user is holding in real time, and when it recognizes a new object, it takes a picture of that moment, adds a timestamp to it, and temporarily stores it.

[1437] The voice and image data recorded on the device are periodically sent to a server. The server stores the received data in a database and checks whether the information is new or not from the past database. The server also uses an emotion engine (e.g., IBM Watson Tone Analyzer or Affectiva) to identify the user's emotional state from the voice and image data. This emotion data is also stored along with the data.

[1438] When a new utterance or item is recognized, the server generates a message containing emotion data. For example, it generates a message in the format "[October 1, 2023, 10:30] My child said 'Mom' (joy)" and automatically posts it using a social networking API (e.g., Twitter API or Facebook API).

[1439] Furthermore, the server uses generative AI (e.g., OpenAI GPT-4) to create a diary-style document based on data from a certain period of time (e.g., one week). The voice data is transcribed, and explanatory text is added to the image data. The diary thus created can be shared with the user and their family via a dedicated application or social media. A specific prompt could be, "Please transcribe one week's worth of the user's voice data and compile it into a diary-style document along with emotional data."

[1440] This system allows users to record their growth and emotional state in real time, automatically generating and sharing emotionally rich growth records.

[1441] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1442] Step 1: Audio monitoring and recognition

[1443] The device constantly monitors the sounds around the user. A speech recognition algorithm (e.g., Google Speech-to-Text) is installed on the device, and it analyzes the user's speech in real time. It takes in the surrounding audio data as input and searches for specific utterances (e.g., "Mom"). If recognized, it extracts the utterance and outputs it as audio data.

[1444] Step 2: Recording audio data

[1445] The device records the voice data recognized in step 1 and assigns a timestamp. This voice data is temporarily stored in the device's local storage. Specifically, a timestamp such as "October 1, 2023, 10:30" is assigned to the voice data. The input is the recognized voice data, and the output is voice data with the timestamp.

[1446] Step 3: Send to the server

[1447] The device periodically transmits time-stamped voice data to the server. The transmitted voice data is stored in a database on the server. Specifically, the device uploads data to the server via the Internet. The input is the time-stamped voice data, and the output is the voice data stored in the database on the server.

[1448] Step 4: Emotion Engine Analysis

[1449] The server analyzes the stored voice data using an emotion engine (e.g., IBM Watson Tone Analyzer). It identifies the user's emotional state (e.g., joy, sadness, anger, etc.) from the voice data and generates emotion data. The input is the voice data stored on the server, and the output is emotion data. Specifically, the voice data is passed through an algorithm to analyze the emotional state.

[1450] Step 5: Share on social media

[1451] The server generates a message for social media based on the data recognized as new speech. A message containing emotion data is created in the following format, for example: "[October 1, 2023, 10:30] My child said 'Mom' (joy)." This message is automatically posted using a social media API (e.g., Twitter API). The input is voice data with emotion data, and the output is a post on social media. The specific operation is to post the message via the API.

[1452] Step 6: Image monitoring and recognition

[1453] The device (home camera) monitors the user's movements. It analyzes the camera image in real time and uses an image recognition algorithm (e.g., OpenCV) to detect what is being held in the user's hand. The input is live video data, and the output is recognized image data. As a concrete example, it detects the moment when the user picks up a new toy.

[1454] Step 7: Recording image data

[1455] The device assigns a timestamp to the recognized image data and records it. This data is temporarily stored in local storage. The recognized image data is used as input, and image data with a timestamp is generated as output. Specifically, the device assigns a timestamp such as "October 1, 2023, 14:30" to the image.

[1456] Step 8: Send to the server

[1457] The terminal sends the image data with the timestamp to the server. The sent data is stored in a database on the server. The input is the image data with the timestamp, and the output is the image data stored on the server. In concrete terms, the data is uploaded to the server via the Internet.

[1458] Step 9: Analysis by Emotion Engine (Image)

[1459] The server analyzes the stored image data using an emotion engine (e.g., Affectiva). This identifies the user's emotional state from their facial expressions and movements. The input is the image data stored on the server, and the output is emotion data. Specifically, the image data is passed through an algorithm to analyze the emotional state.

[1460] Step 10: Share on social media (images)

[1461] The server generates a message containing emotional data based on the data recognized as a new item. For example, it creates a message in the format "[October 1, 2023, 14:30] My child brought home a new toy. (Surprised)" and automatically posts it using a Facebook API or similar. The input is image data with emotional data, and the output is a post on social media. The specific operation is to post the message via the API.

[1462] Step 11: Data collection

[1463] The server extracts audio and image data for a certain period (e.g., one week) from a database. The input is a request specifying a specific period, and the output is the data collected during that period. Specifically, the data is extracted through a database query.

[1464] Step 12: Diary-style document generation

[1465] The server uses a generative AI (e.g., OpenAI GPT-4) to create a diary-style document based on the extracted data. As a specific example, the prompt is "Please transcribe one week's worth of user voice data and compile it into a diary-style document along with emotional data." The input is the collected voice data and image data, and the output is a diary-style document.

[1466] Step 13: Share your diary

[1467] The generated diary can be displayed and shared with the user or family members through a dedicated application or social networking site. The input is a diary-format document, and the output is a display on the application or a post on the social networking site. The specific operation involves posting the generated document to the application or social networking site.

[1468] (Application example 2)

[1469] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1470] To strengthen home security, it is important to be able to detect suspicious behavior and abnormal emotional states in real time and respond quickly to them. However, current home security systems do not adequately identify abnormal behavior and emotional states, which can lead to delayed responses in emergencies. In addition, there is a lack of means to quickly communicate detected information to family members or administrators, which delays information sharing and makes it difficult to take appropriate action.

[1471] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for recognizing the speech of a specific person, recording the voice data with a timestamp, recognizing an object held by the specific person, recording image data with a timestamp, saving the recorded voice data and image data in a database and automatically sharing them via SNS, and means for monitoring abnormal behavior and emotional states in specific locations in real time and issuing an alarm. This makes it possible to quickly detect abnormal behavior and emotional states within the home and share information in real time.

[1472] "Means for recognizing the speech of a specific person, adding a timestamp and recording the voice data" is a function that uses voice recognition technology to detect the speech of a selected person and saves the time at which the speech was made, along with a record of the time at which it was made.

[1473] "Means for recognizing an object held by a specific person, assigning a timestamp and recording image data" is a function that uses image analysis technology to detect an object held by a selected subject and saves a record of the time the object was held.

[1474] "Means for storing recorded audio and image data in a database and automatically sharing it through social networking services" refers to a function for building a database for systematically storing collected audio and image data and automatically sharing that data through social networking services.

[1475] "Means for monitoring abnormal behavior and emotional states in a specific location in real time and issuing an alarm" refers to a function that constantly monitors abnormal behavior and emotional states occurring within a designated space and issues an alarm based on the results.

[1476] "A means of using generative AI to create diary-style documents based on saved voice and image data" is a function that uses a machine learning model to generate natural-looking sentences based on collected voice and image data, and compiles them in diary format.

[1477] "Means for displaying and sharing the created diary through social networking services and dedicated applications" refers to a function for displaying and sharing the created diary using social networking services and specific mobile applications.

[1478] This invention is a system that automatically records what specific people say and what they hold in their hands, and shares that data in real time to enhance home security. The system mainly consists of a terminal, a server, and a generative AI model.

[1479] Terminal part

[1480] The device can be a smartphone or a home camera, and has the following main functions:

[1481] 1. Audio Recording:

[1482] It has a voice recognition function that constantly monitors the user's speech.

[1483] When a specific utterance is detected, the audio for that part is recorded and a timestamp is added.

[1484] The recorded data is temporarily stored and later sent to the server.

[1485] 2. Taking an image:

[1486] Cameras with image recognition capabilities for monitoring activity within the home.

[1487] When it detects that a specific object is being held in your hand, it takes an image of that moment and adds a timestamp.

[1488] The captured image data is also temporarily stored and later sent to the server.

[1489] Server part

[1490] The server processes the received audio and image data and generates the diary using generation AI.

[1491] 1. Data storage and collation:

[1492] The server stores the audio data and image data transmitted from the terminal in a database.

[1493] It is compared with past data to check whether the utterance or the item is new.

[1494] 2. Emotion engine analysis:

[1495] The audio and image data is analyzed to determine the user's emotional state.

[1496] This emotion data is also stored in the database.

[1497] 3. Share on social media:

[1498] When new data is recognized, a message containing the emotion data is generated for the SNS.

[1499] For example, it could automatically send a message like, "[October 1, 2023, 10:30] The sound of 'joy' was detected in your home."

[1500] Generative AI Models

[1501] The generative AI model automatically generates diaries from data stored on the server.

[1502] 1. Data Collection:

[1503] The server extracts audio data and image data for a certain period of time (for example, one week).

[1504] 2. Diary generation:

[1505] Generative AI is used to create diary-style documents based on collected data.

[1506] The audio data is transcribed and the image data is accompanied by explanatory text.

[1507] Emotional data is also reflected, providing a more detailed growth record.

[1508] 3. Share your diary:

[1509] The created diary can be displayed and shared via social media or dedicated applications.

[1510] For example, if suspicious activity is detected in the middle of the night in a home, the system will immediately record and analyze the change, and then send a message to social media saying, "October 1, 2023, 2:30 AM: A 'frightening' sound has been detected in the home," alerting the entire family.

[1511] Example prompts to input to a generative AI model:

[1512] "Please generate a sample program for an application that uses the emotion recognition system to monitor and share abnormal behavior and emotional states within the home in real time."

[1513] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1514] Step 1:

[1515] The terminal recognizes the user's speech and records the voice data.

[1516] Input: User utterance

[1517] How it works: The device's voice recognition function constantly monitors the user's speech, and when it detects a specific utterance, it records that part of the audio and adds a timestamp.

[1518] Output: Audio data with timestamps

[1519] Step 2:

[1520] The device temporarily stores the recorded audio data and sends it to the server.

[1521] Input: Audio data with timestamps

[1522] Operation: The audio data temporarily stored on the device is sent to the server.

[1523] Output: Audio data sent to the server

[1524] Step 3:

[1525] The server receives the audio data and stores it in a database.

[1526] Input: Audio data sent to the server

[1527] How it works: The server stores the received voice data in a database and checks it against past databases to see if it is a new utterance.

[1528] Output: Audio data stored in a database

[1529] Step 4:

[1530] The server analyzes the voice data using an emotion engine to identify the user's emotional state.

[1531] Input: Audio data stored in the database

[1532] How it works: The server analyzes the voice data with an emotion engine to determine the user's emotional state.

[1533] Output: Emotion data

[1534] Step 5:

[1535] The server generates a message for SNS based on the voice data and emotion data and automatically sends it.

[1536] Input: Voice data, emotion data

[1537] Behavior: If the server recognizes a new utterance, it generates a message for SNS containing emotion data. Example: "[October 1, 2023, 10:30] The voice of 'joy' was detected in the home."

[1538] Output: Message sent to SNS

[1539] Step 6:

[1540] The device recognizes what the user is holding in their hand and takes a picture.

[1541] Input: User action

[1542] How it works: When the device's camera recognizes an object being held by the user, it takes a picture of that moment and adds a timestamp.

[1543] Output: Image data with timestamp

[1544] Step 7:

[1545] The device temporarily stores the captured image data and sends it to the server.

[1546] Input: Image data with timestamp

[1547] Operation: Image data temporarily stored on the device is sent to the server.

[1548] Output: Image data sent to the server

[1549] Step 8:

[1550] The server receives the image data and stores it in a database.

[1551] Input: Image data sent to the server

[1552] How it works: The server stores the image data it receives in a database and checks it against previous databases to see if it is new.

[1553] Output: Image data stored in a database

[1554] Step 9:

[1555] The server analyzes the image data using an emotion engine to identify the user's emotional state.

[1556] Input: Image data stored in a database

[1557] How it works: The server analyzes the image data with an emotion engine to determine the user's emotional state.

[1558] Output: Emotion data

[1559] Step 10:

[1560] The server generates a message for SNS based on the image data and emotional data and automatically sends it.

[1561] Input: image data, emotion data

[1562] Behavior: If the server recognizes a new possession, it generates a message for social media containing emotion data. Example: "[October 1, 2023, 14:30] 'Surprising' images detected in the home."

[1563] Output: Message sent to SNS

[1564] Step 11:

[1565] The server analyzes a certain period of audio and image data using a generative AI model and creates a diary-style document.

[1566] Input: Audio data and image data for a certain period of time

[1567] How it works: The server uses a generative AI model to create a diary-style document based on the extracted data. Audio data is transcribed, image data is annotated with explanatory text, and emotion data is also reflected in the diary.

[1568] Output: Diary-style document

[1569] Step 12:

[1570] The server displays and shares the created diary document through social networking sites or dedicated applications.

[1571] Input: diary-style document

[1572] How it works: The server displays and shares the created diary with the user and their family via social media or a dedicated application.

[1573] Output: View and share diary entries

[1574] Example prompts to input to a generative AI model:

[1575] "Please generate a sample program for an application that uses the emotion recognition system to monitor and share abnormal behavior and emotional states within the home in real time."

[1576] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1577] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1578] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1579] [Fourth embodiment]

[1580] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1581] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1582] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1583] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1584] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1585] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1586] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1587] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1588] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1589] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1590] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1591] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1592] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1593] The present invention is a system that automatically records what a specific person is saying and what they are holding, and shares that information via social networking services. A specific embodiment of this system will be described below.

[1594] This system consists of a device (e.g., a smartphone or home camera) and a server. It recognizes the user's (child's) speech and the objects they are holding, records them with a timestamp, and automatically shares them via social media.

[1595] Audio recording

[1596] 1. Audio monitoring and recognition

[1597] The device is equipped with a voice recognition function that constantly monitors the sounds around the user (child). The voice recognition algorithm detects specific utterances (e.g., "Mom," "Dad," etc.).

[1598] 2. Recording audio data

[1599] When the device recognizes a user's speech, it records that portion of the audio and adds a timestamp to it, which clearly indicates the date and time of the recording.

[1600] 3. Send to the server

[1601] The device sends the recorded voice data to a server, which stores the received voice data in a database and checks whether the same utterance has been recorded in the past.

[1602] 4. Sharing on social media

[1603] If the server recognizes the utterance as new, it will generate a message to share on SNS (e.g., LINE) and automatically send it. The shared content will include a timestamp, indicating the date and time of the utterance.

[1604] Image recording

[1605] 1. Image monitoring and recognition

[1606] The device (e.g., a home camera) monitors the user's (child's) movements and uses image recognition to identify the object being held.

[1607] 2. Recording image data

[1608] When the device recognizes a new item, it takes a picture and adds a timestamp to it. For example, if a user brings home a new toy, it will record an image of that exact moment.

[1609] 3. Send to the server

[1610] The device sends the captured image data to a server, which stores the received image data in a database and checks whether the same object has been recorded previously.

[1611] 4. Sharing on social media

[1612] If the server recognizes the item as new, it will automatically generate and send a message to share on social media, including a timestamp to indicate when the item was picked up.

[1613] Diary creation using generative AI

[1614] 1. Data Collection

[1615] The server extracts audio data and image data for a certain period (e.g., one week) from the database.

[1616] 2. Diary-style document generation

[1617] The server uses generative AI to create diary-style documents based on the extracted data, transcribes audio data, and provides explanatory text for image data.

[1618] 3. Share your diary

[1619] The created diary can be displayed and shared with users and their families via social media or a dedicated application, allowing users to check their child's growth record in diary format.

[1620] Specific examples

[1621] Audio recording example

[1622] When a user (child) says "Mom," the device recognizes this and records it. The recorded voice data is given a timestamp, such as "October 1, 2023, 10:30." The server receives this data, compares it with its existing database, and if it determines that it has not been recorded, shares the content on a social networking site (e.g., Line) with the message, "[October 1, 2023, 10:30] My child said 'Mom.'"

[1623] Image recording example

[1624] When a user (child) brings home a new toy, the device (home camera) recognizes this and takes a picture. The image data is given a timestamp, such as "October 1, 2023, 14:30." The server receives this data, compares it with its existing database, and if it determines that it has not been recorded, it shares the message on social media, saying, "[October 1, 2023, 14:30] Your child has brought home a new toy."

[1625] This system allows for real-time recording of a child's growth and automatically sharing it. It also uses generative AI to create diary-style documents based on the saved data, which can be shared with family members, providing a more detailed record of a child's growth.

[1626] The processing flow will be explained below.

[1627] Audio recording

[1628] Step 1:

[1629] The device (smartphone app) activates its voice recognition module and constantly monitors the sounds around the user (child), acquiring the voice data stream and preparing to analyze the voice patterns in real time.

[1630] Step 2:

[1631] The device detects a specific speech pattern, for example, "Mama," and the speech recognition module trims that portion of the audio and recognizes it as speech.

[1632] Step 3:

[1633] The device temporarily stores the recognized speech in memory and timestamps the recording with the current date and time, ensuring an accurate recording of the speech.

[1634] Step 4:

[1635] The device sends time-stamped audio data to the server, which then encrypts it and transfers it securely over the network to the server.

[1636] Step 5:

[1637] The server stores the received voice data in a database, and an algorithm runs to check whether the same utterance exists in the database by comparing it with previous records.

[1638] Step 6:

[1639] When the server recognizes a new utterance, it converts the information into a message for use on social media. For example, it generates a message in the format: "[October 1, 2023, 10:30] My child said 'Mom.'"

[1640] Step 7:

[1641] The server automatically sends the generated message to an SNS (e.g., Line), and the recipient can receive the message in the specified account.

[1642] Image recording

[1643] Step 1:

[1644] The device (home camera) activates the image recognition module and constantly monitors the user's (child's) movements, preparing to analyze the camera footage in real time to detect objects being held.

[1645] Step 2:

[1646] The device recognizes that the user (child) is holding a new object, and the image recognition module identifies the object and captures an image.

[1647] Step 3:

[1648] The device temporarily stores the captured image in memory and timestamps the image data with the current date and time, ensuring an accurate record of the photo.

[1649] Step 4:

[1650] The device sends image data with a timestamp to the server, where it is encrypted and securely transferred to the server over the network.

[1651] Step 5:

[1652] The server stores the received image data in a database, where an algorithm runs to check against previous databases to see if the same object has been recorded before.

[1653] Step 6:

[1654] When the server recognizes a new item, it converts that information into a message for social media, such as "My child brought home a new toy [October 1, 2023, 14:30]."

[1655] Step 7:

[1656] The server automatically sends the generated message to an SNS (e.g., Line), and the recipient can receive the message in the specified account.

[1657] Diary creation using generative AI

[1658] Step 1:

[1659] The server extracts audio and image data for a certain period (e.g., one week) from the database, and the extracted data is processed by the AI ​​module.

[1660] Step 2:

[1661] The server uses generative AI to create diary-style documents based on the extracted data, automatically transcribes audio data, and provides explanatory text for image data.

[1662] Step 3:

[1663] The server uploads the created diary to social networking sites or dedicated applications, allowing users and their families to view their child's growth record in diary format.

[1664] Step 4:

[1665] Users can view and share the diary entries they create with their families via social media or a dedicated application. This information is organized with timestamps and can be used as an accurate record of their child's growth.

[1666] Example 1

[1667] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1668] In recent years, there has been a growing need to record children's developmental progress at home and share it with family members. However, conventional methods require time-consuming manual recording and sharing, and often lack real-time performance and accuracy. Furthermore, managing images and audio individually is cumbersome, creating a need for a unified recording method. Furthermore, recorded data needs to be displayed in an easily understandable and organized format when reviewing it later.

[1669] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1670] In this invention, the server includes means for recognizing the speech of a specific person using voice recognition technology, recording the voice data with a timestamp, means for recognizing an object held by the specific person using image recognition technology, recording image data with a timestamp, and means for saving the recorded voice data and image data in a database and automatically sharing them via SNS. This makes it possible to record a child's growth process in real time and automatically share it.

[1671] "Voice recognition technology" is a technology that identifies the speech of a specific person in real time and converts it into text format.

[1672] A "timestamp" is information indicating the date and time when data was recorded, and includes the date and time.

[1673] "Audio data" refers to recorded audio information stored in digital format.

[1674] "Image recognition technology" is a technology that uses computer vision algorithms to automatically identify objects and features in images.

[1675] "Image data" refers to photographs or videos taken by a device such as a camera and stored in digital format.

[1676] A "database" is a system that structures and stores information such as audio data and image data, and allows for efficient management and retrieval.

[1677] "SNS" is an abbreviation for social networking service, a platform that supports communication and information sharing between users via the Internet.

[1678] "Generative AI technology" is an artificial intelligence technology that learns from large amounts of data and generates new data and content.

[1679] A "diary-style document" is a document that describes events that occurred during a specific period in chronological order.

[1680] A "specialized application" is software designed for a specific function or purpose, in this case, to display and share growth records.

[1681] This invention is a system that automatically records what a specific person is saying and what they are holding, and shares that information via social networking services. This system consists of a device (e.g., a smartphone or home camera) and a server. It recognizes what the user (child) is saying and what they are holding, records it with a timestamp, and automatically shares it on social networking services.

[1682] Audio recording

[1683] 1. Audio monitoring and recognition

[1684] The device uses voice recognition technology to constantly monitor the sounds around the user (child). Specifically, it uses the microphone on a smartphone or home camera and uses a voice recognition algorithm such as "Google Cloud Speech-to-Text" to detect specific utterances (e.g., "Mom" or "Dad") in real time.

[1685] 2. Recording audio data

[1686] When the device recognizes a specific utterance, it records that portion of the audio and adds a timestamp using Python's datetime module. For example, the recording date and time is added to the data in the format "October 1, 2023, 10:30." This data is temporarily stored on the device.

[1687] 3. Send to the server

[1688] The device sends the recorded voice data to the server via an HTTP POST request. When the server receives this data, it stores it in a database (e.g., MySQL) and compares it with past data to determine whether the utterance is new.

[1689] 4. Sharing on social media

[1690] If the server recognizes the utterance as new, it uses the API of a social networking service (e.g., Line) to generate and send a message stating, "[October 1, 2023, 10:30 AM] My child said 'Mom.'" This allows the content of the utterance to be shared with family members in real time.

[1691] Image recording

[1692] 1. Image monitoring and recognition

[1693] The device (e.g., a home camera) uses image recognition technology (e.g., OpenCV, TensorFlow) to constantly monitor the user's (child's) movements, thereby identifying the object being held in real time.

[1694] 2. Recording image data

[1695] When the device recognizes a new item, it takes a photo of that moment and adds a timestamp (e.g., "October 1, 2023, 14:30"), which is also temporarily stored on the device.

[1696] 3. Send to the server

[1697] The device sends the captured image data to the server using an HTTP POST request, which stores the data in a database and similarly compares it with past data to determine whether the item is new.

[1698] 4. Sharing on social media

[1699] If the server recognizes the item as new, it generates and sends a message to the social networking site saying, "[October 1, 2023, 14:30] Your child has brought home a new toy."

[1700] Diary creation using generative AI

[1701] 1. Data Collection

[1702] The server extracts audio and image data for a certain period (e.g., one week) from the database, and stores this data temporarily for diary generation.

[1703] 2. Diary-style document generation

[1704] The server uses generative AI technology (e.g., OpenAI GPT-4) to generate diary-style documents based on the extracted data. Audio data is transcribed and image data is annotated with explanatory text. This process is performed by inputting prompts to the generative AI model.

[1705] Example prompt sentence:

[1706] Create a diary of your child's growth based on one week's worth of audio and video data. Here is an example of the data:

[1707] Audio data:

[1708] 1. [October 1, 2023 10:30] "Mom"

[1709] 2. [October 2, 2023 11:00] "Dad"

[1710] Image data:

[1711] 1. [October 1, 2023 14:30] A user has come home with a new toy.

[1712] 2. [October 2, 2023, 15:00] A photo of a user playing in a park

[1713] Based on this data, please write a detailed diary of your child's growth.

[1714] 3. Share your diary

[1715] The created diary can be displayed and shared via social media or a dedicated application. The server converts the created diary into PDF or HTML format and generates a sharing link to provide to the user or their family.

[1716] As described above, the embodiments of the present invention have been specifically described, which allow users to seamlessly manage and share growth records, thereby stimulating communication within the home.

[1717] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1718] Processing Steps

[1719] Audio recording

[1720] Step 1: Audio monitoring and recognition

[1721] The device uses the microphone of a smartphone or home camera to constantly monitor the sounds around the user (child). It uses voice recognition technology (e.g., Google Cloud Speech-to-Text) to detect specific utterances (e.g., "Mom" or "Dad") in real time.

[1722] Input: Ambient sound captured by the microphone

[1723] Processing: Speech recognition algorithms detect specific utterances in real time

[1724] Output: A trigger event that a specific utterance was detected

[1725] Step 2: Recording audio data

[1726] When the device recognizes a specific utterance, it records that portion of the audio and adds a timestamp. Using the Python datetime module, it adds the recording date and time to the data in the format "October 1, 2023, 10:30." The audio data is temporarily stored on the device.

[1727] Input: Trigger event, audio data

[1728] Processing: Recording and time stamping audio data

[1729] Output: Audio data with timestamps

[1730] Step 3: Send to the server

[1731] The device sends the recorded voice data to the server via an HTTP POST request. The data is converted to JSON format and sent to the server's API endpoint. The server receives this data and stores it in a database (e.g., MySQL). A Python script is used to compare it with existing data and determine whether it is a new utterance.

[1732] Input: Time-stamped audio data

[1733] Processing: Convert to JSON format, send via HTTP POST request, save to database, and collate data

[1734] Output: New utterance determination result

[1735] Step 4: Share on social media

[1736] If the server determines that the utterance is new, it uses the API of the social networking service (e.g., Line) to generate and send a shared message. For example, it might send a message to the social networking service saying, "[October 1, 2023, 10:30] My child said 'Mom.'"

[1737] Input: New utterance determination result

[1738] Processing: Message creation and sending via SNS API

[1739] Output: Message sent to SNS

[1740] Image recording

[1741] Step 1: Image monitoring and recognition

[1742] The device (e.g., a home camera) constantly monitors the user's (child's) movements and uses image recognition technology (e.g., OpenCV, TensorFlow) to identify the object being held in real time.

[1743] Input: Image data obtained from the camera

[1744] Processing: Object recognition using image recognition algorithms

[1745] Output: Trigger event that new possession is recognized

[1746] Step 2: Recording image data

[1747] When the device recognizes a new item, it takes an image of that moment and uses the Python datetime module to add a timestamp (e.g., "October 1, 2023, 14:30"), which is also temporarily stored on the device.

[1748] Input: Trigger event, image data

[1749] Processing: Taking an image and adding a timestamp

[1750] Output: Image data with time stamp

[1751] Step 3: Send to the server

[1752] The device sends the captured image data to the server using an HTTP POST request. The data is converted to JSON format and sent to the server's API endpoint. The server receives this data and stores it in a database. It compares it with existing image data to determine if it is a new item.

[1753] Input: Image data with timestamp

[1754] Processing: Convert to JSON format, send via HTTP POST request, save to database, and collate data

[1755] Output: New possession determination result

[1756] Step 4: Share on social media

[1757] If the server determines that the item is new, it uses the SNS API to generate and send a sharing message. For example, it might send a message to the SNS saying, "[October 1, 2023, 14:30] Your child has brought home a new toy."

[1758] Input: New possession determination result

[1759] Processing: Message creation and sending via SNS API

[1760] Output: Message sent to SNS

[1761] Diary creation using generative AI

[1762] Step 1: Data collection

[1763] The server extracts audio and image data for a certain period (e.g., one week) from the database, and stores this data temporarily for diary generation.

[1764] Input: Audio data and image data in the database

[1765] Processing: Data extraction, temporary storage

[1766] Output: Data for diary generation

[1767] Step 2: Creating a diary-style document

[1768] The server uses generative AI technology (e.g., OpenAI GPT-4) to generate diary-style documents based on the extracted data. Audio data is transcribed and image data is annotated with explanatory text. This process is performed by inputting prompts into the generative AI model.

[1769] Input: Data for diary generation, prompt text

[1770] Processing: Generative AI models generate text, transcribe audio data, and add descriptions to image data

[1771] Output: Diary-style document

[1772] Step 3: Share your diary

[1773] The created diary can be displayed and shared via social media or a dedicated application. The server converts the created diary into PDF or HTML format and generates a sharing link to provide to the user or their family.

[1774] Input: diary-style document

[1775] Processing: Convert to PDF or HTML format, generate a link for sharing, send via social media or dedicated applications

[1776] Output: Shareable link, diary display on social media or dedicated application

[1777] This is the specific processing flow of the program for this system, which allows users to automatically manage and share their child's growth record in real time through audio and images.

[1778] (Application example 1)

[1779] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1780] In traditional brick-and-mortar stores, it is possible for customers to talk about products and check the products they have picked up, but there is a lack of a way to properly record customer behavior and the content of their conversations and provide personalized information in real time. It is also difficult to store customer consumption behavior in a database and use it for effective marketing or individualized support. For these reasons, a new system was needed to improve the customer experience and streamline store operations.

[1781] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1782] In this invention, the server includes means for recognizing the speech of a specific person, assigning a timestamp and recording the speech data, means for recognizing an object held by a specific person, assigning a timestamp and recording image data, means for saving the recorded speech data and image data in a database and automatically sharing them via social networking services, means for displaying the speech data and image data on an information display device in real time, and means for generating diary-style documents using a generative AI model based on past records of speech data and image data. This makes it possible to record customer speech and behavior in real time and provide personalized information, thereby improving the efficiency of store operations and the customer experience.

[1783] "Specified Person" refers to the person who will be monitored and recorded by the system.

[1784] "Utterance" refers to the sound produced by a particular person.

[1785] A "timestamp" refers to information indicating the date and time when audio or image data was recorded.

[1786] "Audio data" refers to an audio file that records the speech of a specific person.

[1787] "Handheld object" refers to an object held in the hand of a particular person.

[1788] "Image data" refers to an image file that captures an object being held by a specific person.

[1789] "Database" refers to a system for storing and managing audio and image data.

[1790] "SNS" is an abbreviation for social networking service, and refers to an online platform where users can share information with each other.

[1791] "Automatically sharing" means that the system automatically sends data to social media sites, etc., without the need for human operation.

[1792] "Information display device" refers to a device for visually displaying audio data and image data.

[1793] A "generative AI model" refers to an algorithm that uses machine learning techniques to generate sentences or documents from input data.

[1794] A "diary-style document" refers to a file that records events in chronological order and is organized as text.

[1795] "Personalized information" refers to information and services that are customized for a specific individual.

[1796] "Recording" refers to saving audio data or image data.

[1797] "Displayed in real time" means that data is displayed as soon as it is generated, with little delay.

[1798] The system that realizes this application example has the functions of recognizing the speech of a specific person, recording the voice data with a timestamp, recognizing the object held by the specific person, recording the image data with a timestamp, and storing the recorded voice and image data in a database, which can then be automatically shared via social networking sites.

[1799] Hardware and software used

[1800] This system uses the following hardware and software:

[1801] Microphone: A device for collecting speech from a specific person. It is built into smart glasses.

[1802] Camera: A device that takes pictures of objects held by a specific person. It is built into smart glasses.

[1803] Smart glasses: Information display devices that display speech and image data in real time.

[1804] Server: A computer system that processes and stores data and manages sharing to social media.

[1805] For example, we use the Python speech_recognition library for speech recognition, the cv2 (OpenCV) and pytesseract (OCR) libraries for image recognition, and the requests library for data processing and transmission.

[1806] Data processing and calculation

[1807] 1. Speech Recognition:

[1808] A user wears smart glasses and the microphone collects the speech of a specific person.

[1809] The audio data collected by the microphone is converted to text using the speech_recognition library, which is then time-stamped and the resulting audio data is generated.

[1810] The generated voice data is immediately sent to the server and stored in a database.

[1811] 2. Image Recognition:

[1812] A camera captures an image of an object being held by a particular person.

[1813] The captured image data is then extracted using the cv2 and pytesseract libraries, which then timestamp the image and generate the resulting image data.

[1814] The generated image data is immediately sent to a server and stored in a database.

[1815] 3. Save to database and share on social media:

[1816] The server stores the received audio data and image data in a database.

[1817] New data stored in the database is automatically shared via social media. For example, if a voice utterance such as "What do you think of this product?" is registered, that information will be posted to social media in real time.

[1818] 4. Generative AI diary creation:

[1819] The server extracts past audio and image data for a certain period of time (e.g., one week) and generates a diary-style document using a generative AI model.

[1820] The created diary can be shared with users via social media or dedicated applications.

[1821] Specific examples

[1822] For example, if a customer wearing smart glasses walks through a store and says, "What are the characteristics of this wine?", the microphone collects the voice and the voice recognition software converts it into text. This voice data is time-stamped and sent to the server. In addition, the camera also captures information about the wine bottle the customer picks up, and the image recognition software analyzes the content and sends it to the server.

[1823] The collected data is processed on the server and automatically shared on social media in the form of, for example, "[Date and time] A customer said, 'What are the features of this wine?' Wine information: Red wine, 750ml." At the end of the week, the generative AI model generates a diary-style document based on past data, which is shared via social media and dedicated applications with content such as, "This week's new customer interest product was red wine."

[1824] Prompt Sentence Examples

[1825] What is special about this wine?

[1826] "Tell me more about this product."

[1827] - "What are the new customer interest products this week?"

[1828] This process allows the system to record customer utterances and behavior in real time and provide personalized information, making it a powerful tool for improving customer experience and streamlining store operations.

[1829] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1830] Step 1:

[1831] The microphone in the smart glasses collects the user's speech. The microphone constantly monitors the surrounding sound of a specific person and captures voice data. The input is voice waveform data, and the output is an audio file.

[1832] Step 2:

[1833] Convert audio files to text using the speech_recognition library. Audio data is input and text data is output. Specifically, the speech recognition algorithm analyzes the waveform of the audio and converts it into meaningful words.

[1834] Step 3:

[1835] A timestamp is added to the converted text data. The input is text data obtained by speech recognition, and the current date and time are added as a timestamp. The output is text data with a timestamp.

[1836] Step 4:

[1837] Sends text data with a timestamp to the server. Audio data and a timestamp are input and sent to the server as an HTTP request. The output is the data received on the server side.

[1838] Step 5:

[1839] The server stores the received audio data in a database. The input is time-stamped audio data that is inserted into a database table. The output is an audio recording stored in the database.

[1840] Step 6:

[1841] The camera in the smart glasses takes a picture of the object held by the user. The camera is constantly capturing images of the surroundings, and the input is image data. The output is an image file.

[1842] Step 7:

[1843] Extract text information from an image using the cv2 and pytesseract libraries. The input is image data, and the output is extracted text information. Specifically, OCR technology analyzes the text in the image and outputs it as a string of characters.

[1844] Step 8:

[1845] A timestamp is added to the converted image text data. The input is text data obtained by OCR, and the current date and time are added as a timestamp. The output is text data with a timestamp.

[1846] Step 9:

[1847] Send image text data with a timestamp to the server. Image data and a timestamp are input and sent to the server as an HTTP request. The output is the data received on the server side.

[1848] Step 10:

[1849] The server stores the received image text data in a database. The input is time-stamped image data that is inserted into a database table. The output is an image record stored in the database.

[1850] Step 11:

[1851] The server automatically posts new data to SNS. The input is new data stored in the database, and the server calls the SNS API to generate a message to post. The output is the message posted to SNS.

[1852] Step 12:

[1853] Periodically, the server extracts historical audio and image data. The input is the past week's worth of data from the database, and the output is an extracted data set.

[1854] Step 13:

[1855] The server sends the extracted data to a generative AI model to generate a diary-style document. The input is data from the past week, and the AI ​​model generates sentences. The output is a diary-style document.

[1856] Step 14:

[1857] The server shares the created diary through social networking sites or dedicated applications. The input is the created diary document, which is sent to the social networking site API or application API. The output is the diary data that can be accessed by users.

[1858] In this way, it is possible to record the user's speech and actions in real time and provide personalized information.

[1859] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1860] The present invention combines an emotion engine with a system that automatically records the speech and objects held by a specific person and shares them via social networking services. A specific embodiment of this system is described below.

[1861] This system consists of a device (e.g., smartphone, home camera) and a server. It recognizes the user's (child's) speech and the objects they are holding, records them with a timestamp, and automatically shares them via social media. It also uses an emotion engine to recognize the user's emotional state, and records and shares it.

[1862] Audio recording

[1863] 1. Audio monitoring and recognition

[1864] The device is equipped with a voice recognition function that constantly monitors the sounds around the user (child). A voice recognition algorithm detects specific speech.

[1865] 2. Recording audio data

[1866] When the device recognizes the user's speech, it records that part of the audio, adds a timestamp, and temporarily saves the recorded data.

[1867] 3. Send to the server

[1868] The device sends the recorded voice data to a server, which stores the received voice data in a database and checks whether the voice is new by comparing it with past data.

[1869] 4. Analysis by Emotion Engine

[1870] The server analyzes the voice data with an emotion engine to identify the user's emotional state (e.g., joy, sadness, anger, etc.), and this emotion data is also stored along with the voice data.

[1871] 5. Sharing on social media

[1872] If the server recognizes the utterance as new, it generates a message containing emotional data for the social networking site. For example, it generates a message in the format "[October 1, 2023, 10:30] My child said 'Mom' (joy)" and automatically sends it to the social networking site.

[1873] Image recording

[1874] 1. Image monitoring and recognition

[1875] The device (home camera) activates image recognition and monitors the user's (child's) movements. It analyzes the camera footage to detect what is being held in the user's hand.

[1876] 2. Recording image data

[1877] When the device detects a new object, it takes a photo of that moment, adds a timestamp to it, and then temporarily stores this data.

[1878] 3. Send to the server

[1879] The device sends the captured image data to a server, which stores the received image data in a database and checks it against previous databases to see if it is a new item.

[1880] 4. Analysis by Emotion Engine

[1881] The server analyzes the image data using an emotion engine and identifies the user's emotional state from their facial expressions and movements. The emotion data is saved together with the image data.

[1882] 5. Sharing on social media

[1883] If the server recognizes the item as new, it generates a message containing emotional data for the social networking site. For example, it could generate a message in the format "[October 1, 2023, 14:30] My child brought home a new toy. (Surprised)" and automatically send it to the social networking site.

[1884] Diary creation using generative AI

[1885] 1. Data Collection

[1886] The server extracts audio data and image data for a certain period of time (e.g., one week) from the database.

[1887] 2. Diary-style document generation

[1888] The server uses generative AI to create a diary-style document based on the extracted data. Audio data is transcribed, image data is provided with explanatory text, and emotional data is also reflected in the diary.

[1889] 3. Share your diary

[1890] The created diary can be displayed and shared with the user and their family via social media or a dedicated application, allowing the user to check their growth record in diary format.

[1891] Specific examples

[1892] Audio recording example

[1893] When a user (child) says "Mom," the device recognizes this and records it. The recorded voice data is given a timestamp, such as "October 1, 2023, 10:30." When the server receives this data and analyzes it with its emotion engine, the emotion of joy is detected. The server then shares this information in the form of a message on social media. For example, a message could be sent that reads, "[October 1, 2023, 10:30] Child said 'Mom' (Joy)."

[1894] Image recording example

[1895] When a user (child) brings home a new toy, the device (home camera) recognizes this and takes a picture. The image data is given a timestamp, such as "October 1, 2023, 14:30." When the server receives this data and analyzes it with its emotion engine, the emotion of surprise is detected. The server then shares this information in the form of a message on social media. For example, a message could be sent saying, "[October 1, 2023, 14:30] Your child brought home a new toy. (Surprised)"

[1896] This system allows users to record both their growth and emotions in real time and share them automatically. Generative AI also creates diary-style documents based on the saved data, which can be shared with family members, providing a more detailed and emotional record of their growth.

[1897] The processing flow will be explained below.

[1898] Audio recording

[1899] Step 1:

[1900] The device (smartphone app) activates its voice recognition module and constantly monitors the sounds around the user (child), acquiring the voice data stream and preparing to analyze the voice patterns in real time.

[1901] Step 2:

[1902] The device detects speech. For example, if a voice pattern of "mama" is detected, the voice recognition module trims that part of the audio and recognizes it as speech.

[1903] Step 3:

[1904] The device temporarily stores the recognized speech in memory and adds the current date and time as a timestamp to the recorded data.

[1905] Step 4:

[1906] The device sends time-stamped audio data to the server, which then encrypts it and transfers it securely over the network to the server.

[1907] Step 5:

[1908] The server stores the received voice data in a database, and an algorithm runs to check whether the same utterance exists in the database by comparing it with previous records.

[1909] Step 6:

[1910] The server analyzes the voice data with an emotion engine to identify the user's emotional state (e.g., joy, sadness, anger, etc.). The emotion data is stored together with the voice data.

[1911] Step 7:

[1912] If the server recognizes the utterance as new, it generates a message containing emotional data for the social networking site. For example, it generates a message in the format "[October 1, 2023, 10:30] My child said 'Mom' (joy)" and automatically sends it to the social networking site.

[1913] Image recording

[1914] Step 1:

[1915] The device (home camera) activates the image recognition module to monitor the user's (child's) movements, preparing to analyze the camera footage in real time to detect objects being held in the hands.

[1916] Step 2:

[1917] The device recognizes that the user (child) is holding a new object, and the image recognition module identifies the object and captures an image.

[1918] Step 3:

[1919] The terminal temporarily stores the captured image in memory, and adds the current date and time as a timestamp to the image data.

[1920] Step 4:

[1921] The device sends image data with a timestamp to the server, where it is encrypted and securely transferred to the server over the network.

[1922] Step 5:

[1923] The server stores the received image data in a database, where an algorithm runs to check against previous databases to see if the same object has been recorded before.

[1924] Step 6:

[1925] The server analyzes the image data using an emotion engine and identifies the user's emotional state from their facial expressions and movements. The emotion data is saved together with the image data.

[1926] Step 7:

[1927] If the server recognizes the item as new, it generates a message containing emotional data for the social networking site. For example, it could generate a message in the format "[October 1, 2023, 14:30] My child brought home a new toy. (Surprised)" and automatically send it to the social networking site.

[1928] Diary creation using generative AI

[1929] Step 1:

[1930] The server extracts audio data and image data for a certain period of time (e.g., one week) from the database.

[1931] Step 2:

[1932] The server uses generative AI to create a diary-style document based on the extracted data. Audio data is transcribed, and image data is provided with explanatory text. Emotional data is also reflected in the diary.

[1933] Step 3:

[1934] The server uploads the created diary to social media or a dedicated application, allowing users and their families to view the child's growth record in diary format.

[1935] Step 4:

[1936] Users can view and share the diary entries they create with their families via social media or a dedicated application. The entries are organized with timestamps and include emotional records, making them an accurate and emotional record of their child's growth.

[1937] Example 2

[1938] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1939] Currently, there is a lack of systems that can accurately reflect a user's emotional state and appropriately record and share what a specific person is saying or holding. Furthermore, there is a lack of functionality to automatically generate diary-style documents based on recorded data and share them with the user and their family. This makes it difficult to record a user's growth in a comprehensive and emotional way.

[1940] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes: means for monitoring the voice of a specific person, recognizing the speech, assigning a timestamp, and recording the voice data; means for monitoring an object held by the specific person, recognizing the speech, assigning a timestamp, and recording image data; means for transmitting the recorded voice data and image data to the server and storing them in a database; means for analyzing the emotional state contained in the stored voice data and image data; and means for generating a message including the analyzed emotional data and automatically sharing it via an SNS. This makes it possible to share messages reflecting the emotional state of the user on an SNS by recording the user's speech and belongings. In addition, a diary-style document automatically generated based on the collected data allows the user to emotionally record their growth.

[1941] "Voice data" refers to information on voice that has been recorded by recognizing the speech of a specific person and adding a timestamp.

[1942] "Image data" is image information that is recorded by recognizing an object held by a specific person and adding a timestamp.

[1943] A "timestamp" is information that is added to audio data and image data and indicates the date and time of recording.

[1944] A "server" is a computer system that receives recorded audio and image data, stores it in a database, and performs further processing.

[1945] "Database" refers to an information system for storing and managing recorded audio and video data, as well as related emotional data.

[1946] An "emotion engine" is an algorithm and software that analyzes audio and image data to identify the emotional state of a particular person.

[1947] "SNS" is a social networking service that allows users to share recorded audio data, image data, and emotional data with other users.

[1948] "Message generation" is the process of automatically creating messages to be shared on social media based on analyzed emotional data.

[1949] "Artificial intelligence" refers to algorithms and software technology that analyzes large amounts of data and automatically learns and infers.

[1950] A "diary-style document" is written information that summarizes events and emotions over a specific period of time, generated based on saved audio and image data.

[1951] "Specialized Application" means a specific software application for viewing and sharing the generated diary-style documents.

[1952] This invention combines an emotion engine with a system that automatically records what a specific person is saying and what they are holding, and shares that information via social networking services. Specifically, the system records audio and image data, stores it in a database, analyzes emotions, and generates shared messages that are then automatically posted to social networking services.

[1953] This system mainly consists of a terminal and a server. The terminal can be a smartphone or a home camera. The server is a cloud-based computer system that works in conjunction with a database for processing and storing large amounts of data.

[1954] First, the device has a voice recognition function to constantly monitor the user's voice. For example, it uses voice recognition technology such as Google Speech-to-Text to analyze the user's speech in real time. When it detects a specific utterance (e.g., "Mom"), it records the voice data, adds a timestamp, and temporarily stores it.

[1955] Next, the device uses the home camera to monitor what the user is holding in their hand. Using image recognition technology (e.g., OpenCV), it analyzes what the user is holding in real time, and when it recognizes a new object, it takes a picture of that moment, adds a timestamp to it, and temporarily stores it.

[1956] The voice and image data recorded on the device are periodically sent to a server. The server stores the received data in a database and checks whether the information is new or not from the past database. The server also uses an emotion engine (e.g., IBM Watson Tone Analyzer or Affectiva) to identify the user's emotional state from the voice and image data. This emotion data is also stored along with the data.

[1957] When a new utterance or item is recognized, the server generates a message containing emotion data. For example, it generates a message in the format "[October 1, 2023, 10:30] My child said 'Mom' (joy)" and automatically posts it using a social networking API (e.g., Twitter API or Facebook API).

[1958] Furthermore, the server uses generative AI (e.g., OpenAI GPT-4) to create a diary-style document based on data from a certain period of time (e.g., one week). The voice data is transcribed, and explanatory text is added to the image data. The diary thus created can be shared with the user and their family via a dedicated application or social media. A specific prompt could be, "Please transcribe one week's worth of the user's voice data and compile it into a diary-style document along with emotional data."

[1959] This system allows users to record their growth and emotional state in real time, automatically generating and sharing emotionally rich growth records.

[1960] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1961] Step 1: Audio monitoring and recognition

[1962] The device constantly monitors the sounds around the user. A speech recognition algorithm (e.g., Google Speech-to-Text) is installed on the device, and it analyzes the user's speech in real time. It takes in the surrounding audio data as input and searches for specific utterances (e.g., "Mom"). If recognized, it extracts the utterance and outputs it as audio data.

[1963] Step 2: Recording audio data

[1964] The device records the voice data recognized in step 1 and assigns a timestamp. This voice data is temporarily stored in the device's local storage. Specifically, a timestamp such as "October 1, 2023, 10:30" is assigned to the voice data. The input is the recognized voice data, and the output is voice data with the timestamp.

[1965] Step 3: Send to the server

[1966] The device periodically transmits time-stamped voice data to the server. The transmitted voice data is stored in a database on the server. Specifically, the device uploads data to the server via the Internet. The input is the time-stamped voice data, and the output is the voice data stored in the database on the server.

[1967] Step 4: Emotion Engine Analysis

[1968] The server analyzes the stored voice data using an emotion engine (e.g., IBM Watson Tone Analyzer). It identifies the user's emotional state (e.g., joy, sadness, anger, etc.) from the voice data and generates emotion data. The input is the voice data stored on the server, and the output is emotion data. Specifically, the voice data is passed through an algorithm to analyze the emotional state.

[1969] Step 5: Share on social media

[1970] The server generates a message for social media based on the data recognized as new speech. A message containing emotion data is created in the following format, for example: "[October 1, 2023, 10:30] My child said 'Mom' (joy)." This message is automatically posted using a social media API (e.g., Twitter API). The input is voice data with emotion data, and the output is a post on social media. The specific operation is to post the message via the API.

[1971] Step 6: Image monitoring and recognition

[1972] The device (home camera) monitors the user's movements. It analyzes the camera image in real time and uses an image recognition algorithm (e.g., OpenCV) to detect what is being held in the user's hand. The input is live video data, and the output is recognized image data. As a concrete example, it detects the moment when the user picks up a new toy.

[1973] Step 7: Recording image data

[1974] The device assigns a timestamp to the recognized image data and records it. This data is temporarily stored in local storage. The recognized image data is used as input, and image data with a timestamp is generated as output. Specifically, the device assigns a timestamp such as "October 1, 2023, 14:30" to the image.

[1975] Step 8: Send to the server

[1976] The terminal sends the image data with the timestamp to the server. The sent data is stored in a database on the server. The input is the image data with the timestamp, and the output is the image data stored on the server. In concrete terms, the data is uploaded to the server via the Internet.

[1977] Step 9: Analysis by Emotion Engine (Image)

[1978] The server analyzes the stored image data using an emotion engine (e.g., Affectiva). This identifies the user's emotional state from their facial expressions and movements. The input is the image data stored on the server, and the output is emotion data. Specifically, the image data is passed through an algorithm to analyze the emotional state.

[1979] Step 10: Share on social media (images)

[1980] The server generates a message containing emotional data based on the data recognized as a new item. For example, it creates a message in the format "[October 1, 2023, 14:30] My child brought home a new toy. (Surprised)" and automatically posts it using a Facebook API or similar. The input is image data with emotional data, and the output is a post on social media. The specific operation is to post the message via the API.

[1981] Step 11: Data collection

[1982] The server extracts audio and image data for a certain period (e.g., one week) from a database. The input is a request specifying a specific period, and the output is the data collected during that period. Specifically, the data is extracted through a database query.

[1983] Step 12: Diary-style document generation

[1984] The server uses a generative AI (e.g., OpenAI GPT-4) to create a diary-style document based on the extracted data. As a specific example, the prompt is "Please transcribe one week's worth of user voice data and compile it into a diary-style document along with emotional data." The input is the collected voice data and image data, and the output is a diary-style document.

[1985] Step 13: Share your diary

[1986] The generated diary can be displayed and shared with the user or family members through a dedicated application or social networking site. The input is a diary-format document, and the output is a display on the application or a post on the social networking site. The specific operation involves posting the generated document to the application or social networking site.

[1987] (Application example 2)

[1988] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1989] To strengthen home security, it is important to be able to detect suspicious behavior and abnormal emotional states in real time and respond quickly to them. However, current home security systems do not adequately identify abnormal behavior and emotional states, which can lead to delayed responses in emergencies. In addition, there is a lack of means to quickly communicate detected information to family members or administrators, which delays information sharing and makes it difficult to take appropriate action.

[1990] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for recognizing the speech of a specific person, recording the voice data with a timestamp, recognizing an object held by the specific person, recording image data with a timestamp, saving the recorded voice data and image data in a database and automatically sharing them via SNS, and means for monitoring abnormal behavior and emotional states in specific locations in real time and issuing an alarm. This makes it possible to quickly detect abnormal behavior and emotional states within the home and share information in real time.

[1991] "Means for recognizing the speech of a specific person, adding a timestamp and recording the voice data" is a function that uses voice recognition technology to detect the speech of a selected person and saves the time at which the speech was made, along with a record of the time at which it was made.

[1992] "Means for recognizing an object held by a specific person, assigning a timestamp and recording image data" is a function that uses image analysis technology to detect an object held by a selected subject and saves a record of the time the object was held.

[1993] "Means for storing recorded audio and image data in a database and automatically sharing it through social networking services" refers to a function for building a database for systematically storing collected audio and image data and automatically sharing that data through social networking services.

[1994] "Means for monitoring abnormal behavior and emotional states in a specific location in real time and issuing an alarm" refers to a function that constantly monitors abnormal behavior and emotional states occurring within a designated space and issues an alarm based on the results.

[1995] "A means of using generative AI to create diary-style documents based on saved voice and image data" is a function that uses a machine learning model to generate natural-looking sentences based on collected voice and image data, and compiles them in diary format.

[1996] "Means for displaying and sharing the created diary through social networking services and dedicated applications" refers to a function for displaying and sharing the created diary using social networking services and specific mobile applications.

[1997] This invention is a system that automatically records what specific people say and what they hold in their hands, and shares that data in real time to enhance home security. The system mainly consists of a terminal, a server, and a generative AI model.

[1998] Terminal part

[1999] The device can be a smartphone or a home camera, and has the following main functions:

[2000] 1. Audio Recording:

[2001] It has a voice recognition function that constantly monitors the user's speech.

[2002] When a specific utterance is detected, the audio for that part is recorded and a timestamp is added.

[2003] The recorded data is temporarily stored and later sent to the server.

[2004] 2. Taking an image:

[2005] Cameras with image recognition capabilities for monitoring activity within the home.

[2006] When it detects that a specific object is being held in your hand, it takes an image of that moment and adds a timestamp.

[2007] The captured image data is also temporarily stored and later sent to the server.

[2008] Server part

[2009] The server processes the received audio and image data and generates the diary using generation AI.

[2010] 1. Data storage and collation:

[2011] The server stores the audio data and image data transmitted from the terminal in a database.

[2012] It is compared with past data to check whether the utterance or the item is new.

[2013] 2. Emotion engine analysis:

[2014] The audio and image data is analyzed to determine the user's emotional state.

[2015] This emotion data is also stored in the database.

[2016] 3. Share on social media:

[2017] When new data is recognized, a message containing the emotion data is generated for the SNS.

[2018] For example, it could automatically send a message like, "[October 1, 2023, 10:30] The sound of 'joy' was detected in your home."

[2019] Generative AI Models

[2020] The generative AI model automatically generates diaries from data stored on the server.

[2021] 1. Data Collection:

[2022] The server extracts audio data and image data for a certain period of time (for example, one week).

[2023] 2. Diary generation:

[2024] Generative AI is used to create diary-style documents based on collected data.

[2025] The audio data is transcribed and the image data is accompanied by explanatory text.

[2026] Emotional data is also reflected, providing a more detailed growth record.

[2027] 3. Share your diary:

[2028] The created diary can be displayed and shared via social media or dedicated applications.

[2029] For example, if suspicious activity is detected in the middle of the night in a home, the system will immediately record and analyze the change, and then send a message to social media saying, "October 1, 2023, 2:30 AM: A 'frightening' sound has been detected in the home," alerting the entire family.

[2030] Example prompts to input to a generative AI model:

[2031] "Please generate a sample program for an application that uses the emotion recognition system to monitor and share abnormal behavior and emotional states within the home in real time."

[2032] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[2033] Step 1:

[2034] The terminal recognizes the user's speech and records the voice data.

[2035] Input: User utterance

[2036] How it works: The device's voice recognition function constantly monitors the user's speech, and when it detects a specific utterance, it records that part of the audio and adds a timestamp.

[2037] Output: Audio data with timestamps

[2038] Step 2:

[2039] The device temporarily stores the recorded audio data and sends it to the server.

[2040] Input: Audio data with timestamps

[2041] Operation: The audio data temporarily stored on the device is sent to the server.

[2042] Output: Audio data sent to the server

[2043] Step 3:

[2044] The server receives the audio data and stores it in a database.

[2045] Input: Audio data sent to the server

[2046] How it works: The server stores the received voice data in a database and checks it against past databases to see if it is a new utterance.

[2047] Output: Audio data stored in a database

[2048] Step 4:

[2049] The server analyzes the voice data using an emotion engine to identify the user's emotional state.

[2050] Input: Audio data stored in the database

[2051] How it works: The server analyzes the voice data with an emotion engine to determine the user's emotional state.

[2052] Output: Emotion data

[2053] Step 5:

[2054] The server generates a message for SNS based on the voice data and emotion data and automatically sends it.

[2055] Input: Voice data, emotion data

[2056] Behavior: If the server recognizes a new utterance, it generates a message for SNS containing emotion data. Example: "[October 1, 2023, 10:30] The voice of 'joy' was detected in the home."

[2057] Output: Message sent to SNS

[2058] Step 6:

[2059] The device recognizes what the user is holding in their hand and takes a picture.

[2060] Input: User action

[2061] How it works: When the device's camera recognizes an object being held by the user, it takes a picture of that moment and adds a timestamp.

[2062] Output: Image data with timestamp

[2063] Step 7:

[2064] The device temporarily stores the captured image data and sends it to the server.

[2065] Input: Image data with timestamp

[2066] Operation: Image data temporarily stored on the device is sent to the server.

[2067] Output: Image data sent to the server

[2068] Step 8:

[2069] The server receives the image data and stores it in a database.

[2070] Input: Image data sent to the server

[2071] How it works: The server stores the image data it receives in a database and checks it against previous databases to see if it is new.

[2072] Output: Image data stored in a database

[2073] Step 9:

[2074] The server analyzes the image data using an emotion engine to identify the user's emotional state.

[2075] Input: Image data stored in a database

[2076] How it works: The server analyzes the image data with an emotion engine to determine the user's emotional state.

[2077] Output: Emotion data

[2078] Step 10:

[2079] The server generates a message for SNS based on the image data and emotional data and automatically sends it.

[2080] Input: image data, emotion data

[2081] Behavior: If the server recognizes a new possession, it generates a message for social media containing emotion data. Example: "[October 1, 2023, 14:30] 'Surprising' images detected in the home."

[2082] Output: Message sent to SNS

[2083] Step 11:

[2084] The server analyzes a certain period of audio and image data using a generative AI model and creates a diary-style document.

[2085] Input: Audio data and image data for a certain period of time

[2086] How it works: The server uses a generative AI model to create a diary-style document based on the extracted data. Audio data is transcribed, image data is annotated with explanatory text, and emotion data is also reflected in the diary.

[2087] Output: Diary-style document

[2088] Step 12:

[2089] The server displays and shares the created diary document through social networking sites or dedicated applications.

[2090] Input: diary-style document

[2091] How it works: The server displays and shares the created diary with the user and their family via social media or a dedicated application.

[2092] Output: View and share diary entries

[2093] Example prompts to input to a generative AI model:

[2094] "Please generate a sample program for an application that uses the emotion recognition system to monitor and share abnormal behavior and emotional states within the home in real time."

[2095] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[2096] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[2097] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[2098] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[2099] FIG. 9 illustrates an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and behaviors arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[2100] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[2101] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[2102] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.j...

Claims

1. A means for recognizing speech of a specific person and recording the speech data with a timestamp; A means for recognizing an object held by a specific person and recording image data with a timestamp; A means to store the recorded audio and image data in a database and automatically share it via social media, A system including:

2. The system of claim 1, further comprising means for generating a diary-style document based on the stored voice data and image data using a generative AI.

3. The system according to claim 1 , further comprising means for displaying and sharing the created diary through a social networking site and a dedicated application.

Images