system

Abstract

A system is provided.SOLUTION: A system, comprising: means for recording audio of a particular person; means for timestamping the audio data; means for storing the audio data in cloud storage; and means for automatically posting the timestamped audio data to an SNS platform.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] The present invention aims to solve the problem of how difficult it is to easily record a child's growing moments and share them with family and relatives. In particular, recording a child's speech or objects they pick up outside is fleeting and difficult to do manually. Therefore, there is a need for a system that can automatically and efficiently record these important moments and easily share them. [Means for solving the problem]

[0005] The present invention solves the above-mentioned problems by providing the following means.

[0006] 1. Provided is a system including a means for recording the voice of a specific person, a means for assigning a timestamp to the voice data, a means for storing the voice data in cloud storage, and a means for automatically posting the time-stamped voice data to a social networking platform.

[0007] 2. Provide a system including means for detecting an object held by a specific person, means for taking an image of the detected object, means for adding a timestamp to the image data, means for storing the image data in cloud storage, and means for automatically posting the time-stamped image data to a social networking platform.

[0008] 3. Provide a system further comprising means for detecting new utterances from the recorded audio.

[0009] This allows users to automatically record their child's growth moments and easily share them with family and relatives.

[0010] "Specific person" refers to an individual who is the subject of recording and recognition in this system, and is usually a person set by the user.

[0011] "Audio recording means" refers to a device or software for capturing and storing audio data in digital form.

[0012] The term "means for adding a time stamp" refers to a device or software that has the function of adding information about the date and time when the recorded data was generated.

[0013] "Audio Data" means a digital representation of recorded audio.

[0014] "Cloud storage" refers to remote servers and services for storing and managing data over the internet.

[0015] An "SNS platform" refers to a web service that allows users to interact with each other and exchange information over the Internet, and examples include Facebook and LINE.

[0016] "Means for detecting objects" refers to devices or software for identifying specific objects in video data and confirming their presence.

[0017] "Image data" refers to visual information of an object that is represented in digital form.

[0018] "Means for detecting new utterances from recorded audio" refers to a device or software that identifies and extracts newly uttered words from continuously recorded audio data. [Brief explanation of the drawings]

[0019] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0020] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0021] First, the terms used in the following description will be explained.

[0022] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0023] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0024] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0025] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0026] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0027] [First embodiment]

[0028] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0029] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0030] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0031] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0032] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0033] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0034] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0035] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0036] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0037] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0038] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0039] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0040] The present invention provides a system for automatically recording audio and images of a specific person, adding a timestamp, and automatically sharing the recorded audio and images through a social networking platform. The following describes in detail an embodiment of the present invention.

[0041] 1. Voice Recording Function

[0042] 1.1 Device (smartphone) operation

[0043] 1. The user launches the app

[0044] The user launches a voice recording application installed on the smartphone and enables the voice recording function.

[0045] 2. Device Settings

[0046] The device will run in the background and be configured to capture audio constantly using the microphone.

[0047] 3. Capture audio data

[0048] The device uses AI models to analyze voice data in real time and detect new utterances.

[0049] 4. Adding a timestamp to audio data

[0050] Any new audio clips detected will automatically be given a timestamp based on the current date and time.

[0051] 5. Data transmission

[0052] The device sends the recorded audio data and a timestamp to the server.

[0053] 1.2 Server Operation

[0054] 1. Receiving Data

[0055] The server receives the voice data sent from the device and stores it in cloud storage.

[0056] 2. Adding Metadata

[0057] The server adds metadata to the stored audio data, such as timestamps and a transcription of the audio content.

[0058] 3. Posting to social media

[0059] The server automatically creates and sends a post containing the audio clip and metadata to the configured social media platform (e.g., LINE).

[0060] 2. Image Recording Function

[0061] 2.1 Device (home camera) operation

[0062] 1. The user installs the camera

[0063] A user installs a home camera and configures it to recognize specific people.

[0064] 2. Real-time recognition

[0065] The device analyzes the video stream and uses AI models to identify specific people (children) in real time.

[0066] 3. Object Detection

[0067] When the camera recognizes that a specific person is holding a new object, it automatically takes a picture of the object.

[0068] 4. Add a timestamp to the image

[0069] Each captured image is given a timestamp based on the current date and time.

[0070] 5. Data transmission

[0071] The terminal transmits the image data and a timestamp to the server.

[0072] 2.2 Server Operation

[0073] 1. Receiving Data

[0074] The server receives the image data sent from the terminal and stores it in cloud storage.

[0075] 2. Adding Metadata

[0076] The server adds metadata such as timestamps and object recognition results to the stored image data.

[0077] 3. Posting to social media

[0078] The server automatically creates and sends a post containing the image and metadata to the configured social media platform (e.g., Instagram).

[0079] Specific examples

[0080] Examples of voice recording functions

[0081] 1. The user launches the app

[0082] The user launches a voice recording app on their smartphone and enables the recording function.

[0083] 2. Capture audio data

[0084] The moment your child says "mom" for the first time, the device detects the new utterance.

[0085] 3. Adding a timestamp

[0086] The audio clip will be timestamped with "2023-10-01 10:00:00".

[0087] 4. Data transmission and storage

[0088] The device sends the voice data to the server, which stores it in cloud storage.

[0089] 5. Social Media Posting

[0090] The server automatically posts the audio clip to LINE with the message "2023-10-01 10:00:00 First comment: Mom."

[0091] Example of image recording function

[0092] 1. The user installs the camera

[0093] The user installs the home camera and configures the recording settings.

[0094] 2. Object detection and photography

[0095] The camera detects when a child has a new toy (a red car) and automatically captures the moment.

[0096] 3. Adding a timestamp

[0097] The image will be given a timestamp of "2023-10-01 15:00:00".

[0098] 4. Data transmission and storage

[0099] The device sends the image data to the server, which stores it in cloud storage.

[0100] 5. Social Media Posting

[0101] The server automatically posts the image to Instagram with the message "2023-10-01 15:00:00 New Toy: Red Car."

[0102] As described above, the present invention allows users to effortlessly record moments of their children's growth and easily share them with family and relatives.

[0103] The processing flow will be explained below.

[0104] Voice recording function processing steps

[0105] Step 1:

[0106] The user launches a voice recording application installed on the smartphone and enables the voice recording function.

[0107] Step 2:

[0108] Set your device to run in the background and constantly capture audio using the microphone.

[0109] Step 3:

[0110] The device analyzes the captured voice data in real time to detect new utterances, using an AI model to analyze the voice data and identify new utterances.

[0111] Step 4:

[0112] When a new utterance is detected, the device generates an audio clip and gives it a timestamp based on the current date and time.

[0113] Step 5:

[0114] The device transmits the audio clip and timestamp to the server, using a common network communication protocol.

[0115] Step 6:

[0116] The server receives the voice data sent from the terminal and checks the integrity of the data.

[0117] Step 7:

[0118] The server stores the received audio data in cloud storage, along with metadata such as timestamps.

[0119] Step 8:

[0120] The server creates a post on a social networking platform (e.g., LINE) based on the stored audio data. This post includes the audio clip, a timestamp, and a brief text description of the audio.

[0121] Step 9:

[0122] The server sends the created post to the social media platform, where it is automatically posted to the user's configured account.

[0123] Image recording function processing steps

[0124] Step 1:

[0125] A user installs a home camera and configures it to recognize specific people.

[0126] Step 2:

[0127] The device (home camera) analyzes the video stream and is configured to identify specific people (children) in real time using an AI model.

[0128] Step 3:

[0129] The device detects the object being held by a particular person using an object recognition algorithm.

[0130] Step 4:

[0131] When a new object is detected, the device automatically captures the moment and generates an image.

[0132] Step 5:

[0133] The device will add a timestamp based on the current date and time to the generated image.

[0134] Step 6:

[0135] The device transmits the image data and a timestamp to the server, using a common network communication protocol.

[0136] Step 7:

[0137] The server receives the image data sent from the terminal and checks the consistency of the data.

[0138] Step 8:

[0139] The server stores the received image data in cloud storage, along with metadata such as timestamps and object recognition results.

[0140] Step 9:

[0141] The server creates a post on a social media platform (e.g., Instagram) based on the stored image data, which includes the image, a timestamp, and a brief description of the object.

[0142] Step 10:

[0143] The server sends the created post to the social media platform, where it is automatically posted to the user's configured account.

[0144] Through the above processing steps, audio and image recordings are automatically performed, and the data can be shared quickly and easily.

[0145] Example 1

[0146] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0147] Conventional audio and image recording systems are inadequate in efficiently recording important moments of specific people and automatically sharing them via social media due to the time and effort required. Furthermore, technology to improve the accuracy of audio and image analysis using AI models is underdeveloped. Therefore, there is a need for a method that allows users to record and share information quickly, accurately, and without the hassle of manual effort.

[0148] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0149] In this invention, the server includes means for recording the voice of a specific person, means for timestamp-adding the voice data, means for storing the voice data in cloud storage, means for automatically posting the time-stamped voice data to a social networking platform, means for utilizing a generative AI model for analyzing the voice, and means for performing named entity recognition, thereby enabling users to effortlessly and accurately record and quickly share important moments of a specific person.

[0150] An "audio recording device" is a device whose purpose is to capture the voice of a specific person and store it as digital data.

[0151] "Means for adding time stamps" refers to a means for adding information about the date and time of recording or shooting to recorded audio data or photographed image data.

[0152] "Means for saving audio data in cloud storage" refers to means for storing captured audio data in a data storage service on the Internet.

[0153] "Means for automatically posting audio data with a timestamp to a social networking platform" means a means for automatically publishing and sharing audio data with a timestamp on a pre-designated social networking service.

[0154] "Means using generative AI models to analyze speech" refers to means that utilize generated artificial intelligence models to analyze recorded speech data and identify important utterances and speech features.

[0155] "Means for performing named entity recognition" refers to means for extracting and identifying specific names or proper nouns from analyzed speech data.

[0156] The "means for detecting an object held by a specific person" is a means for automatically recognizing an object held by a specific person using a camera.

[0157] The "means for taking an image of a detected object" refers to a means for capturing an image of a recognized object and storing it as digital data.

[0158] "Means for saving image data in cloud storage" refers to means for storing captured image data in a data storage service on the Internet.

[0159] "Means for automatically posting image data with a timestamp to a social networking platform" refers to a means for automatically publishing and sharing image data with a timestamp attached to a pre-designated social networking service.

[0160] "Means utilizing a generative AI model for image recognition" means means utilizing a generative artificial intelligence model to analyze captured image data and identify objects within the image.

[0161] "Means for identifying object characteristics" refers to means for extracting and identifying characteristics such as size, shape, and color of a specific object from captured image data.

[0162] The "means for detecting new utterances" is a means for automatically identifying and extracting new utterances that are different from conventional ones from recorded audio data.

[0163] The present invention is a system that automatically records the voice and images of a specific person, adds a timestamp, and automatically shares them through a social networking platform. The detailed steps for implementing this system are described below.

[0164] Voice recording function embodiment

[0165] Device (smartphone) operation

[0166] 1. The user launches the app

[0167] The user launches the audio recording app installed on their smartphone and taps the "Start Recording" button to enable the recording function, which puts the device into audio recording mode.

[0168] 2. Configure your device

[0169] The device gets the necessary permissions from the user to operate in the background, including permission to use the microphone and to operate in the background, so the device is always ready to capture audio.

[0170] 3. Capture audio data

[0171] The device uses a microphone to capture voice data in real time, and uses the Google Speech-to-Text API to analyze the captured voice data and detect new utterances with an AI model.

[0172] 4. Adding a timestamp to audio data

[0173] When a new utterance is detected, the audio clip is timestamped with the current date and time information. For example, if the word "Mom" was uttered on 2023-10-01 10:00:00, the audio clip will be timestamped with this time.

[0174] 5. Send the data

[0175] The audio data and timestamp are sent to the cloud server using the HTTPS protocol.

[0176] Server Operation

[0177] 1. Receive data

[0178] The server receives the audio data and timestamps sent from the device, and stores this data in a database such as AWS RDS.

[0179] 2. Add metadata

[0180] The server converts the voice data into text and obtains the text data. It also adds timestamps and named entity recognition metadata. For example, the converted text is saved as "2023-10-01 10:00:00 Mom."

[0181] 3. Post to social media

[0182] The server uses the LINE Messaging API to automatically create and send a post containing the audio clip and metadata to the social media platform.

[0183] Image recording function embodiment

[0184] Device (home camera) operation

[0185] 1. The user installs the camera

[0186] The user simply installs the device in a suitable location and configures the camera via a dedicated smartphone app. Once the settings, including network connection, are complete, the camera goes into standby mode.

[0187] 2. Real-time recognition

[0188] The camera captures the video stream in real time and uses the Google Vision API to identify specific people, which is then used to determine whether a specific person is present in the frame.

[0189] 3. Detecting objects

[0190] The camera detects the object a specific person is holding using the YOLO model, etc. For example, if a child is holding a new toy (a red car), the camera will capture that moment.

[0191] 4. Add a timestamp to your images

[0192] A timestamp based on the current date and time is added to the captured image. For example, a timestamp such as "2023-10-01 15:00:00" is added to the image.

[0193] 5. Send the data

[0194] The captured image data and timestamp are sent to a cloud server using the HTTPS protocol.

[0195] Server Operation

[0196] 1. Receive data

[0197] The server receives the image data and timestamp sent from the device, and stores the data in cloud storage such as Google Cloud Storage.

[0198] 2. Add metadata

[0199] The server performs object recognition processing on the image data and adds the recognition results and a timestamp as metadata. For example, it adds metadata such as "2023-10-01 15:00:00 New toy: Red car."

[0200] 3. Post to social media

[0201] The server automatically creates and sends a post containing image data and metadata to the specified social media platform using the Instagram Graph API.

[0202] Specific examples

[0203] Examples of voice recording functions

[0204] 1. The user launches the app

[0205] The user launches a voice recording app on their smartphone and enables the recording function.

[0206] 2. Capture audio data

[0207] The moment your child says "mom" for the first time, the device detects the new utterance.

[0208] 3. Add a timestamp

[0209] The audio clip will be timestamped with "2023-10-01 10:00:00".

[0210] 4. Send and store data

[0211] The device sends the voice data to the server, which stores it in cloud storage.

[0212] 5. Post to social media

[0213] The server automatically posts the audio clip to LINE with the message "2023-10-01 10:00:00 First comment: Mom."

[0214] Example of image recording function

[0215] 1. The user installs the camera

[0216] The user installs a home camera and configures the recording settings.

[0217] 2. Detect and photograph objects

[0218] The camera detects when a child has a new toy (a red car) and automatically captures the moment.

[0219] 3. Add a timestamp

[0220] The image will be given a timestamp of "2023-10-01 15:00:00".

[0221] 4. Send and store data

[0222] The device sends the image data to the server, which stores it in cloud storage.

[0223] 5. Post to social media

[0224] The server automatically posts the image to Instagram with the message "2023-10-01 15:00:00 New Toy: Red Car."

[0225] This system allows users to effortlessly record important moments of specific people with high accuracy and quickly share them via social media.

[0226] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0227] Voice recording function embodiment

[0228] Device (smartphone) operation

[0229] Step 1:

[0230] The user launches the audio recording app installed on their smartphone. The user taps the "Start Recording" button to enable recording.

[0231] Input: User taps, launched applications

[0232] Output: Recording start trigger

[0233] Specific behavior:

[0234] Tap the application icon.

[0235] The application will launch and a Start Recording button will appear.

[0236] The user taps the start recording button.

[0237] Step 2:

[0238] The device will ask the user for the necessary permissions to operate in the background, such as permission to use the microphone and permission to operate in the background.

[0239] Input: User permission operation

[0240] Output: Allow background operation and microphone usage

[0241] Specific behavior:

[0242] The device will display a popup asking permission to run in the background and use the microphone.

[0243] The user selects "Allow."

[0244] Step 3:

[0245] The device captures audio in real time through a microphone and analyzes it using a generative AI model (Google Speech-to-Text API) to detect new utterances.

[0246] Input: Real-time captured audio data

[0247] Output: Audio clip where new utterances were detected

[0248] Specific behavior:

[0249] The microphone continues to capture audio.

[0250] The audio data is sent to the Google Speech-to-Text API.

[0251] The API analyzes the audio data and detects new utterances.

[0252] Step 4:

[0253] When a new utterance is detected, the audio clip is given a timestamp based on the current date and time.

[0254] Input: Audio data in which new utterances are detected

[0255] Output: Time-stamped audio data

[0256] Specific behavior:

[0257] Get the current date and time information.

[0258] Add timestamps to audio clips.

[0259] Step 5:

[0260] The device sends the time-stamped audio data to the server using the HTTPS protocol.

[0261] Input: Timestamped audio data

[0262] Output: The audio data is sent to the server.

[0263] Specific behavior:

[0264] The data is sent to the server via HTTPS.

[0265] The server receives the data.

[0266] Server Operation

[0267] Step 6:

[0268] The server receives the audio data and timestamp sent from the device and stores it in cloud storage (AWS RDS).

[0269] Input: Audio data with timestamps

[0270] Output: Audio data stored in cloud storage

[0271] Specific behavior:

[0272] The server receives the data.

[0273] Store the audio data in AWS RDS.

[0274] Step 7:

[0275] The server performs text conversion processing on the voice data and adds a timestamp and named entity recognition data to the converted text data.

[0276] Input: Saved audio data

[0277] Output: Audio data with text conversion and metadata

[0278] Specific behavior:

[0279] The server sends the voice data to the generative AI model (voice analysis model).

[0280] Time stamps and named entities are added to the obtained text data.

[0281] Step 8:

[0282] The server automatically creates and sends a post containing the audio clip and the attached metadata to the SNS platform (LINE) using the SNS API.

[0283] Input: Audio data with metadata

[0284] Output: Audio data posted to SNS

[0285] Specific behavior:

[0286] Use the LINE Messaging API to create a message.

[0287] Post the generated message to LINE.

[0288] Image recording function embodiment

[0289] Device (home camera) operation

[0290] Step 1:

[0291] The user installs the home camera in an appropriate location and configures the camera via their smartphone.

[0292] Input: Camera settings operated by the user

[0293] Output: Camera starts operating

[0294] Specific behavior:

[0295] Connect the camera to a power source and position it appropriately.

[0296] Use the smartphone app to configure the camera settings (such as network connection settings).

[0297] Step 2:

[0298] The camera captures the video stream in real time and uses a generative AI model (Google Vision API) to identify specific people.

[0299] Input: Real-time captured video data

[0300] Output: Video data in which a specific person is identified

[0301] Specific behavior:

[0302] The camera continues to capture footage.

[0303] The video data is sent to the Google Vision API.

[0304] The API identifies a specific person.

[0305] Step 3:

[0306] Use an AI model (YOLO) to detect objects held by a specific person. Check if a person is holding a new object.

[0307] Input: Video data of identified person

[0308] Output: Data showing the object being held

[0309] Specific behavior:

[0310] Send the video data to the YOLO model.

[0311] The model detects objects and recognizes what a person is holding.

[0312] Step 4:

[0313] When a new object is detected, the camera automatically captures the moment and adds a timestamp to the image data.

[0314] Input: Video data of a person holding a new object

[0315] Output: Image data with timestamps

[0316] Specific behavior:

[0317] Get the current date and time information.

[0318] Add a timestamp to the image.

[0319] Step 5:

[0320] The device sends the image data and a timestamp to the server using the HTTPS protocol.

[0321] Input: Image data with timestamps

[0322] Output: Image data sent to the server

[0323] Specific behavior:

[0324] Send the data to the server over HTTPS.

[0325] The server receives the data.

[0326] Server Operation

[0327] Step 6:

[0328] The server receives the image data and timestamp sent from the device and stores them in Google Cloud Storage.

[0329] Input: Image data with timestamp

[0330] Output: Image data stored in cloud storage

[0331] Specific behavior:

[0332] The server receives the data.

[0333] Store image data in Google Cloud Storage.

[0334] Step 7:

[0335] The server performs object recognition processing on the image data and adds the recognition results and a timestamp as metadata.

[0336] Input: Saved image data

[0337] Output: Image data with metadata

[0338] Specific behavior:

[0339] The server sends the image data to the Google Vision API.

[0340] The object recognition result and a timestamp are added to the image data.

[0341] Step 8:

[0342] The server automatically creates and sends a post containing image data and attached metadata to a social networking platform (Instagram) using the social networking API.

[0343] Input: Image data with metadata

[0344] Output: Image data posted to SNS

[0345] Specific behavior:

[0346] Use the Instagram Graph API to generate messages.

[0347] Post the generated message to Instagram.

[0348] The above is the flow of the specific processing steps of this system. In this way, users can easily and accurately record important moments of specific people and quickly share them via social media.

[0349] (Application example 1)

[0350] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0351] Improving safety and security within the home has become an important issue in modern society. Responding quickly to suspicious sounds or movements, especially when the homeowner is away, is difficult. Conventional home security systems require constant monitoring, which is time-consuming and may not provide timely notifications. While recorded data and images are stored in cloud storage, there is no mechanism in place for automatically posting this data as an alert via social networking services (SNS). Therefore, to improve home security, a system that automatically detects suspicious sounds and movements and notifies users in real time is needed.

[0352] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0353] In this invention, the server includes means for recording the voice of a specific person, means for assigning a timestamp to the voice data, means for storing the voice data in cloud storage, means for automatically posting the time-stamped voice data to a social networking service platform, means for detecting suspicious voices, and means for notifying the social networking service of the detected suspicious voices as an alert, thereby enabling the real-time detection of suspicious voices occurring while the homeowner is away and a prompt response to the detected suspicious voices.

[0354] Furthermore, by including a means for detecting an object held by a specific person, a means for capturing an image of the detected object, a means for adding a timestamp to the image data, a means for storing the image data in cloud storage, a means for automatically posting the time-stamped image data to a social networking service platform, a means for detecting suspicious movements, and a means for notifying the social networking service of the detected suspicious movements as an alert, it is possible to instantly record suspicious movements within the home and automatically send an alert, thereby significantly improving home security.

[0355] A "means for recording the voice of a specific person" is equipment or software for recording the voice spoken by a specific individual as digital data.

[0356] The "means for adding a timestamp to audio data" refers to a device or software for adding a timestamp indicating the date and time of recording to recorded audio data.

[0357] "Means for storing voice data in cloud storage" refers to equipment or software for storing recorded voice data in a remote data storage service on the Internet.

[0358] "Means for automatically posting time-stamped audio data to a social networking service platform" refers to equipment or software for automatically posting time-stamped audio data to a network service such as a social media site.

[0359] "Means for detecting suspicious sounds" refers to equipment or software for detecting abnormal sounds or sounds that indicate a specific danger from recorded audio data.

[0360] "Means for notifying social networking services of detected suspicious audio as an alert" refers to equipment or software for sending information about detected suspicious audio as an alert to services on the network, such as social media sites.

[0361] A "means for detecting an object held by a particular person" is a device or software for recognizing and identifying an item held by a particular individual.

[0362] The "means for taking an image of a detected object" refers to a device or software for taking an image of a detected object using a device such as a camera.

[0363] The "means for adding a timestamp to image data" refers to a device or software for adding a timestamp indicating the date and time of photography to photographed image data.

[0364] "Means for storing image data in cloud storage" refers to equipment or software for storing recorded image data in a remote data storage service on the Internet.

[0365] "Means for automatically posting image data with a timestamp to a social networking service platform" refers to equipment or software for automatically posting image data with a timestamp to a service on a network, such as a social media site.

[0366] "Means for detecting suspicious activity" refers to equipment or software for detecting abnormal or fraudulent behavior from captured video data.

[0367] "Means for notifying social networking services of detected suspicious activity as an alert" refers to equipment or software for sending information about detected suspicious activity as an alert to services on the network, such as social media sites.

[0368] The present invention relates to a home security system that automatically detects suspicious sounds and movements and notifies a social networking service in real time. Specific embodiments of the system will be described in detail below.

[0369] 1. Voice Recording Function

[0370] Device (smartphone) operation

[0371] 1. The user launches the app

[0372] The user launches a voice recording application installed on the smartphone and enables the voice recording function.

[0373] 2. Device Settings

[0374] The device will run in the background and be configured to capture audio constantly using the microphone.

[0375] 3. Capture audio data

[0376] The device uses an AI voice recognition engine (e.g., Google Cloud Speech-to-Text) to analyze voice data in real time and detect suspicious voices.

[0377] 4. Adding a timestamp to audio data

[0378] Any suspicious audio detected is automatically given a timestamp based on the current date and time.

[0379] 5. Data transmission

[0380] The device sends the recorded audio data and a timestamp to the server.

[0381] Server Operation

[0382] 1. Receiving Data

[0383] The server receives the voice data sent from the device and stores it in cloud storage (e.g., AWS S3).

[0384] 2. Adding Metadata

[0385] The server adds metadata to the stored audio data, such as timestamps and a transcription of the audio content.

[0386] 3. Posting to social media

[0387] The server uses social media APIs (e.g., Twitter API) to automatically create and send a warning post containing the suspicious audio clip and metadata.

[0388] Example (voice recording function)

[0389] While the homeowner is away, the moment the sound of a window breaking is detected, the timestamp "2023-10-01 22:30:00" is added. The app automatically posts a message to Twitter saying "2023-10-01 22:30:00 The sound of a window breaking was detected!"

[0390] 2. Image Recording Function

[0391] Device (home camera) operation

[0392] 1. The user installs the camera

[0393] Users install home cameras and configure them to monitor specific areas.

[0394] 2. Real-time recognition

[0395] The device analyzes the live video stream and uses an AI image recognition engine (e.g., AWS Rekognition) to identify suspicious activity in real time.

[0396] 3. Motion Detection

[0397] When suspicious activity is detected, the camera automatically records the moment.

[0398] 4. Add a timestamp to the image

[0399] Recorded images are given a timestamp based on the current date and time.

[0400] 5. Data transmission

[0401] The terminal transmits the image data and a timestamp to the server.

[0402] Server Operation

[0403] 1. Receiving Data

[0404] The server receives the image data sent from the terminal and stores it in cloud storage.

[0405] 2. Adding Metadata

[0406] The server adds metadata to the stored image data, such as timestamps and suspicious movement detection results.

[0407] 3. Posting to social media

[0408] The server uses the SNS API to automatically create and send alerts containing images and metadata of suspicious activity.

[0409] Example (image recording function)

[0410] If an image of a stranger walking around the house while the homeowner is out is detected, it will be given the timestamp "2023-10-02 02:15:00." The app will then automatically post the message "2023-10-02 02:15:00 A stranger is walking around the house!" to Twitter.

[0411] Example prompts for generative AI models

[0412] Audio capture: The sound of a window breaking has been detected. Recorded at 2023-10-01 22:30:00.

[0413] Image Capture: An unknown individual walking through the home. Recorded at 2023-10-02 02:15:00.

[0414] Twitter Security Warning

[0415] According to the above-described embodiment, the present invention can significantly improve security within the home.

[0416] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0417] Step 1:

[0418] The user launches the app. The user launches the audio recording app installed on their smartphone and enables the audio recording function. This puts the microphone in a state where it is always ready to capture audio. The input is the user's operation, and the output is the state where the recording function is enabled.

[0419] Step 2:

[0420] Configure the device so that it runs in the background and constantly captures audio using the microphone. Input is the system setting, and output is the microphone constantly capturing audio.

[0421] Step 3:

[0422] Voice data is captured. The device uses an AI voice recognition engine (e.g., Google Cloud Speech-to-Text) to analyze the voice data in real time and detect suspicious voices. The input is real-time voice data from the microphone, and the output is analyzed voice text data. Specifically, the voice data is sent to the AI ​​voice recognition engine and converted into text.

[0423] Step 4:

[0424] A timestamp is added to the audio data. A timestamp based on the current date and time is automatically added to detected suspicious audio. The input is analyzed audio text data, and the output is audio text data with a timestamp. Specifically, the current date and time is obtained and added to the audio data.

[0425] Step 5:

[0426] Data is transmitted. The terminal transmits the recorded voice data and a timestamp to the server. The input is voice text data with a timestamp, and the output is data transmission to the server. Specifically, the data is transmitted to the server using an HTTP request.

[0427] Step 6:

[0428] The server receives the data. The server receives the voice data sent from the device and stores it in cloud storage (e.g., AWS S3). The input is time-stamped voice text data, and the output is data storage in cloud storage. Specifically, the received data is stored using the cloud storage API.

[0429] Step 7:

[0430] Add metadata. The server adds metadata such as timestamps and text conversion of the audio content to the stored audio data. The input is audio data stored in cloud storage, and the output is audio data with metadata. Specific operations include checking the timestamp and adding the text conversion results.

[0431] Step 8:

[0432] Post to social media. The server automatically creates and sends a warning post containing the suspicious audio clip and metadata using a social media API (e.g., Twitter API). The input is audio data with metadata, and the output is a warning post to the social media platform. The specific operation is to send a message and audio clip using the social media API.

[0433] Step 9:

[0434] The user installs a camera. The user installs a home camera and configures it to monitor a specific area. The input is the user's operation, and the output is the camera installation completion and monitoring configuration.

[0435] Step 10:

[0436] Real-time recognition. The device analyzes the live video stream and uses an AI image recognition engine (e.g., AWS Rekognition) to identify suspicious activity in real time. The input is the live video stream from the camera, and the output is analyzed motion data. Specifically, the video stream is sent to the AI ​​image recognition engine for analysis.

[0437] Step 11:

[0438] It detects motion. When a suspicious movement is detected, the camera automatically records the moment. The input is the analyzed movement data, and the output is the recorded image data. Specifically, a frame is captured at the moment an abnormality is detected.

[0439] Step 12:

[0440] A timestamp is added to image data. A timestamp based on the current date and time is added to recorded images. The input is recorded image data, and the output is image data with a timestamp. Specifically, the current date and time is obtained and added to the image data.

[0441] Step 13:

[0442] Data is sent. The terminal sends image data and a timestamp to the server. The input is image data with a timestamp, and the output is data sent to the server. Specifically, the data is sent to the server using an HTTP request.

[0443] Step 14:

[0444] The server receives the data. The server receives the image data sent from the device and stores it in cloud storage. The input is image data with a timestamp, and the output is data storage in cloud storage. Specifically, the received data is stored using the cloud storage API.

[0445] Step 15:

[0446] Add metadata. The server adds metadata such as a timestamp and suspicious movement recognition results to the stored image data. The input is image data stored in cloud storage, and the output is image data with metadata. Specific operations include checking the timestamp and adding movement recognition results.

[0447] Step 16:

[0448] Post to SNS. The server uses SNS API to automatically create and send an alert containing images and metadata of suspicious activity. The input is image data with metadata, and the output is a warning post to the SNS platform. The specific operation is to send a message and image data using the SNS API.

[0449] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0450] The present invention aims to combine an emotion engine with a system that automatically records the voices and images of specific people, timestamps the data, and automatically shares it through a social networking platform, thereby recognizing the user's emotions based on the recorded data and providing information corresponding to those emotions.

[0451] 1. Voice Recording Function

[0452] 1.1 Device (smartphone) operation

[0453] 1. The user launches the app

[0454] The user launches a voice recording application installed on the smartphone and enables the voice recording function.

[0455] 2. Device Settings

[0456] The device will run in the background and be configured to capture audio constantly using the microphone.

[0457] 3. Capture audio data

[0458] It uses AI models to analyze voice data in real time and detect new utterances.

[0459] 4. Emotion Recognition by Emotion Engine

[0460] The device sends the captured voice data to an emotion engine to recognize the user's emotions, which analyzes the tone, strength, and speed of the voice to identify emotions.

[0461] 5. Adding a timestamp to audio data

[0462] Any new audio clips detected will be given a timestamp based on the current date and time.

[0463] 6. Data transmission

[0464] The device sends the recorded voice data, a timestamp, and information on the recognized emotion to the server.

[0465] 1.2 Server Operation

[0466] 1. Receiving Data

[0467] The server receives the voice data and emotion information sent from the terminal and checks the consistency of the data.

[0468] 2. Adding Metadata

[0469] The server adds metadata to the received audio data, such as timestamps, text conversion of the audio content, and emotional information.

[0470] 3. Posting to social media

[0471] The server creates and automatically sends a post to the configured social networking platform (e.g., LINE), which includes the audio clip, a timestamp, the text of the audio, and emotional information.

[0472] 4. Emotion-based replies and comments generation

[0473] The server generates appropriate replies and comments based on the recognized emotions and automatically posts them to social media platforms.

[0474] 2. Image Recording Function

[0475] 2.1 Device (home camera) operation

[0476] 1. The user installs the camera

[0477] A user installs a home camera and configures it to recognize specific people.

[0478] 2. Real-time recognition

[0479] The device analyzes the video stream and uses AI models to identify specific people (children) in real time.

[0480] 3. Object Detection

[0481] When the camera recognizes that a specific person is holding a new object, it automatically takes a picture of the object.

[0482] 4. Add a timestamp to the image

[0483] Each captured image is given a timestamp based on the current date and time.

[0484] 5. Data transmission

[0485] The terminal transmits the image data and a timestamp to the server.

[0486] 2.2 Server Operation

[0487] 1. Receiving Data

[0488] The server receives the image data sent from the terminal and checks the integrity of the data.

[0489] 2. Adding Metadata

[0490] The server adds metadata such as a timestamp and object recognition results to the received image data.

[0491] 3. Posting to social media

[0492] The server creates and automatically sends a post to the configured social media platform (e.g., Instagram), which includes the image, a timestamp, and a description of the object.

[0493] 4. Emotion-based replies and comments generation

[0494] The server generates appropriate replies and comments based on the recognized emotions and automatically posts them to social media platforms.

[0495] Specific examples

[0496] Examples of voice recording functions

[0497] 1. The user launches the app

[0498] The user launches a voice recording app on their smartphone and enables the recording function.

[0499] 2. Capture audio data

[0500] The moment a child says "Daddy" for the first time, the device detects the new utterance and uses its emotion engine to recognize that the utterance contains the emotion of "joy."

[0501] 3. Adding a timestamp

[0502] The audio clip will be timestamped with "2023-10-02 13:20:00".

[0503] 4. Data transmission and storage

[0504] The terminal transmits the voice data, timestamp, and emotion information to the server, which stores them in cloud storage.

[0505] 5. Social media posts and emotional comment generation

[0506] The server automatically posts an audio clip to LINE with the message "2023-10-02 13:20:00 He said 'Daddy' (joy)," and also posts the comment "What a wonderful first word!"

[0507] Example of image recording function

[0508] 1. The user installs the camera

[0509] The user installs the home camera and configures the recording settings.

[0510] 2. Object detection and photography

[0511] The camera detects when a child is holding a new toy (a blue ball) and automatically captures the moment.

[0512] 3. Emotion Recognition by Emotion Engine

[0513] The emotion engine analyzes the audio and video data obtained from the captured images and recognizes that the child is "excited."

[0514] 4. Adding a timestamp

[0515] The image will be given a timestamp of "2023-10-02 14:10:00".

[0516] 5. Data transmission and storage

[0517] The terminal transmits the image data, timestamp, and emotion information to the server, which stores the data in cloud storage.

[0518] 6. Social media posts and emotional comment generation

[0519] The server automatically posts the image to Instagram with the message "2023-10-02 14:10:00 New toy: Blue ball (excited)" and also posts the comment "You're so excited about your new toy!"

[0520] With the above-described configuration, the present invention not only allows users to effortlessly record moments of their child's growth and easily share them with family and relatives, but also allows users to provide information according to their emotions.

[0521] The processing flow will be explained below.

[0522] Voice recording function processing steps

[0523] Step 1:

[0524] The user launches a voice recording application installed on the smartphone and enables the voice recording function.

[0525] Step 2:

[0526] Set your device to run in the background and constantly capture audio using the microphone.

[0527] Step 3:

[0528] The device analyzes the captured voice data in real time to detect new utterances, using an AI model to analyze the voice data and identify new utterances.

[0529] Step 4:

[0530] When a new utterance is detected, the device generates an audio clip and gives it a timestamp based on the current date and time.

[0531] Step 5:

[0532] The device sends the audio clip to an emotion engine that analyzes the user's emotions, which determine emotions based on factors such as tone, volume, and vocal inflection.

[0533] Step 6:

[0534] The emotion engine recognizes emotions (e.g., joy, sadness, excitement, etc.) contained in an audio clip and attaches the emotion data to the audio clip.

[0535] Step 7:

[0536] The device transmits the audio clip, timestamp, and emotion data to the server using a common network communication protocol.

[0537] Step 8:

[0538] The server receives the voice data sent from the terminal and checks the integrity of the data.

[0539] Step 9:

[0540] The server stores the received audio data in cloud storage, along with metadata such as timestamps and emotional data.

[0541] Step 10:

[0542] The server creates a post on a social networking platform (e.g., LINE) based on the stored audio data, which includes the audio clip, a timestamp, a transcription of the audio content, and emotion data.

[0543] Step 11:

[0544] The server sends the created post to the social media platform, where it is automatically posted to the user's configured account.

[0545] Step 12:

[0546] The server generates an appropriate reply or comment based on the recognized emotion and automatically posts it to the social media platform. For example, if "joy" is recognized, a comment such as "That's a great first word!" will be generated.

[0547] Image recording function processing steps

[0548] Step 1:

[0549] A user installs a home camera and configures it to recognize specific people.

[0550] Step 2:

[0551] The device (home camera) analyzes the video stream and is configured to identify specific people (children) in real time using an AI model.

[0552] Step 3:

[0553] The device detects the object being held by a particular person using an object recognition algorithm.

[0554] Step 4:

[0555] When a new object is detected, the device automatically captures the moment and generates an image.

[0556] Step 5:

[0557] The device will add a timestamp based on the current date and time to the generated image.

[0558] Step 6:

[0559] The device sends image data to the emotion engine, which analyzes the user's emotions. The emotion engine identifies emotions based on the user's facial expressions, movements, voice, etc.

[0560] Step 7:

[0561] The emotion engine recognizes emotions (e.g., excitement, surprise, joy, etc.) contained in the image data and attaches the emotion data to the image data.

[0562] Step 8:

[0563] The device transmits the image data, timestamp, and emotion data to the server, using a common network communication protocol.

[0564] Step 9:

[0565] The server receives the image data sent from the terminal and checks the consistency of the data.

[0566] Step 10:

[0567] The server stores the received image data in cloud storage, along with metadata such as timestamps and emotion data.

[0568] Step 11:

[0569] The server creates a post for a social media platform (e.g., Instagram) based on the stored image data, which includes the image, a timestamp, and emotion data.

[0570] Step 12:

[0571] The server sends the created post to the social media platform, where it is automatically posted to the user's configured account.

[0572] Step 13:

[0573] The server generates an appropriate reply or comment based on the recognized emotion and automatically posts it to the social media platform. For example, if "excitement" is recognized, a comment such as "You're crazy about your new toy!" will be generated.

[0574] Through the above processing steps, audio and image recording combined with an emotion engine is automatically performed, and the data can be shared quickly and easily, while also providing information according to emotions.

[0575] Example 2

[0576] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0577] Current systems that record audio and images and share them on social media platforms have difficulty in properly recognizing users' emotions and providing information. Furthermore, they lack the functionality to detect new utterances from recorded audio or automatically recognize objects in images, requiring users to put in a lot of effort.

[0578] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes means for recording the voice of a specific person, means for analyzing the voice data and recognizing the content of the speech, means for recognizing emotions from the voice data, means for assigning a timestamp to the voice data, means for storing the voice data and the recognized emotional information in cloud storage, means for automatically posting the time-stamped voice data including the voice data and the recognized emotional information to an SNS platform, means for detecting an object held by the specific person, means for capturing an image of the detected object, means for analyzing image data and extracting a description of the object, means for recognizing emotions from the image data, means for assigning a timestamp to the image data, means for storing the image data and the recognized emotional information in cloud storage, means for automatically posting the time-stamped image data including the image data, the recognized object description, and the emotional information to an SNS platform, and means for detecting new comments from the recorded voice. This makes it possible to efficiently record a user's voice and image data along with the recognized emotions and automatically share them on an SNS platform.

[0579] "Specific person" refers to an individual who has been designated by the system as a target for recording audio and image data.

[0580] "Audio recording means" refers to devices or software that use a microphone to capture environmental sounds and store them as digital data.

[0581] "Means for analyzing and recognizing what is being said" refers to natural language processing technology that converts voice data into text and understands its content.

[0582] "Means for recognizing emotions" refers to algorithms or engines for identifying emotions from audio or image data.

[0583] "Means for adding a time stamp" refers to a function that adds the date and time of collection to recorded data in a digital format.

[0584] "Cloud storage means" refers to a service that allows data to be uploaded and stored on a remote server via the Internet.

[0585] "Means for automatically posting to social media platforms" refers to a program that connects to a designated social media service and automatically shares data.

[0586] "Means for detecting objects" refers to technology for analyzing image data to recognize and identify specific objects.

[0587] "Means for taking images of detected objects" refers to devices or software that use a camera to capture images containing recognized objects and store the digital data.

[0588] "Means for analyzing image data and extracting object descriptions" refers to a technique for analyzing image data and generating text information about objects contained therein.

[0589] "Means for detecting new utterances from audio recordings" refers to algorithms for identifying and detecting newly uttered words or phrases from continuously recorded audio data.

[0590] The present invention aims to combine an emotion engine with a system that automatically records the voices and images of specific people, timestamps the data, and automatically shares it through a social networking platform, thereby recognizing the user's emotions based on the recorded data and providing information corresponding to those emotions.

[0591] Voice recording function embodiment

[0592] Device (smartphone) operation

[0593] 1. The user launches the app

[0594] A user launches a voice recording app installed on their smartphone and enables the recording function. The app uses the Google Speech-to-Text API to convert the voice data into text in real time.

[0595] 2. Device Settings

[0596] The device runs in the background and is set to capture audio using the microphone at all times. Users can enable the recording feature in a specific settings menu and allow it to run in the background.

[0597] 3. Capture audio data

[0598] When voice is input, the app on the device calls the Google Speech-to-Text API to convert it into text and checks the content.

[0599] 4. Emotion Recognition by Emotion Engine

[0600] The converted text and audio data is sent to IBM Watson Tone Analyzer, where emotions are analyzed based on tone, strength, speed, etc.

[0601] 5. Adding a timestamp to audio data

[0602] The detected audio clips are given a timestamp based on the current date and time obtained from the device's internal clock.

[0603] 6. Data transmission

[0604] The device uses HTTPS communication to send the recorded audio data, timestamps, and emotion information to the server.

[0605] Server Operation

[0606] 1. Receiving Data

[0607] The server receives the voice data and emotion information sent from the device via the HTTPS protocol, and then checks the data checksum to verify its integrity.

[0608] 2. Adding Metadata

[0609] Metadata such as text conversion results and emotional information is added to the received voice data and stored in a database.

[0610] 3. Posting to social media

[0611] The server automatically creates and sends a post to the configured social networking platform (e.g., LINE) containing the audio clip, a timestamp, the text of the audio content, and emotional information.

[0612] 4. Emotion-based replies and comments generation

[0613] Based on the recognized sentiment, appropriate replies and comments are generated and automatically posted to social media platforms.

[0614] Image recording function embodiment

[0615] Device (home camera) operation

[0616] 1. The user installs the camera

[0617] A user installs a home camera in a specific location and configures the camera to identify specific people (e.g., children) within a certain range.

[0618] 2. Real-time recognition

[0619] The camera uses OpenCV to analyze the video stream in real time and recognize specific people.

[0620] 3. Object Detection

[0621] Detects when a specific person is holding a new object, and captures the moment with the camera.

[0622] 4. Add a timestamp to the image

[0623] Captured images are automatically time-stamped based on the current date and time.

[0624] 5. Data transmission

[0625] The captured image data and timestamp are sent to the server via HTTPS.

[0626] Server Operation

[0627] 1. Receiving Data

[0628] The server receives the image data sent from the terminal and checks the integrity of the data.

[0629] 2. Adding Metadata

[0630] A timestamp and object recognition results are added as metadata to the received image data.

[0631] 3. Posting to social media

[0632] The server automatically creates and sends a post to the configured social media platform (e.g., Instagram) containing the image, a timestamp, and a description of the object.

[0633] 4. Emotion-based replies and comments generation

[0634] Based on the recognized sentiment, appropriate replies and comments are generated and automatically posted to social media platforms.

[0635] Specific examples

[0636] Examples of voice recording functions

[0637] 1. The user launches the app

[0638] The user launches a voice recording app on their smartphone and enables the recording function.

[0639] 2. Capture audio data

[0640] The moment a child says "Daddy" for the first time, the device detects the new utterance and uses its emotion engine to recognize that the utterance contains the emotion of "joy."

[0641] 3. Adding a timestamp

[0642] The audio clip will be timestamped with "2023-10-02 13:20:00".

[0643] 4. Data transmission and storage

[0644] The terminal transmits the voice data, timestamp, and emotion information to the server, which stores them in cloud storage.

[0645] 5. Social media posts and emotional comment generation

[0646] The server automatically posts an audio clip to LINE with the message "2023-10-02 13:20:00 He said 'Daddy' (joy)" and also posts the comment "What a wonderful first word!"

[0647] Example of image recording function

[0648] 1. The user installs the camera

[0649] The user sets up the home camera and configures the recording settings.

[0650] 2. Object detection and photography

[0651] The camera detects when a child is holding a new toy (a blue ball) and automatically captures the moment.

[0652] 3. Emotion Recognition by Emotion Engine

[0653] The emotion engine analyzes the audio and video data obtained from the captured images and recognizes that the child is "excited."

[0654] 4. Adding a timestamp

[0655] The image will be given a timestamp of "2023-10-02 14:10:00".

[0656] 5. Data transmission and storage

[0657] The terminal transmits the image data, timestamp, and emotion information to the server, which stores the data in cloud storage.

[0658] 6. Social media posts and emotional comment generation

[0659] The server automatically posts the image to Instagram with the message "2023-10-02 14:10:00 New toy: Blue ball (excited)" and also posts the comment "You're so excited about your new toy!"

[0660] According to the above-described embodiment, the present invention enables users to record and share audio and image data without any special effort, and provides information according to their emotions.

[0661] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0662] Voice recording function processing steps

[0663] Device (smartphone) operation

[0664] Step 1:

[0665] The user launches the app

[0666] Input: The user operates the smartphone and taps the icon of the voice recording app.

[0667] Specific operation: The user finds the audio recording app in the list of apps on their smartphone and taps the icon.

[0668] Output: The audio recording app will launch and the initial screen for the recording function will be displayed.

[0669] Step 2:

[0670] Device settings

[0671] Input: The user taps the "Start Recording" button in the app's settings menu to enable the recording feature.

[0672] What it does: The user taps the "Start Recording" button on the app's main screen and approves background recording if prompted.

[0673] Output: The device will have access to the microphone and will be configured to capture audio continuously in the background.

[0674] Step 3:

[0675] Capture audio data

[0676] Input: Audio signals from the environment.

[0677] How it works: The device uses the Google Speech-to-Text API to convert voice data to text in real time, and this conversion occurs whenever new speech is detected.

[0678] Output: Text conversion results and analysis results of the audio data.

[0679] Step 4:

[0680] Emotion recognition by emotion engine

[0681] Input: Converted text data and audio data.

[0682] Specific operation: Text and audio data are sent to IBM Watson Tone Analyzer, which analyzes emotions based on tone, strength, speed, etc.

[0683] Output: Parsed emotion information (e.g., happiness, sadness, excitement, etc.).

[0684] Step 5:

[0685] Adding a timestamp to audio data

[0686] Input: Audio clip and analyzed emotion information.

[0687] Specific operation: Obtains date and time information from the device's internal clock and adds a timestamp as metadata to the audio clip.

[0688] Output: Timestamp audio data and emotion information.

[0689] Step 6:

[0690] Sending data

[0691] Input: Timestamp audio data, emotion information.

[0692] What it does: Sends recorded data to a server using the HTTPS protocol.

[0693] Output: Voice data and emotion information sent to the server.

[0694] Server Operation

[0695] Step 1:

[0696] Receiving data

[0697] Input: Voice data and emotion information sent from the device.

[0698] Specific operation: The server receives the data via the HTTPS protocol and checks the checksum to verify the integrity of the data.

[0699] Output: Received voice data, emotion information, and consistency check results.

[0700] Step 2:

[0701] Adding metadata

[0702] Input: Received voice data and emotion information.

[0703] Specific operation: Adds timestamps, text conversion results of audio content, emotional information, etc. to received data.

[0704] Output: Audio data with metadata.

[0705] Step 3:

[0706] Posting to social media

[0707] Input: Audio data with metadata.

[0708] Specific operation: Calls the API of a social media platform (e.g., LINE) and posts the audio clip, timestamp, audio text, and emotional information.

[0709] Output: Audio data posted on SNS.

[0710] Step 4:

[0711] Generate replies and comments based on emotions

[0712] Input: Parsed emotion information.

[0713] Specific behavior: Generate replies and comments based on emotional information and automatically post them to social media platforms.

[0714] Output: Replies and comments posted to social media.

[0715] Image recording function processing steps

[0716] Device (home camera) operation

[0717] Step 1:

[0718] The user installs the camera

[0719] Input: The user installs the home camera in a specific location and configures it.

[0720] Specific operation: The user fixes the camera in a designated position and sets up facial recognition for a specific person (e.g., a child) through the app.

[0721] Output: Camera installation completed and target person set.

[0722] Step 2:

[0723] Real-time recognition

[0724] Input: A video stream acquired in real time.

[0725] How it works: The camera uses OpenCV to analyze the video stream and identify specific people in real time.

[0726] Output: Identification result of a specific person.

[0727] Step 3:

[0728] Object detection

[0729] Input: Image data about an object held by a particular person.

[0730] What it does: It uses an object detection algorithm to recognize when a new object is being held by a person and captures that moment.

[0731] Output: Image data containing the new object.

[0732] Step 4:

[0733] Add a timestamp to the image

[0734] Input: Captured image data.

[0735] Specific operation: Obtains date and time information from the device's internal clock and adds a timestamp as metadata to the image data.

[0736] Output: Timestamped image data.

[0737] Step 5:

[0738] Sending data

[0739] Input: Timestamped image data.

[0740] Specific operation: Image data is sent to the server using the HTTPS protocol.

[0741] Output: Image data sent to the server.

[0742] Server Operation

[0743] Step 1:

[0744] Receiving data

[0745] Input: Image data sent from the device.

[0746] Specific operation: The server receives the data via the HTTPS protocol and checks the checksum to verify the integrity of the data.

[0747] Output: Received image data, consistency check results.

[0748] Step 2:

[0749] Adding metadata

[0750] Input: Received image data.

[0751] Specific operation: Adds metadata such as timestamps and object recognition results to the received data.

[0752] Output: Image data with metadata.

[0753] Step 3:

[0754] Posting to social media

[0755] Input: Image data with metadata.

[0756] Specific operation: Calls the API of a social media platform (e.g., Instagram) and posts an image, timestamp, and object description.

[0757] Output: Image data posted on SNS.

[0758] Step 4:

[0759] Generate replies and comments based on emotions

[0760] Input: Parsed emotion information.

[0761] Specific behavior: Generate replies and comments based on emotional information and automatically post them to social media platforms.

[0762] Output: Replies and comments posted to social media.

[0763] The above are the specific processing steps of the present invention, which allow users to record and share audio and image data without any special effort, and provide information according to their emotions.

[0764] (Application example 2)

[0765] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0766] Modern life demands that necessary information and services be provided based on a user's emotional state. However, existing technologies lack systems that can effectively analyze a user's emotions and provide appropriate responses based on those analysis results. In particular, it is difficult to automatically collect and analyze audio and image data to provide information tailored to the user's emotions. Furthermore, there is no system in place to reliably store this data in cloud storage and automatically post it to social media platforms. This makes it difficult to realize the personalized experiences users desire.

[0767] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0768] In this invention, the server includes means for recording the voice of a specific person, means for assigning a timestamp to the voice data, means for storing the voice data in cloud storage, means for automatically posting the time-stamped voice data to a social networking service platform, means for analyzing the voice data using an emotion analysis engine and recognizing the user's emotion, and means for providing the user with appropriate information based on the recognized emotion. This makes it possible to analyze the user's emotion in real time and provide information and services according to the emotion.

[0769] The term "specific person" refers to a specific individual who is a predetermined target.

[0770] "Audio recording means" means a means for storing the speech of a particular person in digital form using a device.

[0771] "Audio data" refers to data that has been converted into digital form and recorded.

[0772] The "means for adding a time stamp" is a means for adding the date and time when the data was recorded to the audio data or image data.

[0773] "Cloud storage" is a set of remote servers that store data over the internet.

[0774] A "social networking service platform" is an online service that enables users to share information and communicate with other users via the Internet.

[0775] "Automatic posting means" refers to a means for publishing specific data on a given platform without user intervention.

[0776] An "emotion analysis engine" is a set of algorithms and software that analyzes audio and image data and identifies the user's emotions from them.

[0777] The "means for providing appropriate information" is a means for providing information and services that are considered to be optimal for the user based on the recognized emotions.

[0778] This invention is a system that automatically records the voice and images of a specific person, adds a timestamp to the data, and automatically shares it through a social networking platform. Furthermore, by combining it with an emotion analysis engine, it can recognize the user's emotions from the recorded data and provide information according to those emotions.

[0779] A system for implementing the present invention uses the following configuration and associated hardware and software.

[0780] System configuration

[0781] 1. Device Operation

[0782] 1. Audio recording and emotion analysis

[0783] The device is equipped with a microphone for recording voice and an AI model (e.g., EmotionRecognizer) for analyzing the voice data. When a user launches the app and engages in normal conversation, the device captures the voice. The voice is analyzed in real time through an emotion analysis engine to recognize the user's emotions. This emotion analysis uses the tone, strength, and speed of the voice.

[0784] 2. Image recording and emotion analysis

[0785] If the device is equipped with a camera, it will detect an object held by a specific person and take a picture of it. The captured image data will also be passed through an emotion analysis engine to recognize the user's emotions. This analysis uses facial expressions and the situation contained in the image.

[0786] 3. Adding a timestamp and sending data

[0787] The recorded audio and video data is time-stamped based on the current date and time, stored in cloud storage, and then automatically sent to social media platforms.

[0788] 2. Server Operation

[0789] 1. Data Receipt and Storage

[0790] The server receives the voice and image data sent from the device, checks the integrity of the data, and also receives emotional information. It adds additional metadata (e.g., converted voice content and object recognition results) to the received data and stores it in cloud storage.

[0791] 2. Posting and providing information on social media

[0792] The server then posts to the configured social networking platform in the specified format, including audio clips, images, timestamps, analyzed text, and emotion information. Appropriate replies and comments are also automatically generated and posted based on the recognized emotion.

[0793] Hardware and software used

[0794] Hardware: Smartphone (microphone, speaker, camera), cloud server, storage

[0795] Software: EmotionRecognizer (AI model for emotion recognition), SNS API (for posting to SNS)

[0796] Specific examples

[0797] Example: A user opens a smartphone app and orders dinner while uttering emotional speech. For example, the user says, "I'm tired today." The app recognizes this speech as "sad" and suggests "comfort ice cream" as a meal. This information is then posted to social media with a timestamp, sharing the following: "2023-10-05 18:30:00 - Emotion: Sad, Suggested Meal: Comfort Ice Cream."

[0798] Example prompt sentence:

[0799] The app allows users to record their voices expressing their emotions, recognize their emotions based on the voice data, and recommend appropriate meals. After that, users can share this information on social media.

[0800] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0801] Step 1:

[0802] The device records the audio.

[0803] When a user launches the app and speaks, the device's microphone captures the audio. The input is the user's speech, and the output is digital audio data that can be used in subsequent processes.

[0804] Step 2:

[0805] The device tags the audio data with a timestamp.

[0806] This system adds a timestamp to recorded audio data by adding the current date and time information to the audio data. The input is digital audio data, and the output is audio data with a timestamp.

[0807] Step 3:

[0808] The device sends the voice data to the emotion analysis engine.

[0809] The device sends the recorded voice data to an emotion analysis engine, which analyzes the voice. The input is time-stamped voice data, and the output is analyzed emotion information. For example, the tone, strength, and speed of the voice are used to identify emotions.

[0810] Step 4:

[0811] The device saves the data to cloud storage.

[0812] The analyzed emotion information and the time-stamped voice data are sent to and stored in cloud storage. The input is the time-stamped voice data and the analyzed emotion information, and the output is the data stored in cloud storage.

[0813] Step 5:

[0814] The server receives the time-stamped audio data and emotion information.

[0815] The server receives the data sent to the cloud storage and checks the integrity of the data. The input is the data from the cloud storage, and the output is the data whose integrity has been checked.

[0816] Step 6:

[0817] The server adds metadata to the received data.

[0818] This converts audio data into text and adds additional metadata (e.g., a textual representation of the audio content). The input is time-stamped audio data with integrity check and analyzed emotion information, and the output is audio data with metadata.

[0819] Step 7:

[0820] The server automatically posts to the social networking service platform.

[0821] This system automatically posts time-stamped audio data, metadata, and emotional information to a social media platform. The input is audio data with metadata and analyzed emotional information, and the output is a post to a social media platform.

[0822] Step 8:

[0823] The server generates a comment based on the emotion and posts it to social media.

[0824] The system posts automatically generated comments based on the recognized emotions to a social media platform. The input is the analyzed emotion information, and the output is a social media post with the emotion-based comment.

[0825] Specifically, when a user says, "I'm tired today," the voice is recorded and a timestamp is added. The emotion analysis engine recognizes the voice as "sad" and saves it in cloud storage. The server then receives the data and posts the following to social media: "2023-10-05 18:30:00 - Emotion: Sad, Suggested Meal: Comforting Ice Cream." The server also simultaneously posts the comment, "Thank you for your hard work! Refresh yourself with some ice cream!"

[0826] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0827] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0828] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0829] [Second embodiment]

[0830] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0831] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0832] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0833] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0834] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0835] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0836] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0837] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0838] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0839] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0840] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0841] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0842] The present invention provides a system for automatically recording audio and images of a specific person, adding a timestamp, and automatically sharing the recorded audio and images through a social networking platform. The following describes in detail an embodiment of the present invention.

[0843] 1. Voice Recording Function

[0844] 1.1 Device (smartphone) operation

[0845] 1. The user launches the app

[0846] The user launches a voice recording application installed on the smartphone and enables the voice recording function.

[0847] 2. Device Settings

[0848] The device will run in the background and be configured to capture audio constantly using the microphone.

[0849] 3. Capture audio data

[0850] The device uses AI models to analyze voice data in real time and detect new utterances.

[0851] 4. Adding a timestamp to audio data

[0852] Any new audio clips detected will automatically be given a timestamp based on the current date and time.

[0853] 5. Data transmission

[0854] The device sends the recorded audio data and a timestamp to the server.

[0855] 1.2 Server Operation

[0856] 1. Receiving Data

[0857] The server receives the voice data sent from the device and stores it in cloud storage.

[0858] 2. Adding Metadata

[0859] The server adds metadata to the stored audio data, such as timestamps and a transcription of the audio content.

[0860] 3. Posting to social media

[0861] The server automatically creates and sends a post containing the audio clip and metadata to the configured social media platform (e.g., LINE).

[0862] 2. Image Recording Function

[0863] 2.1 Device (home camera) operation

[0864] 1. The user installs the camera

[0865] A user installs a home camera and configures it to recognize specific people.

[0866] 2. Real-time recognition

[0867] The device analyzes the video stream and uses AI models to identify specific people (children) in real time.

[0868] 3. Object Detection

[0869] When the camera recognizes that a specific person is holding a new object, it automatically takes a picture of the object.

[0870] 4. Add a timestamp to the image

[0871] Each captured image is given a timestamp based on the current date and time.

[0872] 5. Data transmission

[0873] The terminal transmits the image data and a timestamp to the server.

[0874] 2.2 Server Operation

[0875] 1. Receiving Data

[0876] The server receives the image data sent from the terminal and stores it in cloud storage.

[0877] 2. Adding Metadata

[0878] The server adds metadata such as timestamps and object recognition results to the stored image data.

[0879] 3. Posting to social media

[0880] The server automatically creates and sends a post containing the image and metadata to the configured social media platform (e.g., Instagram).

[0881] Specific examples

[0882] Examples of voice recording functions

[0883] 1. The user launches the app

[0884] The user launches a voice recording app on their smartphone and enables the recording function.

[0885] 2. Capture audio data

[0886] The moment your child says "mom" for the first time, the device detects the new utterance.

[0887] 3. Adding a timestamp

[0888] The audio clip will be timestamped with "2023-10-01 10:00:00".

[0889] 4. Data transmission and storage

[0890] The device sends the voice data to the server, which stores it in cloud storage.

[0891] 5. Social Media Posting

[0892] The server automatically posts the audio clip to LINE with the message "2023-10-01 10:00:00 First comment: Mom."

[0893] Example of image recording function

[0894] 1. The user installs the camera

[0895] The user installs the home camera and configures the recording settings.

[0896] 2. Object detection and photography

[0897] The camera detects when a child has a new toy (a red car) and automatically captures the moment.

[0898] 3. Adding a timestamp

[0899] The image will be given a timestamp of "2023-10-01 15:00:00".

[0900] 4. Data transmission and storage

[0901] The device sends the image data to the server, which stores it in cloud storage.

[0902] 5. Social Media Posting

[0903] The server automatically posts the image to Instagram with the message "2023-10-01 15:00:00 New Toy: Red Car."

[0904] As described above, the present invention allows users to effortlessly record moments of their children's growth and easily share them with family and relatives.

[0905] The processing flow will be explained below.

[0906] Voice recording function processing steps

[0907] Step 1:

[0908] The user launches a voice recording application installed on the smartphone and enables the voice recording function.

[0909] Step 2:

[0910] Set your device to run in the background and constantly capture audio using the microphone.

[0911] Step 3:

[0912] The device analyzes the captured voice data in real time to detect new utterances, using an AI model to analyze the voice data and identify new utterances.

[0913] Step 4:

[0914] When a new utterance is detected, the device generates an audio clip and gives it a timestamp based on the current date and time.

[0915] Step 5:

[0916] The device transmits the audio clip and timestamp to the server, using a common network communication protocol.

[0917] Step 6:

[0918] The server receives the voice data sent from the terminal and checks the integrity of the data.

[0919] Step 7:

[0920] The server stores the received audio data in cloud storage, along with metadata such as timestamps.

[0921] Step 8:

[0922] The server creates a post on a social networking platform (e.g., LINE) based on the stored audio data. This post includes the audio clip, a timestamp, and a brief text description of the audio.

[0923] Step 9:

[0924] The server sends the created post to the social media platform, where it is automatically posted to the user's configured account.

[0925] Image recording function processing steps

[0926] Step 1:

[0927] A user installs a home camera and configures it to recognize specific people.

[0928] Step 2:

[0929] The device (home camera) analyzes the video stream and is configured to identify specific people (children) in real time using an AI model.

[0930] Step 3:

[0931] The device detects the object being held by a particular person using an object recognition algorithm.

[0932] Step 4:

[0933] When a new object is detected, the device automatically captures the moment and generates an image.

[0934] Step 5:

[0935] The device will add a timestamp based on the current date and time to the generated image.

[0936] Step 6:

[0937] The device transmits the image data and a timestamp to the server, using a common network communication protocol.

[0938] Step 7:

[0939] The server receives the image data sent from the terminal and checks the consistency of the data.

[0940] Step 8:

[0941] The server stores the received image data in cloud storage, along with metadata such as timestamps and object recognition results.

[0942] Step 9:

[0943] The server creates a post on a social media platform (e.g., Instagram) based on the stored image data, which includes the image, a timestamp, and a brief description of the object.

[0944] Step 10:

[0945] The server sends the created post to the social media platform, where it is automatically posted to the user's configured account.

[0946] Through the above processing steps, audio and image recordings are automatically performed, and the data can be shared quickly and easily.

[0947] Example 1

[0948] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0949] Conventional audio and image recording systems are inadequate in efficiently recording important moments of specific people and automatically sharing them via social media due to the time and effort required. Furthermore, technology to improve the accuracy of audio and image analysis using AI models is underdeveloped. Therefore, there is a need for a method that allows users to record and share information quickly, accurately, and without the hassle of manual effort.

[0950] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0951] In this invention, the server includes means for recording the voice of a specific person, means for timestamp-adding the voice data, means for storing the voice data in cloud storage, means for automatically posting the time-stamped voice data to a social networking platform, means for utilizing a generative AI model for analyzing the voice, and means for performing named entity recognition, thereby enabling users to effortlessly and accurately record and quickly share important moments of a specific person.

[0952] An "audio recording device" is a device whose purpose is to capture the voice of a specific person and store it as digital data.

[0953] "Means for adding time stamps" refers to a means for adding information about the date and time of recording or shooting to recorded audio data or photographed image data.

[0954] "Means for saving audio data in cloud storage" refers to means for storing captured audio data in a data storage service on the Internet.

[0955] "Means for automatically posting audio data with a timestamp to a social networking platform" means a means for automatically publishing and sharing audio data with a timestamp on a pre-designated social networking service.

[0956] "Means using generative AI models to analyze speech" refers to means that utilize generated artificial intelligence models to analyze recorded speech data and identify important utterances and speech features.

[0957] "Means for performing named entity recognition" refers to means for extracting and identifying specific names or proper nouns from analyzed speech data.

[0958] The "means for detecting an object held by a specific person" is a means for automatically recognizing an object held by a specific person using a camera.

[0959] The "means for taking an image of a detected object" refers to a means for capturing an image of a recognized object and storing it as digital data.

[0960] "Means for saving image data in cloud storage" refers to means for storing captured image data in a data storage service on the Internet.

[0961] "Means for automatically posting image data with a timestamp to a social networking platform" refers to a means for automatically publishing and sharing image data with a timestamp attached to a pre-designated social networking service.

[0962] "Means utilizing a generative AI model for image recognition" means means utilizing a generative artificial intelligence model to analyze captured image data and identify objects within the image.

[0963] "Means for identifying object characteristics" refers to means for extracting and identifying characteristics such as size, shape, and color of a specific object from captured image data.

[0964] The "means for detecting new utterances" is a means for automatically identifying and extracting new utterances that are different from conventional ones from recorded audio data.

[0965] The present invention is a system that automatically records the voice and images of a specific person, adds a timestamp, and automatically shares them through a social networking platform. The detailed steps for implementing this system are described below.

[0966] Voice recording function embodiment

[0967] Device (smartphone) operation

[0968] 1. The user launches the app

[0969] The user launches the audio recording app installed on their smartphone and taps the "Start Recording" button to enable the recording function, which puts the device into audio recording mode.

[0970] 2. Configure your device

[0971] The device gets the necessary permissions from the user to operate in the background, including permission to use the microphone and to operate in the background, so the device is always ready to capture audio.

[0972] 3. Capture audio data

[0973] The device uses a microphone to capture voice data in real time, and uses the Google Speech-to-Text API to analyze the captured voice data and detect new utterances with an AI model.

[0974] 4. Adding a timestamp to audio data

[0975] When a new utterance is detected, the audio clip is timestamped with the current date and time information. For example, if the word "Mom" was uttered on 2023-10-01 10:00:00, the audio clip will be timestamped with this time.

[0976] 5. Send the data

[0977] The audio data and timestamp are sent to the cloud server using the HTTPS protocol.

[0978] Server Operation

[0979] 1. Receive data

[0980] The server receives the audio data and timestamps sent from the device, and stores this data in a database such as AWS RDS.

[0981] 2. Add metadata

[0982] The server converts the voice data into text and obtains the text data. It also adds timestamps and named entity recognition metadata. For example, the converted text is saved as "2023-10-01 10:00:00 Mom."

[0983] 3. Post to social media

[0984] The server uses the LINE Messaging API to automatically create and send a post containing the audio clip and metadata to the social media platform.

[0985] Image recording function embodiment

[0986] Device (home camera) operation

[0987] 1. The user installs the camera

[0988] The user simply installs the device in a suitable location and configures the camera via a dedicated smartphone app. Once the settings, including network connection, are complete, the camera goes into standby mode.

[0989] 2. Real-time recognition

[0990] The camera captures the video stream in real time and uses the Google Vision API to identify specific people, which is then used to determine whether a specific person is present in the frame.

[0991] 3. Detecting objects

[0992] The camera detects the object a specific person is holding using the YOLO model, etc. For example, if a child is holding a new toy (a red car), the camera will capture that moment.

[0993] 4. Add a timestamp to your images

[0994] A timestamp based on the current date and time is added to the captured image. For example, a timestamp such as "2023-10-01 15:00:00" is added to the image.

[0995] 5. Send the data

[0996] The captured image data and timestamp are sent to a cloud server using the HTTPS protocol.

[0997] Server Operation

[0998] 1. Receive data

[0999] The server receives the image data and timestamp sent from the device, and stores the data in cloud storage such as Google Cloud Storage.

[1000] 2. Add metadata

[1001] The server performs object recognition processing on the image data and adds the recognition results and a timestamp as metadata. For example, it adds metadata such as "2023-10-01 15:00:00 New toy: Red car."

[1002] 3. Post to social media

[1003] The server automatically creates and sends a post containing image data and metadata to the specified social media platform using the Instagram Graph API.

[1004] Specific examples

[1005] Examples of voice recording functions

[1006] 1. The user launches the app

[1007] The user launches a voice recording app on their smartphone and enables the recording function.

[1008] 2. Capture audio data

[1009] The moment your child says "mom" for the first time, the device detects the new utterance.

[1010] 3. Add a timestamp

[1011] The audio clip will be timestamped with "2023-10-01 10:00:00".

[1012] 4. Send and store data

[1013] The device sends the voice data to the server, which stores it in cloud storage.

[1014] 5. Post to social media

[1015] The server automatically posts the audio clip to LINE with the message "2023-10-01 10:00:00 First comment: Mom."

[1016] Example of image recording function

[1017] 1. The user installs the camera

[1018] The user installs a home camera and configures the recording settings.

[1019] 2. Detect and photograph objects

[1020] The camera detects when a child has a new toy (a red car) and automatically captures the moment.

[1021] 3. Add a timestamp

[1022] The image will be given a timestamp of "2023-10-01 15:00:00".

[1023] 4. Send and store data

[1024] The device sends the image data to the server, which stores it in cloud storage.

[1025] 5. Post to social media

[1026] The server automatically posts the image to Instagram with the message "2023-10-01 15:00:00 New Toy: Red Car."

[1027] This system allows users to effortlessly record important moments of specific people with high accuracy and quickly share them via social media.

[1028] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1029] Voice recording function embodiment

[1030] Device (smartphone) operation

[1031] Step 1:

[1032] The user launches the audio recording app installed on their smartphone. The user taps the "Start Recording" button to enable recording.

[1033] Input: User taps, launched applications

[1034] Output: Recording start trigger

[1035] Specific behavior:

[1036] Tap the application icon.

[1037] The application will launch and a Start Recording button will appear.

[1038] The user taps the start recording button.

[1039] Step 2:

[1040] The device will ask the user for the necessary permissions to operate in the background, such as permission to use the microphone and permission to operate in the background.

[1041] Input: User permission operation

[1042] Output: Allow background operation and microphone usage

[1043] Specific behavior:

[1044] The device will display a popup asking permission to run in the background and use the microphone.

[1045] The user selects "Allow."

[1046] Step 3:

[1047] The device captures audio in real time through a microphone and analyzes it using a generative AI model (Google Speech-to-Text API) to detect new utterances.

[1048] Input: Real-time captured audio data

[1049] Output: Audio clip where new utterances were detected

[1050] Specific behavior:

[1051] The microphone continues to capture audio.

[1052] The audio data is sent to the Google Speech-to-Text API.

[1053] The API analyzes the audio data and detects new utterances.

[1054] Step 4:

[1055] When a new utterance is detected, the audio clip is given a timestamp based on the current date and time.

[1056] Input: Audio data in which new utterances are detected

[1057] Output: Time-stamped audio data

[1058] Specific behavior:

[1059] Get the current date and time information.

[1060] Add timestamps to audio clips.

[1061] Step 5:

[1062] The device sends the time-stamped audio data to the server using the HTTPS protocol.

[1063] Input: Timestamped audio data

[1064] Output: The audio data is sent to the server.

[1065] Specific behavior:

[1066] The data is sent to the server via HTTPS.

[1067] The server receives the data.

[1068] Server Operation

[1069] Step 6:

[1070] The server receives the audio data and timestamp sent from the device and stores it in cloud storage (AWS RDS).

[1071] Input: Audio data with timestamps

[1072] Output: Audio data stored in cloud storage

[1073] Specific behavior:

[1074] The server receives the data.

[1075] Store the audio data in AWS RDS.

[1076] Step 7:

[1077] The server performs text conversion processing on the voice data and adds a timestamp and named entity recognition data to the converted text data.

[1078] Input: Saved audio data

[1079] Output: Audio data with text conversion and metadata

[1080] Specific behavior:

[1081] The server sends the voice data to the generative AI model (voice analysis model).

[1082] Time stamps and named entities are added to the obtained text data.

[1083] Step 8:

[1084] The server automatically creates and sends a post containing the audio clip and the attached metadata to the SNS platform (LINE) using the SNS API.

[1085] Input: Audio data with metadata

[1086] Output: Audio data posted to SNS

[1087] Specific behavior:

[1088] Use the LINE Messaging API to create a message.

[1089] Post the generated message to LINE.

[1090] Image recording function embodiment

[1091] Device (home camera) operation

[1092] Step 1:

[1093] The user installs the home camera in an appropriate location and configures the camera via their smartphone.

[1094] Input: Camera settings operated by the user

[1095] Output: Camera starts operating

[1096] Specific behavior:

[1097] Connect the camera to a power source and position it appropriately.

[1098] Use the smartphone app to configure the camera settings (such as network connection settings).

[1099] Step 2:

[1100] The camera captures the video stream in real time and uses a generative AI model (Google Vision API) to identify specific people.

[1101] Input: Real-time captured video data

[1102] Output: Video data in which a specific person is identified

[1103] Specific behavior:

[1104] The camera continues to capture footage.

[1105] The video data is sent to the Google Vision API.

[1106] The API identifies a specific person.

[1107] Step 3:

[1108] Use an AI model (YOLO) to detect objects held by a specific person. Check if a person is holding a new object.

[1109] Input: Video data of identified person

[1110] Output: Data showing the object being held

[1111] Specific behavior:

[1112] Send the video data to the YOLO model.

[1113] The model detects objects and recognizes what a person is holding.

[1114] Step 4:

[1115] When a new object is detected, the camera automatically captures the moment and adds a timestamp to the image data.

[1116] Input: Video data of a person holding a new object

[1117] Output: Image data with timestamps

[1118] Specific behavior:

[1119] Get the current date and time information.

[1120] Add a timestamp to the image.

[1121] Step 5:

[1122] The device sends the image data and a timestamp to the server using the HTTPS protocol.

[1123] Input: Image data with timestamps

[1124] Output: Image data sent to the server

[1125] Specific behavior:

[1126] Send the data to the server over HTTPS.

[1127] The server receives the data.

[1128] Server Operation

[1129] Step 6:

[1130] The server receives the image data and timestamp sent from the device and stores them in Google Cloud Storage.

[1131] Input: Image data with timestamp

[1132] Output: Image data stored in cloud storage

[1133] Specific behavior:

[1134] The server receives the data.

[1135] Store image data in Google Cloud Storage.

[1136] Step 7:

[1137] The server performs object recognition processing on the image data and adds the recognition results and a timestamp as metadata.

[1138] Input: Saved image data

[1139] Output: Image data with metadata

[1140] Specific behavior:

[1141] The server sends the image data to the Google Vision API.

[1142] The object recognition result and a timestamp are added to the image data.

[1143] Step 8:

[1144] The server automatically creates and sends a post containing image data and attached metadata to a social networking platform (Instagram) using the social networking API.

[1145] Input: Image data with metadata

[1146] Output: Image data posted to SNS

[1147] Specific behavior:

[1148] Use the Instagram Graph API to generate messages.

[1149] Post the generated message to Instagram.

[1150] The above is the flow of the specific processing steps of this system. In this way, users can easily and accurately record important moments of specific people and quickly share them via social media.

[1151] (Application example 1)

[1152] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1153] Improving safety and security within the home has become an important issue in modern society. Responding quickly to suspicious sounds or movements, especially when the homeowner is away, is difficult. Conventional home security systems require constant monitoring, which is time-consuming and may not provide timely notifications. While recorded data and images are stored in cloud storage, there is no mechanism in place for automatically posting this data as an alert via social networking services (SNS). Therefore, to improve home security, a system that automatically detects suspicious sounds and movements and notifies users in real time is needed.

[1154] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1155] In this invention, the server includes means for recording the voice of a specific person, means for assigning a timestamp to the voice data, means for storing the voice data in cloud storage, means for automatically posting the time-stamped voice data to a social networking service platform, means for detecting suspicious voices, and means for notifying the social networking service of the detected suspicious voices as an alert, thereby enabling the real-time detection of suspicious voices occurring while the homeowner is away and a prompt response to the detected suspicious voices.

[1156] Furthermore, by including a means for detecting an object held by a specific person, a means for capturing an image of the detected object, a means for adding a timestamp to the image data, a means for storing the image data in cloud storage, a means for automatically posting the time-stamped image data to a social networking service platform, a means for detecting suspicious movements, and a means for notifying the social networking service of the detected suspicious movements as an alert, it is possible to instantly record suspicious movements within the home and automatically send an alert, thereby significantly improving home security.

[1157] A "means for recording the voice of a specific person" is equipment or software for recording the voice spoken by a specific individual as digital data.

[1158] The "means for adding a timestamp to audio data" refers to a device or software for adding a timestamp indicating the date and time of recording to recorded audio data.

[1159] "Means for storing voice data in cloud storage" refers to equipment or software for storing recorded voice data in a remote data storage service on the Internet.

[1160] "Means for automatically posting time-stamped audio data to a social networking service platform" refers to equipment or software for automatically posting time-stamped audio data to a network service such as a social media site.

[1161] "Means for detecting suspicious sounds" refers to equipment or software for detecting abnormal sounds or sounds that indicate a specific danger from recorded audio data.

[1162] "Means for notifying social networking services of detected suspicious audio as an alert" refers to equipment or software for sending information about detected suspicious audio as an alert to services on the network, such as social media sites.

[1163] A "means for detecting an object held by a particular person" is a device or software for recognizing and identifying an item held by a particular individual.

[1164] The "means for taking an image of a detected object" refers to a device or software for taking an image of a detected object using a device such as a camera.

[1165] The "means for adding a timestamp to image data" refers to a device or software for adding a timestamp indicating the date and time of photography to photographed image data.

[1166] "Means for storing image data in cloud storage" refers to equipment or software for storing recorded image data in a remote data storage service on the Internet.

[1167] "Means for automatically posting image data with a timestamp to a social networking service platform" refers to equipment or software for automatically posting image data with a timestamp to a service on a network, such as a social media site.

[1168] "Means for detecting suspicious activity" refers to equipment or software for detecting abnormal or fraudulent behavior from captured video data.

[1169] "Means for notifying social networking services of detected suspicious activity as an alert" refers to equipment or software for sending information about detected suspicious activity as an alert to services on the network, such as social media sites.

[1170] The present invention relates to a home security system that automatically detects suspicious sounds and movements and notifies a social networking service in real time. Specific embodiments of the system will be described in detail below.

[1171] 1. Voice Recording Function

[1172] Device (smartphone) operation

[1173] 1. The user launches the app

[1174] The user launches a voice recording application installed on the smartphone and enables the voice recording function.

[1175] 2. Device Settings

[1176] The device will run in the background and be configured to capture audio constantly using the microphone.

[1177] 3. Capture audio data

[1178] The device uses an AI voice recognition engine (e.g., Google Cloud Speech-to-Text) to analyze voice data in real time and detect suspicious voices.

[1179] 4. Adding a timestamp to audio data

[1180] Any suspicious audio detected is automatically given a timestamp based on the current date and time.

[1181] 5. Data transmission

[1182] The device sends the recorded audio data and a timestamp to the server.

[1183] Server Operation

[1184] 1. Receiving Data

[1185] The server receives the voice data sent from the device and stores it in cloud storage (e.g., AWS S3).

[1186] 2. Adding Metadata

[1187] The server adds metadata to the stored audio data, such as timestamps and a transcription of the audio content.

[1188] 3. Posting to social media

[1189] The server uses social media APIs (e.g., Twitter API) to automatically create and send a warning post containing the suspicious audio clip and metadata.

[1190] Example (voice recording function)

[1191] While the homeowner is away, the moment the sound of a window breaking is detected, the timestamp "2023-10-01 22:30:00" is added. The app automatically posts a message to Twitter saying "2023-10-01 22:30:00 The sound of a window breaking was detected!"

[1192] 2. Image Recording Function

[1193] Device (home camera) operation

[1194] 1. The user installs the camera

[1195] Users install home cameras and configure them to monitor specific areas.

[1196] 2. Real-time recognition

[1197] The device analyzes the live video stream and uses an AI image recognition engine (e.g., AWS Rekognition) to identify suspicious activity in real time.

[1198] 3. Motion Detection

[1199] When suspicious activity is detected, the camera automatically records the moment.

[1200] 4. Add a timestamp to the image

[1201] Recorded images are given a timestamp based on the current date and time.

[1202] 5. Data transmission

[1203] The terminal transmits the image data and a timestamp to the server.

[1204] Server Operation

[1205] 1. Receiving Data

[1206] The server receives the image data sent from the terminal and stores it in cloud storage.

[1207] 2. Adding Metadata

[1208] The server adds metadata to the stored image data, such as timestamps and suspicious movement detection results.

[1209] 3. Posting to social media

[1210] The server uses the SNS API to automatically create and send alerts containing images and metadata of suspicious activity.

[1211] Example (image recording function)

[1212] If an image of a stranger walking around the house while the homeowner is out is detected, it will be given the timestamp "2023-10-02 02:15:00." The app will then automatically post the message "2023-10-02 02:15:00 A stranger is walking around the house!" to Twitter.

[1213] Example prompts for generative AI models

[1214] Audio capture: The sound of a window breaking has been detected. Recorded at 2023-10-01 22:30:00.

[1215] Image Capture: An unknown individual walking through the home. Recorded at 2023-10-02 02:15:00.

[1216] Twitter Security Warning

[1217] According to the above-described embodiment, the present invention can significantly improve security within the home.

[1218] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1219] Step 1:

[1220] The user launches the app. The user launches the audio recording app installed on their smartphone and enables the audio recording function. This puts the microphone in a state where it is always ready to capture audio. The input is the user's operation, and the output is the state where the recording function is enabled.

[1221] Step 2:

[1222] Configure the device so that it runs in the background and constantly captures audio using the microphone. Input is the system setting, and output is the microphone constantly capturing audio.

[1223] Step 3:

[1224] Voice data is captured. The device uses an AI voice recognition engine (e.g., Google Cloud Speech-to-Text) to analyze the voice data in real time and detect suspicious voices. The input is real-time voice data from the microphone, and the output is analyzed voice text data. Specifically, the voice data is sent to the AI ​​voice recognition engine and converted into text.

[1225] Step 4:

[1226] A timestamp is added to the audio data. A timestamp based on the current date and time is automatically added to detected suspicious audio. The input is analyzed audio text data, and the output is audio text data with a timestamp. Specifically, the current date and time is obtained and added to the audio data.

[1227] Step 5:

[1228] Data is transmitted. The terminal transmits the recorded voice data and a timestamp to the server. The input is voice text data with a timestamp, and the output is data transmission to the server. Specifically, the data is transmitted to the server using an HTTP request.

[1229] Step 6:

[1230] The server receives the data. The server receives the voice data sent from the device and stores it in cloud storage (e.g., AWS S3). The input is time-stamped voice text data, and the output is data storage in cloud storage. Specifically, the received data is stored using the cloud storage API.

[1231] Step 7:

[1232] Add metadata. The server adds metadata such as timestamps and text conversion of the audio content to the stored audio data. The input is audio data stored in cloud storage, and the output is audio data with metadata. Specific operations include checking the timestamp and adding the text conversion results.

[1233] Step 8:

[1234] Post to social media. The server automatically creates and sends a warning post containing the suspicious audio clip and metadata using a social media API (e.g., Twitter API). The input is audio data with metadata, and the output is a warning post to the social media platform. The specific operation is to send a message and audio clip using the social media API.

[1235] Step 9:

[1236] The user installs a camera. The user installs a home camera and configures it to monitor a specific area. The input is the user's operation, and the output is the camera installation completion and monitoring configuration.

[1237] Step 10:

[1238] Real-time recognition. The device analyzes the live video stream and uses an AI image recognition engine (e.g., AWS Rekognition) to identify suspicious activity in real time. The input is the live video stream from the camera, and the output is analyzed motion data. Specifically, the video stream is sent to the AI ​​image recognition engine for analysis.

[1239] Step 11:

[1240] It detects motion. When a suspicious movement is detected, the camera automatically records the moment. The input is the analyzed movement data, and the output is the recorded image data. Specifically, a frame is captured at the moment an abnormality is detected.

[1241] Step 12:

[1242] A timestamp is added to image data. A timestamp based on the current date and time is added to recorded images. The input is recorded image data, and the output is image data with a timestamp. Specifically, the current date and time is obtained and added to the image data.

[1243] Step 13:

[1244] Data is sent. The terminal sends image data and a timestamp to the server. The input is image data with a timestamp, and the output is data sent to the server. Specifically, the data is sent to the server using an HTTP request.

[1245] Step 14:

[1246] The server receives the data. The server receives the image data sent from the device and stores it in cloud storage. The input is image data with a timestamp, and the output is data storage in cloud storage. Specifically, the received data is stored using the cloud storage API.

[1247] Step 15:

[1248] Add metadata. The server adds metadata such as a timestamp and suspicious movement recognition results to the stored image data. The input is image data stored in cloud storage, and the output is image data with metadata. Specific operations include checking the timestamp and adding movement recognition results.

[1249] Step 16:

[1250] Post to SNS. The server uses SNS API to automatically create and send an alert containing images and metadata of suspicious activity. The input is image data with metadata, and the output is a warning post to the SNS platform. The specific operation is to send a message and image data using the SNS API.

[1251] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1252] The present invention aims to combine an emotion engine with a system that automatically records the voices and images of specific people, timestamps the data, and automatically shares it through a social networking platform, thereby recognizing the user's emotions based on the recorded data and providing information corresponding to those emotions.

[1253] 1. Voice Recording Function

[1254] 1.1 Device (smartphone) operation

[1255] 1. The user launches the app

[1256] The user launches a voice recording application installed on the smartphone and enables the voice recording function.

[1257] 2. Device Settings

[1258] The device will run in the background and be configured to capture audio constantly using the microphone.

[1259] 3. Capture audio data

[1260] It uses AI models to analyze voice data in real time and detect new utterances.

[1261] 4. Emotion Recognition by Emotion Engine

[1262] The device sends the captured voice data to an emotion engine to recognize the user's emotions, which analyzes the tone, strength, and speed of the voice to identify emotions.

[1263] 5. Adding a timestamp to audio data

[1264] Any new audio clips detected will be given a timestamp based on the current date and time.

[1265] 6. Data transmission

[1266] The device sends the recorded voice data, a timestamp, and information on the recognized emotion to the server.

[1267] 1.2 Server Operation

[1268] 1. Receiving Data

[1269] The server receives the voice data and emotion information sent from the terminal and checks the consistency of the data.

[1270] 2. Adding Metadata

[1271] The server adds metadata to the received audio data, such as timestamps, text conversion of the audio content, and emotional information.

[1272] 3. Posting to social media

[1273] The server creates and automatically sends a post to the configured social networking platform (e.g., LINE), which includes the audio clip, a timestamp, the text of the audio, and emotional information.

[1274] 4. Emotion-based replies and comments generation

[1275] The server generates appropriate replies and comments based on the recognized emotions and automatically posts them to social media platforms.

[1276] 2. Image Recording Function

[1277] 2.1 Device (home camera) operation

[1278] 1. The user installs the camera

[1279] A user installs a home camera and configures it to recognize specific people.

[1280] 2. Real-time recognition

[1281] The device analyzes the video stream and uses AI models to identify specific people (children) in real time.

[1282] 3. Object Detection

[1283] When the camera recognizes that a specific person is holding a new object, it automatically takes a picture of the object.

[1284] 4. Add a timestamp to the image

[1285] Each captured image is given a timestamp based on the current date and time.

[1286] 5. Data transmission

[1287] The terminal transmits the image data and a timestamp to the server.

[1288] 2.2 Server Operation

[1289] 1. Receiving Data

[1290] The server receives the image data sent from the terminal and checks the integrity of the data.

[1291] 2. Adding Metadata

[1292] The server adds metadata such as a timestamp and object recognition results to the received image data.

[1293] 3. Posting to social media

[1294] The server creates and automatically sends a post to the configured social media platform (e.g., Instagram), which includes the image, a timestamp, and a description of the object.

[1295] 4. Emotion-based replies and comments generation

[1296] The server generates appropriate replies and comments based on the recognized emotions and automatically posts them to social media platforms.

[1297] Specific examples

[1298] Examples of voice recording functions

[1299] 1. The user launches the app

[1300] The user launches a voice recording app on their smartphone and enables the recording function.

[1301] 2. Capture audio data

[1302] The moment a child says "Daddy" for the first time, the device detects the new utterance and uses its emotion engine to recognize that the utterance contains the emotion of "joy."

[1303] 3. Adding a timestamp

[1304] The audio clip will be timestamped with "2023-10-02 13:20:00".

[1305] 4. Data transmission and storage

[1306] The terminal transmits the voice data, timestamp, and emotion information to the server, which stores them in cloud storage.

[1307] 5. Social media posts and emotional comment generation

[1308] The server automatically posts an audio clip to LINE with the message "2023-10-02 13:20:00 He said 'Daddy' (joy)," and also posts the comment "What a wonderful first word!"

[1309] Example of image recording function

[1310] 1. The user installs the camera

[1311] The user installs the home camera and configures the recording settings.

[1312] 2. Object detection and photography

[1313] The camera detects when a child is holding a new toy (a blue ball) and automatically captures the moment.

[1314] 3. Emotion Recognition by Emotion Engine

[1315] The emotion engine analyzes the audio and video data obtained from the captured images and recognizes that the child is "excited."

[1316] 4. Adding a timestamp

[1317] The image will be given a timestamp of "2023-10-02 14:10:00".

[1318] 5. Data transmission and storage

[1319] The terminal transmits the image data, timestamp, and emotion information to the server, which stores the data in cloud storage.

[1320] 6. Social media posts and emotional comment generation

[1321] The server automatically posts the image to Instagram with the message "2023-10-02 14:10:00 New toy: Blue ball (excited)" and also posts the comment "You're so excited about your new toy!"

[1322] With the above-described configuration, the present invention not only allows users to effortlessly record moments of their child's growth and easily share them with family and relatives, but also allows users to provide information according to their emotions.

[1323] The processing flow will be explained below.

[1324] Voice recording function processing steps

[1325] Step 1:

[1326] The user launches a voice recording application installed on the smartphone and enables the voice recording function.

[1327] Step 2:

[1328] Set your device to run in the background and constantly capture audio using the microphone.

[1329] Step 3:

[1330] The device analyzes the captured voice data in real time to detect new utterances, using an AI model to analyze the voice data and identify new utterances.

[1331] Step 4:

[1332] When a new utterance is detected, the device generates an audio clip and gives it a timestamp based on the current date and time.

[1333] Step 5:

[1334] The device sends the audio clip to an emotion engine that analyzes the user's emotions, which determine emotions based on factors such as tone, volume, and vocal inflection.

[1335] Step 6:

[1336] The emotion engine recognizes emotions (e.g., joy, sadness, excitement, etc.) contained in an audio clip and attaches the emotion data to the audio clip.

[1337] Step 7:

[1338] The device transmits the audio clip, timestamp, and emotion data to the server using a common network communication protocol.

[1339] Step 8:

[1340] The server receives the voice data sent from the terminal and checks the integrity of the data.

[1341] Step 9:

[1342] The server stores the received audio data in cloud storage, along with metadata such as timestamps and emotional data.

[1343] Step 10:

[1344] The server creates a post on a social networking platform (e.g., LINE) based on the stored audio data, which includes the audio clip, a timestamp, a transcription of the audio content, and emotion data.

[1345] Step 11:

[1346] The server sends the created post to the social media platform, where it is automatically posted to the user's configured account.

[1347] Step 12:

[1348] The server generates an appropriate reply or comment based on the recognized emotion and automatically posts it to the social media platform. For example, if "joy" is recognized, a comment such as "That's a great first word!" will be generated.

[1349] Image recording function processing steps

[1350] Step 1:

[1351] A user installs a home camera and configures it to recognize specific people.

[1352] Step 2:

[1353] The device (home camera) analyzes the video stream and is configured to identify specific people (children) in real time using an AI model.

[1354] Step 3:

[1355] The device detects the object being held by a particular person using an object recognition algorithm.

[1356] Step 4:

[1357] When a new object is detected, the device automatically captures the moment and generates an image.

[1358] Step 5:

[1359] The device will add a timestamp based on the current date and time to the generated image.

[1360] Step 6:

[1361] The device sends image data to the emotion engine, which analyzes the user's emotions. The emotion engine identifies emotions based on the user's facial expressions, movements, voice, etc.

[1362] Step 7:

[1363] The emotion engine recognizes emotions (e.g., excitement, surprise, joy, etc.) contained in the image data and attaches the emotion data to the image data.

[1364] Step 8:

[1365] The device transmits the image data, timestamp, and emotion data to the server, using a common network communication protocol.

[1366] Step 9:

[1367] The server receives the image data sent from the terminal and checks the consistency of the data.

[1368] Step 10:

[1369] The server stores the received image data in cloud storage, along with metadata such as timestamps and emotion data.

[1370] Step 11:

[1371] The server creates a post for a social media platform (e.g., Instagram) based on the stored image data, which includes the image, a timestamp, and emotion data.

[1372] Step 12:

[1373] The server sends the created post to the social media platform, where it is automatically posted to the user's configured account.

[1374] Step 13:

[1375] The server generates an appropriate reply or comment based on the recognized emotion and automatically posts it to the social media platform. For example, if "excitement" is recognized, a comment such as "You're crazy about your new toy!" will be generated.

[1376] Through the above processing steps, audio and image recording combined with an emotion engine is automatically performed, and the data can be shared quickly and easily, while also providing information according to emotions.

[1377] Example 2

[1378] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1379] Current systems that record audio and images and share them on social media platforms have difficulty in properly recognizing users' emotions and providing information. Furthermore, they lack the functionality to detect new utterances from recorded audio or automatically recognize objects in images, requiring users to put in a lot of effort.

[1380] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes means for recording the voice of a specific person, means for analyzing the voice data and recognizing the content of the speech, means for recognizing emotions from the voice data, means for assigning a timestamp to the voice data, means for storing the voice data and the recognized emotional information in cloud storage, means for automatically posting the time-stamped voice data including the voice data and the recognized emotional information to an SNS platform, means for detecting an object held by the specific person, means for capturing an image of the detected object, means for analyzing image data and extracting a description of the object, means for recognizing emotions from the image data, means for assigning a timestamp to the image data, means for storing the image data and the recognized emotional information in cloud storage, means for automatically posting the time-stamped image data including the image data, the recognized object description, and the emotional information to an SNS platform, and means for detecting new comments from the recorded voice. This makes it possible to efficiently record a user's voice and image data along with the recognized emotions and automatically share them on an SNS platform.

[1381] "Specific person" refers to an individual who has been designated by the system as a target for recording audio and image data.

[1382] "Audio recording means" refers to devices or software that use a microphone to capture environmental sounds and store them as digital data.

[1383] "Means for analyzing and recognizing what is being said" refers to natural language processing technology that converts voice data into text and understands its content.

[1384] "Means for recognizing emotions" refers to algorithms or engines for identifying emotions from audio or image data.

[1385] "Means for adding a time stamp" refers to a function that adds the date and time of collection to recorded data in a digital format.

[1386] "Cloud storage means" refers to a service that allows data to be uploaded and stored on a remote server via the Internet.

[1387] "Means for automatically posting to social media platforms" refers to a program that connects to a designated social media service and automatically shares data.

[1388] "Means for detecting objects" refers to technology for analyzing image data to recognize and identify specific objects.

[1389] "Means for taking images of detected objects" refers to devices or software that use a camera to capture images containing recognized objects and store the digital data.

[1390] "Means for analyzing image data and extracting object descriptions" refers to a technique for analyzing image data and generating text information about objects contained therein.

[1391] "Means for detecting new utterances from audio recordings" refers to algorithms for identifying and detecting newly uttered words or phrases from continuously recorded audio data.

[1392] The present invention aims to combine an emotion engine with a system that automatically records the voices and images of specific people, timestamps the data, and automatically shares it through a social networking platform, thereby recognizing the user's emotions based on the recorded data and providing information corresponding to those emotions.

[1393] Voice recording function embodiment

[1394] Device (smartphone) operation

[1395] 1. The user launches the app

[1396] A user launches a voice recording app installed on their smartphone and enables the recording function. The app uses the Google Speech-to-Text API to convert the voice data into text in real time.

[1397] 2. Device Settings

[1398] The device runs in the background and is set to capture audio using the microphone at all times. Users can enable the recording feature in a specific settings menu and allow it to run in the background.

[1399] 3. Capture audio data

[1400] When voice is input, the app on the device calls the Google Speech-to-Text API to convert it into text and checks the content.

[1401] 4. Emotion Recognition by Emotion Engine

[1402] The converted text and audio data is sent to IBM Watson Tone Analyzer, where emotions are analyzed based on tone, strength, speed, etc.

[1403] 5. Adding a timestamp to audio data

[1404] The detected audio clips are given a timestamp based on the current date and time obtained from the device's internal clock.

[1405] 6. Data transmission

[1406] The device uses HTTPS communication to send the recorded audio data, timestamps, and emotion information to the server.

[1407] Server Operation

[1408] 1. Receiving Data

[1409] The server receives the voice data and emotion information sent from the device via the HTTPS protocol, and then checks the data checksum to verify its integrity.

[1410] 2. Adding Metadata

[1411] Metadata such as text conversion results and emotional information is added to the received voice data and stored in a database.

[1412] 3. Posting to social media

[1413] The server automatically creates and sends a post to the configured social networking platform (e.g., LINE) containing the audio clip, a timestamp, the text of the audio content, and emotional information.

[1414] 4. Emotion-based replies and comments generation

[1415] Based on the recognized sentiment, appropriate replies and comments are generated and automatically posted to social media platforms.

[1416] Image recording function embodiment

[1417] Device (home camera) operation

[1418] 1. The user installs the camera

[1419] A user installs a home camera in a specific location and configures the camera to identify specific people (e.g., children) within a certain range.

[1420] 2. Real-time recognition

[1421] The camera uses OpenCV to analyze the video stream in real time and recognize specific people.

[1422] 3. Object Detection

[1423] Detects when a specific person is holding a new object, and captures the moment with the camera.

[1424] 4. Add a timestamp to the image

[1425] Captured images are automatically time-stamped based on the current date and time.

[1426] 5. Data transmission

[1427] The captured image data and timestamp are sent to the server via HTTPS.

[1428] Server Operation

[1429] 1. Receiving Data

[1430] The server receives the image data sent from the terminal and checks the integrity of the data.

[1431] 2. Adding Metadata

[1432] A timestamp and object recognition results are added as metadata to the received image data.

[1433] 3. Posting to social media

[1434] The server automatically creates and sends a post to the configured social media platform (e.g., Instagram) containing the image, a timestamp, and a description of the object.

[1435] 4. Emotion-based replies and comments generation

[1436] Based on the recognized sentiment, appropriate replies and comments are generated and automatically posted to social media platforms.

[1437] Specific examples

[1438] Examples of voice recording functions

[1439] 1. The user launches the app

[1440] The user launches a voice recording app on their smartphone and enables the recording function.

[1441] 2. Capture audio data

[1442] The moment a child says "Daddy" for the first time, the device detects the new utterance and uses its emotion engine to recognize that the utterance contains the emotion of "joy."

[1443] 3. Adding a timestamp

[1444] The audio clip will be timestamped with "2023-10-02 13:20:00".

[1445] 4. Data transmission and storage

[1446] The terminal transmits the voice data, timestamp, and emotion information to the server, which stores them in cloud storage.

[1447] 5. Social media posts and emotional comment generation

[1448] The server automatically posts an audio clip to LINE with the message "2023-10-02 13:20:00 He said 'Daddy' (joy)" and also posts the comment "What a wonderful first word!"

[1449] Example of image recording function

[1450] 1. The user installs the camera

[1451] The user sets up the home camera and configures the recording settings.

[1452] 2. Object detection and photography

[1453] The camera detects when a child is holding a new toy (a blue ball) and automatically captures the moment.

[1454] 3. Emotion Recognition by Emotion Engine

[1455] The emotion engine analyzes the audio and video data obtained from the captured images and recognizes that the child is "excited."

[1456] 4. Adding a timestamp

[1457] The image will be given a timestamp of "2023-10-02 14:10:00".

[1458] 5. Data transmission and storage

[1459] The terminal transmits the image data, timestamp, and emotion information to the server, which stores the data in cloud storage.

[1460] 6. Social media posts and emotional comment generation

[1461] The server automatically posts the image to Instagram with the message "2023-10-02 14:10:00 New toy: Blue ball (excited)" and also posts the comment "You're so excited about your new toy!"

[1462] According to the above-described embodiment, the present invention enables users to record and share audio and image data without any special effort, and provides information according to their emotions.

[1463] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1464] Voice recording function processing steps

[1465] Device (smartphone) operation

[1466] Step 1:

[1467] The user launches the app

[1468] Input: The user operates the smartphone and taps the icon of the voice recording app.

[1469] Specific operation: The user finds the audio recording app in the list of apps on their smartphone and taps the icon.

[1470] Output: The audio recording app will launch and the initial screen for the recording function will be displayed.

[1471] Step 2:

[1472] Device settings

[1473] Input: The user taps the "Start Recording" button in the app's settings menu to enable the recording feature.

[1474] What it does: The user taps the "Start Recording" button on the app's main screen and approves background recording if prompted.

[1475] Output: The device will have access to the microphone and will be configured to capture audio continuously in the background.

[1476] Step 3:

[1477] Capture audio data

[1478] Input: Audio signals from the environment.

[1479] How it works: The device uses the Google Speech-to-Text API to convert voice data to text in real time, and this conversion occurs whenever new speech is detected.

[1480] Output: Text conversion results and analysis results of the audio data.

[1481] Step 4:

[1482] Emotion recognition by emotion engine

[1483] Input: Converted text data and audio data.

[1484] Specific operation: Text and audio data are sent to IBM Watson Tone Analyzer, which analyzes emotions based on tone, strength, speed, etc.

[1485] Output: Parsed emotion information (e.g., happiness, sadness, excitement, etc.).

[1486] Step 5:

[1487] Adding a timestamp to audio data

[1488] Input: Audio clip and analyzed emotion information.

[1489] Specific operation: Obtains date and time information from the device's internal clock and adds a timestamp as metadata to the audio clip.

[1490] Output: Timestamp audio data and emotion information.

[1491] Step 6:

[1492] Sending data

[1493] Input: Timestamp audio data, emotion information.

[1494] What it does: Sends recorded data to a server using the HTTPS protocol.

[1495] Output: Voice data and emotion information sent to the server.

[1496] Server Operation

[1497] Step 1:

[1498] Receiving data

[1499] Input: Voice data and emotion information sent from the device.

[1500] Specific operation: The server receives the data via the HTTPS protocol and checks the checksum to verify the integrity of the data.

[1501] Output: Received voice data, emotion information, and consistency check results.

[1502] Step 2:

[1503] Adding metadata

[1504] Input: Received voice data and emotion information.

[1505] Specific operation: Adds timestamps, text conversion results of audio content, emotional information, etc. to received data.

[1506] Output: Audio data with metadata.

[1507] Step 3:

[1508] Posting to social media

[1509] Input: Audio data with metadata.

[1510] Specific operation: Calls the API of a social media platform (e.g., LINE) and posts the audio clip, timestamp, audio text, and emotional information.

[1511] Output: Audio data posted on SNS.

[1512] Step 4:

[1513] Generate replies and comments based on emotions

[1514] Input: Parsed emotion information.

[1515] Specific behavior: Generate replies and comments based on emotional information and automatically post them to social media platforms.

[1516] Output: Replies and comments posted to social media.

[1517] Image recording function processing steps

[1518] Device (home camera) operation

[1519] Step 1:

[1520] The user installs the camera

[1521] Input: The user installs the home camera in a specific location and configures it.

[1522] Specific operation: The user fixes the camera in a designated position and sets up facial recognition for a specific person (e.g., a child) through the app.

[1523] Output: Camera installation completed and target person set.

[1524] Step 2:

[1525] Real-time recognition

[1526] Input: A video stream acquired in real time.

[1527] How it works: The camera uses OpenCV to analyze the video stream and identify specific people in real time.

[1528] Output: Identification result of a specific person.

[1529] Step 3:

[1530] Object detection

[1531] Input: Image data about an object held by a particular person.

[1532] What it does: It uses an object detection algorithm to recognize when a new object is being held by a person and captures that moment.

[1533] Output: Image data containing the new object.

[1534] Step 4:

[1535] Add a timestamp to the image

[1536] Input: Captured image data.

[1537] Specific operation: Obtains date and time information from the device's internal clock and adds a timestamp as metadata to the image data.

[1538] Output: Timestamped image data.

[1539] Step 5:

[1540] Sending data

[1541] Input: Timestamped image data.

[1542] Specific operation: Image data is sent to the server using the HTTPS protocol.

[1543] Output: Image data sent to the server.

[1544] Server Operation

[1545] Step 1:

[1546] Receiving data

[1547] Input: Image data sent from the device.

[1548] Specific operation: The server receives the data via the HTTPS protocol and checks the checksum to verify the integrity of the data.

[1549] Output: Received image data, consistency check results.

[1550] Step 2:

[1551] Adding metadata

[1552] Input: Received image data.

[1553] Specific operation: Adds metadata such as timestamps and object recognition results to the received data.

[1554] Output: Image data with metadata.

[1555] Step 3:

[1556] Posting to social media

[1557] Input: Image data with metadata.

[1558] Specific operation: Calls the API of a social media platform (e.g., Instagram) and posts an image, timestamp, and object description.

[1559] Output: Image data posted on SNS.

[1560] Step 4:

[1561] Generate replies and comments based on emotions

[1562] Input: Parsed emotion information.

[1563] Specific behavior: Generate replies and comments based on emotional information and automatically post them to social media platforms.

[1564] Output: Replies and comments posted to social media.

[1565] The above are the specific processing steps of the present invention, which allow users to record and share audio and image data without any special effort, and provide information according to their emotions.

[1566] (Application example 2)

[1567] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1568] Modern life demands that necessary information and services be provided based on a user's emotional state. However, existing technologies lack systems that can effectively analyze a user's emotions and provide appropriate responses based on those analysis results. In particular, it is difficult to automatically collect and analyze audio and image data to provide information tailored to the user's emotions. Furthermore, there is no system in place to reliably store this data in cloud storage and automatically post it to social media platforms. This makes it difficult to realize the personalized experiences users desire.

[1569] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1570] In this invention, the server includes means for recording the voice of a specific person, means for assigning a timestamp to the voice data, means for storing the voice data in cloud storage, means for automatically posting the time-stamped voice data to a social networking service platform, means for analyzing the voice data using an emotion analysis engine and recognizing the user's emotion, and means for providing the user with appropriate information based on the recognized emotion. This makes it possible to analyze the user's emotion in real time and provide information and services according to the emotion.

[1571] The term "specific person" refers to a specific individual who is a predetermined target.

[1572] "Audio recording means" means a means for storing the speech of a particular person in digital form using a device.

[1573] "Audio data" refers to data that has been converted into digital form and recorded.

[1574] The "means for adding a time stamp" is a means for adding the date and time when the data was recorded to the audio data or image data.

[1575] "Cloud storage" is a set of remote servers that store data over the internet.

[1576] A "social networking service platform" is an online service that enables users to share information and communicate with other users via the Internet.

[1577] "Automatic posting means" refers to a means for publishing specific data on a given platform without user intervention.

[1578] An "emotion analysis engine" is a set of algorithms and software that analyzes audio and image data and identifies the user's emotions from them.

[1579] The "means for providing appropriate information" is a means for providing information and services that are considered to be optimal for the user based on the recognized emotions.

[1580] This invention is a system that automatically records the voice and images of a specific person, adds a timestamp to the data, and automatically shares it through a social networking platform. Furthermore, by combining it with an emotion analysis engine, it can recognize the user's emotions from the recorded data and provide information according to those emotions.

[1581] A system for implementing the present invention uses the following configuration and associated hardware and software.

[1582] System configuration

[1583] 1. Device Operation

[1584] 1. Audio recording and emotion analysis

[1585] The device is equipped with a microphone for recording voice and an AI model (e.g., EmotionRecognizer) for analyzing the voice data. When a user launches the app and engages in normal conversation, the device captures the voice. The voice is analyzed in real time through an emotion analysis engine to recognize the user's emotions. This emotion analysis uses the tone, strength, and speed of the voice.

[1586] 2. Image recording and emotion analysis

[1587] If the device is equipped with a camera, it will detect an object held by a specific person and take a picture of it. The captured image data will also be passed through an emotion analysis engine to recognize the user's emotions. This analysis uses facial expressions and the situation contained in the image.

[1588] 3. Adding a timestamp and sending data

[1589] The recorded audio and video data is time-stamped based on the current date and time, stored in cloud storage, and then automatically sent to social media platforms.

[1590] 2. Server Operation

[1591] 1. Data Receipt and Storage

[1592] The server receives the voice and image data sent from the device, checks the integrity of the data, and also receives emotional information. It adds additional metadata (e.g., converted voice content and object recognition results) to the received data and stores it in cloud storage.

[1593] 2. Posting and providing information on social media

[1594] The server then posts to the configured social networking platform in the specified format, including audio clips, images, timestamps, analyzed text, and emotion information. Appropriate replies and comments are also automatically generated and posted based on the recognized emotion.

[1595] Hardware and software used

[1596] Hardware: Smartphone (microphone, speaker, camera), cloud server, storage

[1597] Software: EmotionRecognizer (AI model for emotion recognition), SNS API (for posting to SNS)

[1598] Specific examples

[1599] Example: A user opens a smartphone app and orders dinner while uttering emotional speech. For example, the user says, "I'm tired today." The app recognizes this speech as "sad" and suggests "comfort ice cream" as a meal. This information is then posted to social media with a timestamp, sharing the following: "2023-10-05 18:30:00 - Emotion: Sad, Suggested Meal: Comfort Ice Cream."

[1600] Example prompt sentence:

[1601] The app allows users to record their voices expressing their emotions, recognize their emotions based on the voice data, and recommend appropriate meals. After that, users can share this information on social media.

[1602] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1603] Step 1:

[1604] The device records the audio.

[1605] When a user launches the app and speaks, the device's microphone captures the audio. The input is the user's speech, and the output is digital audio data that can be used in subsequent processes.

[1606] Step 2:

[1607] The device tags the audio data with a timestamp.

[1608] This system adds a timestamp to recorded audio data by adding the current date and time information to the audio data. The input is digital audio data, and the output is audio data with a timestamp.

[1609] Step 3:

[1610] The device sends the voice data to the emotion analysis engine.

[1611] The device sends the recorded voice data to an emotion analysis engine, which analyzes the voice. The input is time-stamped voice data, and the output is analyzed emotion information. For example, the tone, strength, and speed of the voice are used to identify emotions.

[1612] Step 4:

[1613] The device saves the data to cloud storage.

[1614] The analyzed emotion information and the time-stamped voice data are sent to and stored in cloud storage. The input is the time-stamped voice data and the analyzed emotion information, and the output is the data stored in cloud storage.

[1615] Step 5:

[1616] The server receives the time-stamped audio data and emotion information.

[1617] The server receives the data sent to the cloud storage and checks the integrity of the data. The input is the data from the cloud storage, and the output is the data whose integrity has been checked.

[1618] Step 6:

[1619] The server adds metadata to the received data.

[1620] This converts audio data into text and adds additional metadata (e.g., a textual representation of the audio content). The input is time-stamped audio data with integrity check and analyzed emotion information, and the output is audio data with metadata.

[1621] Step 7:

[1622] The server automatically posts to the social networking service platform.

[1623] This system automatically posts time-stamped audio data, metadata, and emotional information to a social media platform. The input is audio data with metadata and analyzed emotional information, and the output is a post to a social media platform.

[1624] Step 8:

[1625] The server generates a comment based on the emotion and posts it to social media.

[1626] The system posts automatically generated comments based on the recognized emotions to a social media platform. The input is the analyzed emotion information, and the output is a social media post with the emotion-based comment.

[1627] Specifically, when a user says, "I'm tired today," the voice is recorded and a timestamp is added. The emotion analysis engine recognizes the voice as "sad" and saves it in cloud storage. The server then receives the data and posts the following to social media: "2023-10-05 18:30:00 - Emotion: Sad, Suggested Meal: Comforting Ice Cream." The server also simultaneously posts the comment, "Thank you for your hard work! Refresh yourself with some ice cream!"

[1628] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1629] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1630] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[1631] [Third embodiment]

[1632] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[1633] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[1634] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1635] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[1636] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1637] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1638] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1639] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1640] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1641] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1642] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1643] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[1644] The present invention provides a system for automatically recording audio and images of a specific person, adding a timestamp, and automatically sharing the recorded audio and images through a social networking platform. The following describes in detail an embodiment of the present invention.

[1645] 1. Voice Recording Function

[1646] 1.1 Device (smartphone) operation

[1647] 1. The user launches the app

[1648] The user launches a voice recording application installed on the smartphone and enables the voice recording function.

[1649] 2. Device Settings

[1650] The device will run in the background and be configured to capture audio constantly using the microphone.

[1651] 3. Capture audio data

[1652] The device uses AI models to analyze voice data in real time and detect new utterances.

[1653] 4. Adding a timestamp to audio data

[1654] Any new audio clips detected will automatically be given a timestamp based on the current date and time.

[1655] 5. Data transmission

[1656] The device sends the recorded audio data and a timestamp to the server.

[1657] 1.2 Server Operation

[1658] 1. Receiving Data

[1659] The server receives the voice data sent from the device and stores it in cloud storage.

[1660] 2. Adding Metadata

[1661] The server adds metadata to the stored audio data, such as timestamps and a transcription of the audio content.

[1662] 3. Posting to social media

[1663] The server automatically creates and sends a post containing the audio clip and metadata to the configured social media platform (e.g., LINE).

[1664] 2. Image Recording Function

[1665] 2.1 Device (home camera) operation

[1666] 1. The user installs the camera

[1667] A user installs a home camera and configures it to recognize specific people.

[1668] 2. Real-time recognition

[1669] The device analyzes the video stream and uses AI models to identify specific people (children) in real time.

[1670] 3. Object Detection

[1671] When the camera recognizes that a specific person is holding a new object, it automatically takes a picture of the object.

[1672] 4. Add a timestamp to the image

[1673] Each captured image is given a timestamp based on the current date and time.

[1674] 5. Data transmission

[1675] The terminal transmits the image data and a timestamp to the server.

[1676] 2.2 Server Operation

[1677] 1. Receiving Data

[1678] The server receives the image data sent from the terminal and stores it in cloud storage.

[1679] 2. Adding Metadata

[1680] The server adds metadata such as timestamps and object recognition results to the stored image data.

[1681] 3. Posting to social media

[1682] The server automatically creates and sends a post containing the image and metadata to the configured social media platform (e.g., Instagram).

[1683] Specific examples

[1684] Examples of voice recording functions

[1685] 1. The user launches the app

[1686] The user launches a voice recording app on their smartphone and enables the recording function.

[1687] 2. Capture audio data

[1688] The moment your child says "mom" for the first time, the device detects the new utterance.

[1689] 3. Adding a timestamp

[1690] The audio clip will be timestamped with "2023-10-01 10:00:00".

[1691] 4. Data transmission and storage

[1692] The device sends the voice data to the server, which stores it in cloud storage.

[1693] 5. Social Media Posting

[1694] The server automatically posts the audio clip to LINE with the message "2023-10-01 10:00:00 First comment: Mom."

[1695] Example of image recording function

[1696] 1. The user installs the camera

[1697] The user installs the home camera and configures the recording settings.

[1698] 2. Object detection and photography

[1699] The camera detects when a child has a new toy (a red car) and automatically captures the moment.

[1700] 3. Adding a timestamp

[1701] The image will be given a timestamp of "2023-10-01 15:00:00".

[1702] 4. Data transmission and storage

[1703] The device sends the image data to the server, which stores it in cloud storage.

[1704] 5. Social Media Posting

[1705] The server automatically posts the image to Instagram with the message "2023-10-01 15:00:00 New Toy: Red Car."

[1706] As described above, the present invention allows users to effortlessly record moments of their children's growth and easily share them with family and relatives.

[1707] The processing flow will be explained below.

[1708] Voice recording function processing steps

[1709] Step 1:

[1710] The user launches a voice recording application installed on the smartphone and enables the voice recording function.

[1711] Step 2:

[1712] Set your device to run in the background and constantly capture audio using the microphone.

[1713] Step 3:

[1714] The device analyzes the captured voice data in real time to detect new utterances, using an AI model to analyze the voice data and identify new utterances.

[1715] Step 4:

[1716] When a new utterance is detected, the device generates an audio clip and gives it a timestamp based on the current date and time.

[1717] Step 5:

[1718] The device transmits the audio clip and timestamp to the server, using a common network communication protocol.

[1719] Step 6:

[1720] The server receives the voice data sent from the terminal and checks the integrity of the data.

[1721] Step 7:

[1722] The server stores the received audio data in cloud storage, along with metadata such as timestamps.

[1723] Step 8:

[1724] The server creates a post on a social networking platform (e.g., LINE) based on the stored audio data. This post includes the audio clip, a timestamp, and a brief text description of the audio.

[1725] Step 9:

[1726] The server sends the created post to the social media platform, where it is automatically posted to the user's configured account.

[1727] Image recording function processing steps

[1728] Step 1:

[1729] A user installs a home camera and configures it to recognize specific people.

[1730] Step 2:

[1731] The device (home camera) analyzes the video stream and is configured to identify specific people (children) in real time using an AI model.

[1732] Step 3:

[1733] The device detects the object being held by a particular person using an object recognition algorithm.

[1734] Step 4:

[1735] When a new object is detected, the device automatically captures the moment and generates an image.

[1736] Step 5:

[1737] The device will add a timestamp based on the current date and time to the generated image.

[1738] Step 6:

[1739] The device transmits the image data and a timestamp to the server, using a common network communication protocol.

[1740] Step 7:

[1741] The server receives the image data sent from the terminal and checks the consistency of the data.

[1742] Step 8:

[1743] The server stores the received image data in cloud storage, along with metadata such as timestamps and object recognition results.

[1744] Step 9:

[1745] The server creates a post on a social media platform (e.g., Instagram) based on the stored image data, which includes the image, a timestamp, and a brief description of the object.

[1746] Step 10:

[1747] The server sends the created post to the social media platform, where it is automatically posted to the user's configured account.

[1748] Through the above processing steps, audio and image recordings are automatically performed, and the data can be shared quickly and easily.

[1749] Example 1

[1750] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1751] Conventional audio and image recording systems are inadequate in efficiently recording important moments of specific people and automatically sharing them via social media due to the time and effort required. Furthermore, technology to improve the accuracy of audio and image analysis using AI models is underdeveloped. Therefore, there is a need for a method that allows users to record and share information quickly, accurately, and without the hassle of manual effort.

[1752] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1753] In this invention, the server includes means for recording the voice of a specific person, means for timestamp-adding the voice data, means for storing the voice data in cloud storage, means for automatically posting the time-stamped voice data to a social networking platform, means for utilizing a generative AI model for analyzing the voice, and means for performing named entity recognition, thereby enabling users to effortlessly and accurately record and quickly share important moments of a specific person.

[1754] An "audio recording device" is a device whose purpose is to capture the voice of a specific person and store it as digital data.

[1755] "Means for adding time stamps" refers to a means for adding information about the date and time of recording or shooting to recorded audio data or photographed image data.

[1756] "Means for saving audio data in cloud storage" refers to means for storing captured audio data in a data storage service on the Internet.

[1757] "Means for automatically posting audio data with a timestamp to a social networking platform" means a means for automatically publishing and sharing audio data with a timestamp on a pre-designated social networking service.

[1758] "Means using generative AI models to analyze speech" refers to means that utilize generated artificial intelligence models to analyze recorded speech data and identify important utterances and speech features.

[1759] "Means for performing named entity recognition" refers to means for extracting and identifying specific names or proper nouns from analyzed speech data.

[1760] The "means for detecting an object held by a specific person" is a means for automatically recognizing an object held by a specific person using a camera.

[1761] The "means for taking an image of a detected object" refers to a means for capturing an image of a recognized object and storing it as digital data.

[1762] "Means for saving image data in cloud storage" refers to means for storing captured image data in a data storage service on the Internet.

[1763] "Means for automatically posting image data with a timestamp to a social networking platform" refers to a means for automatically publishing and sharing image data with a timestamp attached to a pre-designated social networking service.

[1764] "Means utilizing a generative AI model for image recognition" means means utilizing a generative artificial intelligence model to analyze captured image data and identify objects within the image.

[1765] "Means for identifying object characteristics" refers to means for extracting and identifying characteristics such as size, shape, and color of a specific object from captured image data.

[1766] The "means for detecting new utterances" is a means for automatically identifying and extracting new utterances that are different from conventional ones from recorded audio data.

[1767] The present invention is a system that automatically records the voice and images of a specific person, adds a timestamp, and automatically shares them through a social networking platform. The detailed steps for implementing this system are described below.

[1768] Voice recording function embodiment

[1769] Device (smartphone) operation

[1770] 1. The user launches the app

[1771] The user launches the audio recording app installed on their smartphone and taps the "Start Recording" button to enable the recording function, which puts the device into audio recording mode.

[1772] 2. Configure your device

[1773] The device gets the necessary permissions from the user to operate in the background, including permission to use the microphone and to operate in the background, so the device is always ready to capture audio.

[1774] 3. Capture audio data

[1775] The device uses a microphone to capture voice data in real time, and uses the Google Speech-to-Text API to analyze the captured voice data and detect new utterances with an AI model.

[1776] 4. Adding a timestamp to audio data

[1777] When a new utterance is detected, the audio clip is timestamped with the current date and time information. For example, if the word "Mom" was uttered on 2023-10-01 10:00:00, the audio clip will be timestamped with this time.

[1778] 5. Send the data

[1779] The audio data and timestamp are sent to the cloud server using the HTTPS protocol.

[1780] Server Operation

[1781] 1. Receive data

[1782] The server receives the audio data and timestamps sent from the device, and stores this data in a database such as AWS RDS.

[1783] 2. Add metadata

[1784] The server converts the voice data into text and obtains the text data. It also adds timestamps and named entity recognition metadata. For example, the converted text is saved as "2023-10-01 10:00:00 Mom."

[1785] 3. Post to social media

[1786] The server uses the LINE Messaging API to automatically create and send a post containing the audio clip and metadata to the social media platform.

[1787] Image recording function embodiment

[1788] Device (home camera) operation

[1789] 1. The user installs the camera

[1790] The user simply installs the device in a suitable location and configures the camera via a dedicated smartphone app. Once the settings, including network connection, are complete, the camera goes into standby mode.

[1791] 2. Real-time recognition

[1792] The camera captures the video stream in real time and uses the Google Vision API to identify specific people, which is then used to determine whether a specific person is present in the frame.

[1793] 3. Detecting objects

[1794] The camera detects the object a specific person is holding using the YOLO model, etc. For example, if a child is holding a new toy (a red car), the camera will capture that moment.

[1795] 4. Add a timestamp to your images

[1796] A timestamp based on the current date and time is added to the captured image. For example, a timestamp such as "2023-10-01 15:00:00" is added to the image.

[1797] 5. Send the data

[1798] The captured image data and timestamp are sent to a cloud server using the HTTPS protocol.

[1799] Server Operation

[1800] 1. Receive data

[1801] The server receives the image data and timestamp sent from the device, and stores the data in cloud storage such as Google Cloud Storage.

[1802] 2. Add metadata

[1803] The server performs object recognition processing on the image data and adds the recognition results and a timestamp as metadata. For example, it adds metadata such as "2023-10-01 15:00:00 New toy: Red car."

[1804] 3. Post to social media

[1805] The server automatically creates and sends a post containing image data and metadata to the specified social media platform using the Instagram Graph API.

[1806] Specific examples

[1807] Examples of voice recording functions

[1808] 1. The user launches the app

[1809] The user launches a voice recording app on their smartphone and enables the recording function.

[1810] 2. Capture audio data

[1811] The moment your child says "mom" for the first time, the device detects the new utterance.

[1812] 3. Add a timestamp

[1813] The audio clip will be timestamped with "2023-10-01 10:00:00".

[1814] 4. Send and store data

[1815] The device sends the voice data to the server, which stores it in cloud storage.

[1816] 5. Post to social media

[1817] The server automatically posts the audio clip to LINE with the message "2023-10-01 10:00:00 First comment: Mom."

[1818] Example of image recording function

[1819] 1. The user installs the camera

[1820] The user installs a home camera and configures the recording settings.

[1821] 2. Detect and photograph objects

[1822] The camera detects when a child has a new toy (a red car) and automatically captures the moment.

[1823] 3. Add a timestamp

[1824] The image will be given a timestamp of "2023-10-01 15:00:00".

[1825] 4. Send and store data

[1826] The device sends the image data to the server, which stores it in cloud storage.

[1827] 5. Post to social media

[1828] The server automatically posts the image to Instagram with the message "2023-10-01 15:00:00 New Toy: Red Car."

[1829] This system allows users to effortlessly record important moments of specific people with high accuracy and quickly share them via social media.

[1830] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1831] Voice recording function embodiment

[1832] Device (smartphone) operation

[1833] Step 1:

[1834] The user launches the audio recording app installed on their smartphone. The user taps the "Start Recording" button to enable recording.

[1835] Input: User taps, launched applications

[1836] Output: Recording start trigger

[1837] Specific behavior:

[1838] Tap the application icon.

[1839] The application will launch and a Start Recording button will appear.

[1840] The user taps the start recording button.

[1841] Step 2:

[1842] The device will ask the user for the necessary permissions to operate in the background, such as permission to use the microphone and permission to operate in the background.

[1843] Input: User permission operation

[1844] Output: Allow background operation and microphone usage

[1845] Specific behavior:

[1846] The device will display a popup asking permission to run in the background and use the microphone.

[1847] The user selects "Allow."

[1848] Step 3:

[1849] The device captures audio in real time through a microphone and analyzes it using a generative AI model (Google Speech-to-Text API) to detect new utterances.

[1850] Input: Real-time captured audio data

[1851] Output: Audio clip where new utterances were detected

[1852] Specific behavior:

[1853] The microphone continues to capture audio.

[1854] The audio data is sent to the Google Speech-to-Text API.

[1855] The API analyzes the audio data and detects new utterances.

[1856] Step 4:

[1857] When a new utterance is detected, the audio clip is given a timestamp based on the current date and time.

[1858] Input: Audio data in which new utterances are detected

[1859] Output: Time-stamped audio data

[1860] Specific behavior:

[1861] Get the current date and time information.

[1862] Add timestamps to audio clips.

[1863] Step 5:

[1864] The device sends the time-stamped audio data to the server using the HTTPS protocol.

[1865] Input: Timestamped audio data

[1866] Output: The audio data is sent to the server.

[1867] Specific behavior:

[1868] The data is sent to the server via HTTPS.

[1869] The server receives the data.

[1870] Server Operation

[1871] Step 6:

[1872] The server receives the audio data and timestamp sent from the device and stores it in cloud storage (AWS RDS).

[1873] Input: Audio data with timestamps

[1874] Output: Audio data stored in cloud storage

[1875] Specific behavior:

[1876] The server receives the data.

[1877] Store the audio data in AWS RDS.

[1878] Step 7:

[1879] The server performs text conversion processing on the voice data and adds a timestamp and named entity recognition data to the converted text data.

[1880] Input: Saved audio data

[1881] Output: Audio data with text conversion and metadata

[1882] Specific behavior:

[1883] The server sends the voice data to the generative AI model (voice analysis model).

[1884] Time stamps and named entities are added to the obtained text data.

[1885] Step 8:

[1886] The server automatically creates and sends a post containing the audio clip and the attached metadata to the SNS platform (LINE) using the SNS API.

[1887] Input: Audio data with metadata

[1888] Output: Audio data posted to SNS

[1889] Specific behavior:

[1890] Use the LINE Messaging API to create a message.

[1891] Post the generated message to LINE.

[1892] Image recording function embodiment

[1893] Device (home camera) operation

[1894] Step 1:

[1895] The user installs the home camera in an appropriate location and configures the camera via their smartphone.

[1896] Input: Camera settings operated by the user

[1897] Output: Camera starts operating

[1898] Specific behavior:

[1899] Connect the camera to a power source and position it appropriately.

[1900] Use the smartphone app to configure the camera settings (such as network connection settings).

[1901] Step 2:

[1902] The camera captures the video stream in real time and uses a generative AI model (Google Vision API) to identify specific people.

[1903] Input: Real-time captured video data

[1904] Output: Video data in which a specific person is identified

[1905] Specific behavior:

[1906] The camera continues to capture footage.

[1907] The video data is sent to the Google Vision API.

[1908] The API identifies a specific person.

[1909] Step 3:

[1910] Use an AI model (YOLO) to detect objects held by a specific person. Check if a person is holding a new object.

[1911] Input: Video data of identified person

[1912] Output: Data showing the object being held

[1913] Specific behavior:

[1914] Send the video data to the YOLO model.

[1915] The model detects objects and recognizes what a person is holding.

[1916] Step 4:

[1917] When a new object is detected, the camera automatically captures the moment and adds a timestamp to the image data.

[1918] Input: Video data of a person holding a new object

[1919] Output: Image data with timestamps

[1920] Specific behavior:

[1921] Get the current date and time information.

[1922] Add a timestamp to the image.

[1923] Step 5:

[1924] The device sends the image data and a timestamp to the server using the HTTPS protocol.

[1925] Input: Image data with timestamps

[1926] Output: Image data sent to the server

[1927] Specific behavior:

[1928] Send the data to the server over HTTPS.

[1929] The server receives the data.

[1930] Server Operation

[1931] Step 6:

[1932] The server receives the image data and timestamp sent from the device and stores them in Google Cloud Storage.

[1933] Input: Image data with timestamp

[1934] Output: Image data stored in cloud storage

[1935] Specific behavior:

[1936] The server receives the data.

[1937] Store image data in Google Cloud Storage.

[1938] Step 7:

[1939] The server performs object recognition processing on the image data and adds the recognition results and a timestamp as metadata.

[1940] Input: Saved image data

[1941] Output: Image data with metadata

[1942] Specific behavior:

[1943] The server sends the image data to the Google Vision API.

[1944] The object recognition result and a timestamp are added to the image data.

[1945] Step 8:

[1946] The server automatically creates and sends a post containing image data and attached metadata to a social networking platform (Instagram) using the social networking API.

[1947] Input: Image data with metadata

[1948] Output: Image data posted to SNS

[1949] Specific behavior:

[1950] Use the Instagram Graph API to generate messages.

[1951] Post the generated message to Instagram.

[1952] The above is the flow of the specific processing steps of this system. In this way, users can easily and accurately record important moments of specific people and quickly share them via social media.

[1953] (Application example 1)

[1954] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1955] Improving safety and security within the home has become an important issue in modern society. Responding quickly to suspicious sounds or movements, especially when the homeowner is away, is difficult. Conventional home security systems require constant monitoring, which is time-consuming and may not provide timely notifications. While recorded data and images are stored in cloud storage, there is no mechanism in place for automatically posting this data as an alert via social networking services (SNS). Therefore, to improve home security, a system that automatically detects suspicious sounds and movements and notifies users in real time is needed.

[1956] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1957] In this invention, the server includes means for recording the voice of a specific person, means for assigning a timestamp to the voice data, means for storing the voice data in cloud storage, means for automatically posting the time-stamped voice data to a social networking service platform, means for detecting suspicious voices, and means for notifying the social networking service of the detected suspicious voices as an alert, thereby enabling the real-time detection of suspicious voices occurring while the homeowner is away and a prompt response to the detected suspicious voices.

[1958] Furthermore, by including a means for detecting an object held by a specific person, a means for capturing an image of the detected object, a means for adding a timestamp to the image data, a means for storing the image data in cloud storage, a means for automatically posting the time-stamped image data to a social networking service platform, a means for detecting suspicious movements, and a means for notifying the social networking service of the detected suspicious movements as an alert, it is possible to instantly record suspicious movements within the home and automatically send an alert, thereby significantly improving home security.

[1959] A "means for recording the voice of a specific person" is equipment or software for recording the voice spoken by a specific individual as digital data.

[1960] The "means for adding a timestamp to audio data" refers to a device or software for adding a timestamp indicating the date and time of recording to recorded audio data.

[1961] "Means for storing voice data in cloud storage" refers to equipment or software for storing recorded voice data in a remote data storage service on the Internet.

[1962] "Means for automatically posting time-stamped audio data to a social networking service platform" refers to equipment or software for automatically posting time-stamped audio data to a network service such as a social media site.

[1963] "Means for detecting suspicious sounds" refers to equipment or software for detecting abnormal sounds or sounds that indicate a specific danger from recorded audio data.

[1964] "Means for notifying social networking services of detected suspicious audio as an alert" refers to equipment or software for sending information about detected suspicious audio as an alert to services on the network, such as social media sites.

[1965] A "means for detecting an object held by a particular person" is a device or software for recognizing and identifying an item held by a particular individual.

[1966] The "means for taking an image of a detected object" refers to a device or software for taking an image of a detected object using a device such as a camera.

[1967] The "means for adding a timestamp to image data" refers to a device or software for adding a timestamp indicating the date and time of photography to photographed image data.

[1968] "Means for storing image data in cloud storage" refers to equipment or software for storing recorded image data in a remote data storage service on the Internet.

[1969] "Means for automatically posting image data with a timestamp to a social networking service platform" refers to equipment or software for automatically posting image data with a timestamp to a service on a network, such as a social media site.

[1970] "Means for detecting suspicious activity" refers to equipment or software for detecting abnormal or fraudulent behavior from captured video data.

[1971] "Means for notifying social networking services of detected suspicious activity as an alert" refers to equipment or software for sending information about detected suspicious activity as an alert to services on the network, such as social media sites.

[1972] The present invention relates to a home security system that automatically detects suspicious sounds and movements and notifies a social networking service in real time. Specific embodiments of the system will be described in detail below.

[1973] 1. Voice Recording Function

[1974] Device (smartphone) operation

[1975] 1. The user launches the app

[1976] The user launches a voice recording application installed on the smartphone and enables the voice recording function.

[1977] 2. Device Settings

[1978] The device will run in the background and be configured to capture audio constantly using the microphone.

[1979] 3. Capture audio data

[1980] The device uses an AI voice recognition engine (e.g., Google Cloud Speech-to-Text) to analyze voice data in real time and detect suspicious voices.

[1981] 4. Adding a timestamp to audio data

[1982] Any suspicious audio detected is automatically given a timestamp based on the current date and time.

[1983] 5. Data transmission

[1984] The device sends the recorded audio data and a timestamp to the server.

[1985] Server Operation

[1986] 1. Receiving Data

[1987] The server receives the voice data sent from the device and stores it in cloud storage (e.g., AWS S3).

[1988] 2. Adding Metadata

[1989] The server adds metadata to the stored audio data, such as timestamps and a transcription of the audio content.

[1990] 3. Posting to social media

[1991] The server uses social media APIs (e.g., Twitter API) to automatically create and send a warning post containing the suspicious audio clip and metadata.

[1992] Example (voice recording function)

[1993] While the homeowner is away, the moment the sound of a window breaking is detected, the timestamp "2023-10-01 22:30:00" is added. The app automatically posts a message to Twitter saying "2023-10-01 22:30:00 The sound of a window breaking was detected!"

[1994] 2. Image Recording Function

[1995] Device (home camera) operation

[1996] 1. The user installs the camera

[1997] Users install home cameras and configure them to monitor specific areas.

[1998] 2. Real-time recognition

[1999] The device analyzes the live video stream and uses an AI image recognition engine (e.g., AWS Rekognition) to identify suspicious activity in real time.

[2000] 3. Motion Detection

[2001] When suspicious activity is detected, the camera automatically records the moment.

[2002] 4. Add a timestamp to the image

[2003] Recorded images are given a timestamp based on the current date and time.

[2004] 5. Data transmission

[2005] The terminal transmits the image data and a timestamp to the server.

[2006] Server Operation

[2007] 1. Receiving Data

[2008] The server receives the image data sent from the terminal and stores it in cloud storage.

[2009] 2. Adding Metadata

[2010] The server adds metadata to the stored image data, such as timestamps and suspicious movement detection results.

[2011] 3. Posting to social media

[2012] The server uses the SNS API to automatically create and send alerts containing images and metadata of suspicious activity.

[2013] Example (image recording function)

[2014] If an image of a stranger walking around the house while the homeowner is out is detected, it will be given the timestamp "2023-10-02 02:15:00." The app will then automatically post the message "2023-10-02 02:15:00 A stranger is walking around the house!" to Twitter.

[2015] Example prompts for generative AI models

[2016] Audio capture: The sound of a window breaking has been detected. Recorded at 2023-10-01 22:30:00.

[2017] Image Capture: An unknown individual walking through the home. Recorded at 2023-10-02 02:15:00.

[2018] Twitter Security Warning

[2019] According to the above-described embodiment, the present invention can significantly improve security within the home.

[2020] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[2021] Step 1:

[2022] The user launches the app. The user launches the audio recording app installed on their smartphone and enables the audio recording function. This puts the microphone in a state where it is always ready to capture audio. The input is the user's operation, and the output is the state where the recording function is enabled.

[2023] Step 2:

[2024] Configure the device so that it runs in the background and constantly captures audio using the microphone. Input is the system setting, and output is the microphone constantly capturing audio.

[2025] Step 3:

[2026] Voice data is captured. The device uses an AI voice recognition engine (e.g., Google Cloud Speech-to-Text) to analyze the voice data in real time and detect suspicious voices. The input is real-time voice data from the microphone, and the output is analyzed voice text data. Specifically, the voice data is sent to the AI ​​voice recognition engine and converted into text.

[2027] Step 4:

[2028] A timestamp is added to the audio data. A timestamp based on the current date and time is automatically added to detected suspicious audio. The input is analyzed audio text data, and the output is audio text data with a timestamp. Specifically, the current date and time is obtained and added to the audio data.

[2029] Step 5:

[2030] Data is transmitted. The terminal transmits the recorded voice data and a timestamp to the server. The input is voice text data with a timestamp, and the output is data transmission to the server. Specifically, the data is transmitted to the server using an HTTP request.

[2031] Step 6:

[2032] The server receives the data. The server receives the voice data sent from the device and stores it in cloud storage (e.g., AWS S3). The input is time-stamped voice text data, and the output is data storage in cloud storage. Specifically, the received data is stored using the cloud storage API.

[2033] Step 7:

[2034] Add metadata. The server adds metadata such as timestamps and text conversion of the audio content to the stored audio data. The input is audio data stored in cloud storage, and the output is audio data with metadata. Specific operations include checking the timestamp and adding the text conversion results.

[2035] Step 8:

[2036] Post to social media. The server automatically creates and sends a warning post containing the suspicious audio clip and metadata using a social media API (e.g., Twitter API). The input is audio data with metadata, and the output is a warning post to the social media platform. The specific operation is to send a message and audio clip using the social media API.

[2037] Step 9:

[2038] The user installs a camera. The user installs a home camera and configures it to monitor a specific area. The input is the user's operation, and the output is the camera installation completion and monitoring configuration.

[2039] Step 10:

[2040] Real-time recognition. The device analyzes the live video stream and uses an AI image recognition engine (e.g., AWS Rekognition) to identify suspicious activity in real time. The input is the live video stream from the camera, and the output is analyzed motion data. Specifically, the video stream is sent to the AI ​​image recognition engine for analysis.

[2041] Step 11:

[2042] It detects motion. When a suspicious movement is detected, the camera automatically records the moment. The input is the analyzed movement data, and the output is the recorded image data. Specifically, a frame is captured at the moment an abnormality is detected.

[2043] Step 12:

[2044] A timestamp is added to image data. A timestamp based on the current date and time is added to recorded images. The input is recorded image data, and the output is image data with a timestamp. Specifically, the current date and time is obtained and added to the image data.

[2045] Step 13:

[2046] Data is sent. The terminal sends image data and a timestamp to the server. The input is image data with a timestamp, and the output is data sent to the server. Specifically, the data is sent to the server using an HTTP request.

[2047] Step 14:

[2048] The server receives the data. The server receives the image data sent from the device and stores it in cloud storage. The input is image data with a timestamp, and the output is data storage in cloud storage. Specifically, the received data is stored using the cloud storage API.

[2049] Step 15:

[2050] Add metadata. The server adds metadata such as a timestamp and suspicious movement recognition results to the stored image data. The input is image data stored in cloud storage, and the output is image data with metadata. Specific operations include checking the timestamp and adding movement recognition results.

[2051] Step 16:

[2052] Post to SNS. The server uses SNS API to automatically create and send an alert containing images and metadata of suspicious activity. The input is image data with metadata, and the output is a warning post to the SNS platform. The specific operation is to send a message and image data using the SNS API.

[2053] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[2054] The present invention aims to combine an emotion engine with a system that automatically records the voices and images of specific people, timestamps the data, and automatically shares it through a social networking platform, thereby recognizing the user's emotions based on the recorded data and providing information corresponding to those emotions.

[2055] 1. Voice Recording Function

[2056] 1.1 Device (smartphone) operation

[2057] 1. The user launches the app

[2058] The user launches a voice recording application installed on the smartphone and enables the voice recording function.

[2059] 2. Device Settings

[2060] The device will run in the background and be configured to capture audio constantly using the microphone.

[2061] 3. Capture audio data

[2062] It uses AI models to analyze voice data in real time and detect new utterances.

[2063] 4. Emotion Recognition by Emotion Engine

[2064] The device sends the captured voice data to an emotion engine to recognize the user's emotions, which analyzes the tone, strength, and speed of the voice to identify emotions.

[2065] 5. Adding a timestamp to audio data

[2066] Any new audio clips detected will be given a timestamp based on the current date and time.

[2067] 6. Data transmission

[2068] The device sends the recorded voice data, a timestamp, and information on the recognized emotion to the server.

[2069] 1.2 Server Operation

[2070] 1. Receiving Data

[2071] The server receives the voice data and emotion information sent from the terminal and checks the consistency of the data.

[2072] 2. Adding Metadata

[2073] The server adds metadata to the received audio data, such as timestamps, text conversion of the audio content, and emotional information.

[2074] 3. Posting to social media

[2075] The server creates and automatically sends a post to the configured social networking platform (e.g., LINE), which includes the audio clip, a timestamp, the text of the audio, and emotional information.

[2076] 4. Emotion-based replies and comments generation

[2077] The server generates appropriate replies and comments based on the recognized emotions and automatically posts them to social media platforms.

[2078] 2. Image Recording Function

[2079] 2.1 Device (home camera) operation

[2080] 1. The user installs the camera

[2081] A user installs a home camera and configures it to recognize specific people.

[2082] 2. Real-time recognition

[2083] The device analyzes the video stream and uses AI models to identify specific people (children) in real time.

[2084] 3. Object Detection

[2085] When the camera recognizes that a specific person is holding a new object, it automatically takes a picture of the object.

[2086] 4. Add a timestamp to the image

[2087] Each captured image is given a timestamp based on the current date and time.

[2088] 5. Data transmission

[2089] The terminal transmits the image data and a timestamp to the server.

[2090] 2.2 Server Operation

[2091] 1. Receiving Data

[2092] The server receives the image data sent from the terminal and checks the integrity of the data.

[2093] 2. Adding Metadata

[2094] The server adds metadata such as a timestamp and object recognition results to the received image data.

[2095] 3. Posting to social media

[2096] The server creates and automatically sends a post to the configured social media platform (e.g., Instagram), which includes the image, a timestamp, and a description of the object.

[2097] 4. Emotion-based replies and comments generation

[2098] The server generates appropriate replies and comments based on the recognized emotions and automatically posts them to social media platforms.

[2099] Specific examples

[2100] Examples of voice recording functions

[2101] 1. The user launches the app

[2102] The user launches a voice recording app on their smartphone and enables the recording function.

[2103] 2. Capture audio data

[2104] The moment a child says "Daddy" for the first time, the device detects the new utterance and uses its emotion engine to recognize that the utterance contains the emotion of "joy."

[2105] 3. Adding a timestamp

[2106] The audio clip will be timestamped with "2023-10-02 13:20:00".

[2107] 4. Data transmission and storage

[2108] The terminal transmits the voice data, timestamp, and emotion information to the server, which stores them in cloud storage.

[2109] 5. Social media posts and emotional comment generation

[2110] The server automatically posts an audio clip to LINE with the message "2023-10-02 13:20:00 He said 'Daddy' (joy)," and also posts the comment "What a wonderful first word!"

[2111] Example of image recording function

[2112] 1. The user installs the camera

[2113] The user installs the home camera and configures the recording settings.

[2114] 2. Object detection and photography

[2115] The camera detects when a child is holding a new toy (a blue ball) and automatically captures the moment.

[2116] 3. Emotion Recognition by Emotion Engine

[2117] The emotion engine analyzes the audio and video data obtained from the captured images and recognizes that the child is "excited."

[2118] 4. Adding a timestamp

[2119] The image will be given a timestamp of "2023-10-02 14:10:00".

[2120] 5. Data transmission and storage

[2121] The terminal transmits the image data, timestamp, and emotion information to the server, which stores the data in cloud storage.

[2122] 6. Social media posts and emotional comment generation

[2123] The server automatically posts the image to Instagram with the message "2023-10-02 14:10:00 New toy: Blue ball (excited)" and also posts the comment "You're so excited about your new toy!"

[2124] With the above-described configuration, the present invention not only allows users to effortlessly record moments of their child's growth and easily share them with family and relatives, but also allows users to provide information according to their emotions.

[2125] The processing flow will be explained below.

[2126] Voice recording function processing steps

[2127] Step 1:

[2128] The user launches a voice recording application installed on the smartphone and enables the voice recording function.

[2129] Step 2:

[2130] Set your device to run in the background and constantly capture audio using the microphone.

[2131] Step 3:

[2132] The device analyzes the captured voice data in real time to detect new utterances, using an AI model to analyze the voice data and identify new utterances.

[2133] Step 4:

[2134] When a new utterance is detected, the device generates an audio clip and gives it a timestamp based on the current date and time.

[2135] Step 5:

[2136] The device sends the audio clip to an emotion engine that analyzes the user's emotions, which determine emotions based on factors such as tone, volume, and vocal inflection.

[2137] Step 6:

[2138] The emotion engine recognizes emotions (e.g., joy, sadness, excitement, etc.) contained in an audio clip and attaches the emotion data to the audio clip.

[2139] Step 7:

[2140] The device transmits the audio clip, timestamp, and emotion data to the server using a common network communication protocol.

[2141] Step 8:

[2142] The server receives the voice data sent from the terminal and checks the integrity of the data.

[2143] Step 9:

[2144] The server stores the received audio data in cloud storage, along with metadata such as timestamps and emotional data.

[2145] Step 10:

[2146] The server creates a post on a social networking platform (e.g., LINE) based on the stored audio data, which includes the audio clip, a timestamp, a transcription of the audio content, and emotion data.

[2147] Step 11:

[2148] The server sends the created post to the social media platform, where it is automatically posted to the user's configured account.

[2149] Step 12:

[2150] The server generates an appropriate reply or comment based on the recognized emotion and automatically posts it to the social media platform. For example, if "joy" is recognized, a comment such as "That's a great first word!" will be generated.

[2151] Image recording function processing steps

[2152] Step 1:

[2153] A user installs a home camera and configures it to recognize specific people.

[2154] Step 2:

[2155] The device (home camera) analyzes the video stream and is configured to identify specific people (children) in real time using an AI model.

[2156] Step 3:

[2157] The device detects the object being held by a particular person using an object recognition algorithm.

[2158] Step 4:

[2159] When a new object is detected, the device automatically captures the moment and generates an image.

[2160] Step 5:

[2161] The device will add a timestamp based on the current date and time to the generated image.

[2162] Step 6:

[2163] The device sends image data to the emotion engine, which analyzes the user's emotions. The emotion engine identifies emotions based on the user's facial expressions, movements, voice, etc.

[2164] Step 7:

[2165] The emotion engine recognizes emotions (e.g., excitement, surprise, joy, etc.) contained in the image data and attaches the emotion data to the image data.

[2166] Step 8:

[2167] The device transmits the image data, timestamp, and emotion data to the server, using a common network communication protocol.

[2168] Step 9:

[2169] The server receives the image data sent from the terminal and checks the consistency of the data.

[2170] Step 10:

[2171] The server stores the received image data in cloud storage, along with metadata such as timestamps and emotion data.

[2172] Step 11:

[2173] The server creates a post for a social media platform (e.g., Instagram) based on the stored image data, which includes the image, a timestamp, and emotion data.

[2174] Step 12:

[2175] The server sends the created post to the social media platform, where it is automatically posted to the user's configured account.

[2176] Step 13:

[2177] The server generates an appropriate reply or comment based on the recognized emotion and automatically posts it to the social media platform. For example, if "excitement" is recognized, a comment such as "You're crazy about your new toy!" will be generated.

[2178] Through the above processing steps, audio and image recording combined with an emotion engine is automatically performed, and the data can be shared quickly and easily, while also providing information according to emotions.

[2179] Example 2

[2180] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[2181] Current systems that record audio and images and share them on social media platforms have difficulty in properly recognizing users' emotions and providing information. Furthermore, they lack the functionality to detect new utterances from recorded audio or automatically recognize objects in images, requiring users to put in a lot of effort.

[2182] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes means for recording the voice of a specific person, means for analyzing the voice data and recognizing the content of the speech, means for recognizing emotions from the voice data, means for assigning a timestamp to the voice data, means for storing the voice data and the recognized emotional information in cloud storage, means for automatically posting the time-stamped voice data including the voice data and the recognized emotional information to an SNS platform, means for detecting an object held by the specific person, means for capturing an image of the detected object, means for analyzing image data and extracting a description of the object, means for recognizing emotions from the image data, means for assigning a timestamp to the image data, means for storing the image data and the recognized emotional information in cloud storage, means for automatically posting the time-stamped image data including the image data, the recognized object description, and the emotional information to an SNS platform, and means for detecting new comments from the recorded voice. This makes it possible to efficiently record a user's voice and image data along with the recognized emotions and automatically share them on an SNS platform.

[2183] "Specific person" refers to an individual who has been designated by the system as a target for recording audio and image data.

[2184] "Audio recording means" refers to devices or software that use a microphone to capture environmental sounds and store them as digital data.

[2185] "Means for analyzing and recognizing what is being said" refers to natural language processing technology that converts voice data into text and understands its content.

[2186] "Means for recognizing emotions" refers to algorithms or engines for identifying emotions from audio or image data.

[2187] "Means for adding a time stamp" refers to a function that adds the date and time of collection to recorded data in a digital format.

[2188] "Cloud storage means" refers to a service that allows data to be uploaded and stored on a remote server via the Internet.

[2189] "Means for automatically posting to social media platforms" refers to a program that connects to a designated social media service and automatically shares data.

[2190] "Means for detecting objects" refers to technology for analyzing image data to recognize and identify specific objects.

[2191] "Means for taking images of detected objects" refers to devices or software that use a camera to capture images containing recognized objects and store the digital data.

[2192] "Means for analyzing image data and extracting object descriptions" refers to a technique for analyzing image data and generating text information about objects contained therein.

[2193] "Means for detecting new utterances from audio recordings" refers to algorithms for identifying and detecting newly uttered words or phrases from continuously recorded audio data.

[2194] The present invention aims to combine an emotion engine with a system that automatically records the voices and images of specific people, timestamps the data, and automatically shares it through a social networking platform, thereby recognizing the user's emotions based on the recorded data and providing information corresponding to those emotions.

[2195] Voice recording function embodiment

[2196] Device (smartphone) operation

[2197] 1. The user launches the app

[2198] A user launches a voice recording app installed on their smartphone and enables the recording function. The app uses the Google Speech-to-Text API to convert the voice data into text in real time.

[2199] 2. Device Settings

[2200] The device runs in the background and is set to capture audio using the microphone at all times. Users can enable the recording feature in a specific settings menu and allow it to run in the background.

[2201] 3. Capture audio data

[2202] When voice is input, the app on the device calls the Google Speech-to-Text API to convert it into text and checks the content.

[2203] 4. Emotion Recognition by Emotion Engine

[2204] The converted text and audio data is sent to IBM Watson Tone Analyzer, where emotions are analyzed based on tone, strength, speed, etc.

[2205] 5. Adding a timestamp to audio data

[2206] The detected audio clips are given a timestamp based on the current date and time obtained from the device's internal clock.

[2207] 6. Data transmission

[2208] The device uses HTTPS communication to send the recorded audio data, timestamps, and emotion information to the server.

[2209] Server Operation

[2210] 1. Receiving Data

[2211] The server receives the voice data and emotion information sent from the device via the HTTPS protocol, and then checks the data checksum to verify its integrity.

[2212] 2. Adding Metadata

[2213] Metadata such as text conversion results and emotional information is added to the received voice data and stored in a database.

[2214] 3. Posting to social media

[2215] The server automatically creates and sends a post to the configured social networking platform (e.g., LINE) containing the audio clip, a timestamp, the text of the audio content, and emotional information.

[2216] 4. Emotion-based replies and comments generation

[2217] Based on the recognized sentiment, appropriate replies and comments are generated and automatically posted to social media platforms.

[2218] Image recording function embodiment

[2219] Device (home camera) operation

[2220] 1. The user installs the camera

[2221] A user installs a home camera in a specific location and configures the camera to identify specific people (e.g., children) within a certain range.

[2222] 2. Real-time recognition

[2223] The camera uses OpenCV to analyze the video stream in real time and recognize specific people.

[2224] 3. Object Detection

[2225] Detects when a specific person is holding a new object, and captures the moment with the camera.

[2226] 4. Add a timestamp to the image

[2227] Captured images are automatically time-stamped based on the current date and time.

[2228] 5. Data transmission

[2229] The captured image data and timestamp are sent to the server via HTTPS.

[2230] Server Operation

[2231] 1. Receiving Data

[2232] The server receives the image data sent from the terminal and checks the integrity of the data.

[2233] 2. Adding Metadata

[2234] A timestamp and object recognition results are added as metadata to the received image data.

[2235] 3. Posting to social media

[2236] The server automatically creates and sends a post to the configured social media platform (e.g., Instagram) containing the image, a timestamp, and a description of the object.

[2237] 4. Emotion-based replies and comments generation

[2238] Based on the recognized sentiment, appropriate replies and comments are generated and automatically posted to social media platforms.

[2239] Specific examples

[2240] Examples of voice recording functions

[2241] 1. The user launches the app

[2242] The user launches a voice recording app on their smartphone and enables the recording function.

[2243] 2. Capture audio data

[2244] The moment a child says "Daddy" for the first time, the device detects the new utterance and uses its emotion engine to recognize that the utterance contains the emotion of "joy."

[2245] 3. Adding a timestamp

[2246] The audio clip will be timestamped with "2023-10-02 13:20:00".

[2247] 4. Data transmission and storage

[2248] The terminal transmits the voice data, timestamp, and emotion information to the server, which stores them in cloud storage.

[2249] 5. Social media posts and emotional comment generation

[2250] The server automatically posts an audio clip to LINE with the message "2023-10-02 13:20:00 He said 'Daddy' (joy)" and also posts the comment "What a wonderful first word!"

[2251] Example of image recording function

[2252] 1. The user installs the camera

[2253] The user sets up the home camera and configures the recording settings.

[2254] 2. Object detection and photography

[2255] The camera detects when a child is holding a new toy (a blue ball) and automatically captures the moment.

[2256] 3. Emotion Recognition by Emotion Engine

[2257] The emotion engine analyzes the audio and video data obtained from the captured images and recognizes that the child is "excited."

[2258] 4. Adding a timestamp

[2259] The image will be given a timestamp of "2023-10-02 14:10:00".

[2260] 5. Data transmission and storage

[2261] The terminal transmits the image data, timestamp, and emotion information to the server, which stores the data in cloud storage.

[2262] 6. Social media posts and emotional comment generation

[2263] The server automatically posts the image to Instagram with the message "2023-10-02 14:10:00 New toy: Blue ball (excited)" and also posts the comment "You're so excited about your new toy!"

[2264] According to the above-described embodiment, the present invention enables users to record and share audio and image data without any special effort, and provides information according to their emotions.

[2265] The flow of the identification process in the second embodiment will be described with reference to FIG.

[2266] Voice recording function processing steps

[2267] Device (smartphone) operation

[2268] Step 1:

[2269] The user launches the app

[2270] Input: The user operates the smartphone and taps the icon of the voice recording app.

[2271] Specific operation: The user finds the audio recording app in the list of apps on their smartphone and taps the icon.

[2272] Output: The audio recording app will launch and the initial screen for the recording function will be displayed.

[2273] Step 2:

[2274] Device settings

[2275] Input: The user taps the "Start Recording" button in the app's settings menu to enable the recording feature.

[2276] What it does: The user taps the "Start Recording" button on the app's main screen and approves background recording if prompted.

[2277] Output: The device will have access to the microphone and will be configured to capture audio continuously in the background.

[2278] Step 3:

[2279] Capture audio data

[2280] Input: Audio signals from the environment.

[2281] How it works: The device uses the Google Speech-to-Text API to convert voice data to text in real time, and this conversion occurs whenever new speech is detected.

[2282] Output: Text conversion results and analysis results of the audio data.

[2283] Step 4:

[2284] Emotion recognition by emotion engine

[2285] Input: Converted text data and audio data.

[2286] Specific operation: Text and audio data are sent to IBM Watson Tone Analyzer, which analyzes emotions based on tone, strength, speed, etc.

[2287] Output: Parsed emotion information (e.g., happiness, sadness, excitement, etc.).

[2288] Step 5:

[2289] Adding a timestamp to audio data

[2290] Input: Audio clip and analyzed emotion information.

[2291] Specific operation: Obtains date and time information from the device's internal clock and adds a timestamp as metadata to the audio clip.

[2292] Output: Timestamp audio data and emotion information.

[2293] Step 6:

[2294] Sending data

[2295] Input: Timestamp audio data, emotion information.

[2296] What it does: Sends recorded data to a server using the HTTPS protocol.

[2297] Output: Voice data and emotion information sent to the server.

[2298] Server Operation

[2299] Step 1:

[2300] Receiving data

[2301] Input: Voice data and emotion information sent from the device.

[2302] Specific operation: The server receives the data via the HTTPS protocol and checks the checksum to verify the integrity of the data.

[2303] Output: Received voice data, emotion information, and consistency check results.

[2304] Step 2:

[2305] Adding metadata

[2306] Input: Received voice data and emotion information.

[2307] Specific operation: Adds timestamps, text conversion results of audio content, emotional information, etc. to received data.

[2308] Output: Audio data with metadata.

[2309] Step 3:

[2310] Posting to social media

[2311] Input: Audio data with metadata.

[2312] Specific operation: Calls the API of a social media platform (e.g., LINE) and posts the audio clip, timestamp, audio text, and emotional information.

[2313] Output: Audio data posted on SNS.

[2314] Step 4:

[2315] Generate replies and comments based on emotions

[2316] Input: Parsed emotion information.

[2317] Specific behavior: Generate replies and comments based on emotional information and automatically post them to social media platforms.

[2318] Output: Replies and comments posted to social media.

[2319] Image recording function processing steps

[2320] Device (home camera) operation

[2321] Step 1:

[2322] The user installs the camera

[2323] Input: The user installs the home camera in a specific location and configures it.

[2324] Specific operation: The user fixes the camera in a designated position and sets up facial recognition for a specific person (e.g., a child) through the app.

[2325] Output: Camera installation completed and target person set.

[2326] Step 2:

[2327] Real-time recognition

[2328] Input: A video stream acquired in real time.

[2329] How it works: The camera uses OpenCV to analyze the video stream and identify specific people in real time.

[2330] Output: Identification result of a specific person.

[2331] Step 3:

[2332] Object detection

[2333] Input: Image data about an object held by a particular person.

[2334] What it does: It uses an object detection algorithm to recognize when a new object is being held by a person and captures that moment.

[2335] Output: Image data containing the new object.

[2336] Step 4:

[2337] Add a timestamp to the image

[2338] Input: Captured image data.

[2339] Specific operation: Obtains date and time information from the device's internal clock and adds a timestamp as metadata to the image data.

[2340] Output: Timestamped image data.

[2341] Step 5:

[2342] Sending data

[2343] Input: Timestamped image data.

[2344] Specific operation: Image data is sent to the server using the HTTPS protocol.

[2345] Output: Image data sent to the server.

[2346] Server Operation

[2347] Step 1:

[2348] Receiving data

[2349] Input: Image data sent from the device.

[2350] Specific operation: The server receives the data via the HTTPS protocol and checks the checksum to verify the integrity of the data.

[2351] Output: Received image data, consistency check results.

[2352] Step 2:

[2353] Adding metadata

[2354] Input: Received image data.

[2355] Specific operation: Adds metadata such as timestamps and object recognition results to the received data.

[2356] Output: Image data with metadata.

[2357] Step 3:

[2358] Posting to social media

[2359] Input: Image data with metadata.

[2360] Specific operation: Calls the API of a social media platform (e.g., Instagram) and posts an image, timestamp, and object description.

[2361] Output: Image data posted on SNS.

[2362] Step 4:

[2363] Generate replies and comments based on emotions

[2364] Input: Parsed emotion information.

[2365] Specific behavior: Generate replies and comments based on emotional information and automatically post them to social media platforms.

[2366] Output: Replies and comments posted to social media.

[2367] The above are the specific processing steps of the present invention, which allow users to record and share audio and image data without any special effort, and provide information according to their emotions.

[2368] (Application example 2)

[2369] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[2370] Modern life demands that necessary information and services be provided based on a user's emotional state. However, existing technologies lack systems that can effectively analyze a user's emotions and provide appropriate responses based on those analysis results. In particular, it is difficult to automatically collect and analyze audio and image data to provide information tailored to the user's emotions. Furthermore, there is no system in place to reliably store this data in cloud storage and automatically post it to social media platforms. This makes it difficult to realize the personalized experiences users desire.

[2371] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[2372] In this invention, the server includes means for recording the voice of a specific person, means for assigning a timestamp to the voice data, means for storing the voice data in cloud storage, means for automatically posting the time-stamped voice data to a social networking service platform, means for analyzing the voice data using an emotion analysis engine and recognizing the user's emotion, and means for providing the user with appropriate information based on the recognized emotion. This makes it possible to analyze the user's emotion in real time and provide information and services according to the emotion.

[2373] The term "specific person" refers to a specific individual who is a predetermined target.

[2374] "Audio recording means" means a means for storing the speech of a particular person in digital form using a device.

[2375] "Audio data" refers to data that has been converted into digital form and recorded.

[2376] The "means for adding a time stamp" is a means for adding the date and time when the data was recorded to the audio data or image data.

[2377] "Cloud storage" is a set of remote servers that store data over the internet.

[2378] A "social networking service platform" is an online service that enables users to share information and communicate with other users via the Internet.

[2379] "Automatic posting means" refers to a means for publishing specific data on a given platform without user intervention.

[2380] An "emotion analysis engine" is a set of algorithms and software that analyzes audio and image data and identifies the user's emotions from them.

[2381] The "means for providing appropriate information" is a means for providing information and services that are considered to be optimal for the user based on the recognized emotions.

[2382] This invention is a system that automatically records the voice and images of a specific person, adds a timestamp to the data, and automatically shares it through a social networking platform. Furthermore, by combining it with an emotion analysis engine, it can recognize the user's emotions from the recorded data and provide information according to those emotions.

[2383] A system for implementing the present invention uses the following configuration and associated hardware and software.

[2384] System configuration

[2385] 1. Device Operation

[2386] 1. Audio recording and emotion analysis

[2387] The device is equipped with a microphone for recording voice and an AI model (e.g., EmotionRecognizer) for analyzing the voice data. When a user launches the app and engages in normal conversation, the device captures the voice. The voice is analyzed in real time through an emotion analysis engine to recognize the user's emotions. This emotion analysis uses the tone, strength, and speed of the voice.

[2388] 2. Image recording and emotion analysis

[2389] If the device is equipped with a camera, it will detect an object held by a specific person and take a picture of it. The captured image data will also be passed through an emotion analysis engine to recognize the user's emotions. This analysis uses facial expressions and the situation contained in the image.

[2390] 3. Adding a timestamp and sending data

[2391] The recorded audio and video data is time-stamped based on the current date and time, stored in cloud storage, and then automatically sent to social media platforms.

[2392] 2. Server Operation

[2393] 1. Data Receipt and Storage

[2394] The server receives the voice and image data sent from the device, checks the integrity of the data, and also receives emotional information. It adds additional metadata (e.g., converted voice content and object recognition results) to the received data and stores it in cloud storage.

[2395] 2. Posting and providing information on social media

[2396] The server then posts to the configured social networking platform in the specified format, including audio clips, images, timestamps, analyzed text, and emotion information. Appropriate replies and comments are also automatically generated and posted based on the recognized emotion.

[2397] Hardware and software used

[2398] Hardware: Smartphone (microphone, speaker, camera), cloud server, storage

[2399] Software: EmotionRecognizer (AI model for emotion recognition), SNS API (for posting to SNS)

[2400] Specific examples

[2401] Example: A user opens a smartphone app and orders dinner while uttering emotional speech. For example, the user says, "I'm tired today." The app recognizes this speech as "sad" and suggests "comfort ice cream" as a meal. This information is then posted to social media with a timestamp, sharing the following: "2023-10-05 18:30:00 - Emotion: Sad, Suggested Meal: Comfort Ice Cream."

[2402] Example prompt sentence:

[2403] The app allows users to record their voices expressing their emotions, recognize their emotions based on the voice data, and recommend appropriate meals. After that, users can share this information on social media.

[2404] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[2405] Step 1:

[2406] The device records the audio.

[2407] When a user launches the app and speaks, the device's microphone captures the audio. The input is the user's speech, and the output is digital audio data that can be used in subsequent processes.

[2408] Step 2:

[2409] The device tags the audio data with a timestamp.

[2410] This system adds a timestamp to recorded audio data by adding the current date and time information to the audio data. The input is digital audio data, and the output is audio data with a timestamp.

[2411] Step 3:

[2412] The device sends the voice data to the emotion analysis engine.

[2413] The device sends the recorded voice data to an emotion analysis engine, which analyzes the voice. The input is time-stamped voice data, and the output is analyzed emotion information. For example, the tone, strength, and speed of the voice are used to identify emotions.

[2414] Step 4:

[2415] The device saves the data to cloud storage.

[2416] The analyzed emotion information and the time-stamped voice data are sent to and stored in cloud storage. The input is the time-stamped voice data and the analyzed emotion information, and the output is the data stored in cloud storage.

[2417] Step 5:

[2418] The server receives the time-stamped audio data and emotion information.

[2419] The server receives the data sent to the cloud storage and checks the integrity of the data. The input is the data from the cloud storage, and the output is the data whose integrity has been checked.

[2420] Step 6:

[2421] The server adds metadata to the received data.

[2422] This converts audio data into text and adds additional metadata (e.g., a textual representation of the audio content). The input is time-stamped audio data with integrity check and analyzed emotion information, and the output is audio data with metadata.

[2423] Step 7:

[2424] The server automatically posts to the social networking service platform.

[2425] This system automatically posts time-stamped audio data, metadata, and emotional information to a social media platform. The input is audio data with metadata and analyzed emotional information, and the output is a post to a social media platform.

[2426] Step 8:

[2427] The server generates a comment based on the emotion and posts it to social media.

[2428] The system posts automatically generated comments based on the recognized emotions to a social media platform. The input is the analyzed emotion information, and the output is a social media post with the emotion-based comment.

[2429] Specifically, when a user says, "I'm tired today," the voice is recorded and a timestamp is added. The emotion analysis engine recognizes the voice as "sad" and saves it in cloud storage. The server then receives the data and posts the following to social media: "2023-10-05 18:30:00 - Emotion: Sad, Suggested Meal: Comforting Ice Cream." The server also simultaneously posts the comment, "Thank you for your hard work! Refresh yourself with some ice cream!"

[2430] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[2431] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[2432] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[2433] [Fourth embodiment]

[2434] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[2435] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[2436] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[2437] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[2438] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[2439] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[2440] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[2441] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[2442] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[2443] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[2444] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[2445] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[2446] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[2447] The present invention provides a system for automatically recording audio and images of a specific person, adding a timestamp, and automatically sharing the recorded audio and images through a social networking platform. The following describes in detail an embodiment of the present invention.

[2448] 1. Voice Recording Function

[2449] 1.1 Device (smartphone) operation

[2450] 1. The user launches the app

[2451] The user launches a voice recording application installed on the smartphone and enables the voice recording function.

[2452] 2. Device Settings

[2453] The device will run in the background and be configured to capture audio constantly using the microphone.

[2454] 3. Capture audio data

[2455] The device uses AI models to analyze voice data in real time and detect new utterances.

[2456] 4. Adding a timestamp to audio data

[2457] Any new audio clips detected will automatically be given a timestamp based on the current date and time.

[2458] 5. Data transmission

[2459] The device sends the recorded audio data and a timestamp to the server.

[2460] 1.2 Server Operation

[2461] 1. Receiving Data

[2462] The server receives the voice data sent from the device and stores it in cloud storage.

[2463] 2. Adding Metadata

[2464] The server adds metadata to the stored audio data, such as timestamps and a transcription of the audio content.

[2465] 3. Posting to social media

[2466] The server automatically creates and sends a post containing the audio clip and metadata to the configured social media platform (e.g., LINE).

[2467] 2. Image Recording Function

[2468] 2.1 Device (home camera) operation

[2469] 1. The user installs the camera

[2470] A user installs a home camera and configures it to recognize specific people.

[2471] 2. Real-time recognition

[2472] The device analyzes the video stream and uses AI models to identify specific people (children) in real time.

[2473] 3. Object Detection

[2474] When the camera recognizes that a specific person is holding a new object, it automatically takes a picture of the object.

[2475] 4. Add a timestamp to the image

[2476] Each captured image is given a timestamp based on the current date and time.

[2477] 5. Data transmission

[2478] The terminal transmits the image data and a timestamp to the server.

[2479] 2.2 Server Operation

[2480] 1. Receiving Data

[2481] The server receives the image data sent from the terminal and stores it in cloud storage.

[2482] 2. Adding Metadata

[2483] The server adds metadata such as timestamps and object recognition results to the stored image data.

[2484] 3. Posting to social media

[2485] The server automatically creates and sends a post containing the image and metadata to the configured social media platform (e.g., Instagram).

[2486] Specific examples

[2487] Examples of voice recording functions

[2488] 1. The user launches the app

[2489] The user launches a voice recording app on their smartphone and enables the recording function.

[2490] 2. Capture audio data

[2491] The moment your child says "mom" for the first time, the device detects the new utterance.

[2492] 3. Adding a timestamp

[2493] The audio clip will be timestamped with "2023-10-01 10:00:00".

[2494] 4. Data transmission and storage

[2495] The device sends the voice data to the server, which stores it in cloud storage.

[2496] 5. Social Media Posting

[2497] The server automatically posts the audio clip to LINE with the message "2023-10-01 10:00:00 First comment: Mom."

[2498] Example of image recording function

[2499] 1. The user installs the camera

[2500] The user installs the home camera and configures the recording settings.

[2501] 2. Object detection and photography

[2502] The camera detects when a child has a new toy (a red car) and automatically captures the moment.

[2503] 3. Adding a timestamp

[2504] The image will be given a timestamp of "2023-10-01 15:00:00".

[2505] 4. Data transmission and storage

[2506] The device sends the image data to the server, which stores it in cloud storage.

[2507] 5. Social Media Posting

[2508] The server automatically posts the image to Instagram with the message "2023-10-01 15:00:00 New Toy: Red Car."

[2509] As described above, the present invention allows users to effortlessly record moments of their children's growth and easily share them with family and relatives.

[2510] The processing flow will be explained below.

[2511] Voice recording function processing steps

[2512] Step 1:

[2513] The user launches a voice recording application installed on the smartphone and enables the voice recording function.

[2514] Step 2:

[2515] Set your device to run in the background and constantly capture audio using the microphone.

[2516] Step 3:

[2517] The device analyzes the captured voice data in real time to detect new utterances, using an AI model to analyze the voice data and identify new utterances.

[2518] Step 4:

[2519] When a new utterance is detected, the device generates an audio clip and gives it a timestamp based on the current date and time.

[2520] Step 5:

[2521] The device transmits the audio clip and timestamp to the server, using a common network communication protocol.

[2522] Step 6:

[2523] The server receives the voice data sent from the terminal and checks the integrity of the data.

[2524] Step 7:

[2525] The server stores the received audio data in cloud storage, along with metadata such as timestamps.

[2526] Step 8:

[2527] The server creates a post on a social networking platform (e.g., LINE) based on the stored audio data. This post includes the audio clip, a timestamp, and a brief text description of the audio.

[2528] Step 9:

[2529] The server sends the created post to the social media platform, where it is automatically posted to the user's configured account.

[2530] Image recording function processing steps

[2531] Step 1:

[2532] A user installs a home camera and configures it to recognize specific people.

[2533] Step 2:

[2534] The device (home camera) analyzes the video stream and is configured to identify specific people (children) in real time using an AI model.

[2535] Step 3:

[2536] The device detects the object being held by a particular person using an object recognition algorithm.

[2537] Step 4:

[2538] When a new object is detected, the device automatically captures the moment and generates an image.

[2539] Step 5:

[2540] The device will add a timestamp based on the current date and time to the generated image.

[2541] Step 6:

[2542] The device transmits the image data and a timestamp to the server, using a common network communication protocol.

[2543] Step 7:

[2544] The server receives the image data sent from the terminal and checks the consistency of the data.

[2545] Step 8:

[2546] The server stores the received image data in cloud storage, along with metadata such as timestamps and object recognition results.

[2547] Step 9:

[2548] The server creates a post on a social media platform (e.g., Instagram) based on the stored image data, which includes the image, a timestamp, and a brief description of the object.

[2549] Step 10:

[2550] The server sends the created post to the social media platform, where it is automatically posted to the user's configured account.

[2551] Through the above processing steps, audio and image recordings are automatically performed, and the data can be shared quickly and easily.

[2552] Example 1

[2553] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[2554] Conventional audio and image recording systems are inadequate in efficiently recording important moments of specific people and automatically sharing them via social media due to the time and effort required. Furthermore, technology to improve the accuracy of audio and image analysis using AI models is underdeveloped. Therefore, there is a need for a method that allows users to record and share information quickly, accurately, and without the hassle of manual effort.

[2555] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[2556] In this invention, the server includes means for recording the voice of a specific person, means for timestamp-adding the voice data, means for storing the voice data in cloud storage, means for automatically posting the time-stamped voice data to a social networking platform, means for utilizing a generative AI model for analyzing the voice, and means for performing named entity recognition, thereby enabling users to effortlessly and accurately record and quickly share important moments of a specific person.

[2557] An "audio recording device" is a device whose purpose is to capture the voice of a specific person and store it as digital data.

[2558] "Means for adding time stamps" refers to a means for adding information about the date and time of recording or shooting to recorded audio data or photographed image data.

[2559] "Means for saving audio data in cloud storage" refers to means for storing captured audio data in a data storage service on the Internet.

[2560] "Means for automatically posting audio data with a timestamp to a social networking platform" means a means for automatically publishing and sharing audio data with a timestamp on a pre-designated social networking service.

[2561] "Means using generative AI models to analyze speech" refers to means that utilize generated artificial intelligence models to analyze recorded speech data and identify important utterances and speech features.

[2562] "Means for performing named entity recognition" refers to means for extracting and identifying specific names or proper nouns from analyzed speech data.

[2563] The "means for detecting an object held by a specific person" is a means for automatically recognizing an object held by a specific person using a camera.

[2564] The "means for taking an image of a detected object" refers to a means for capturing an image of a recognized object and storing it as digital data.

[2565] "Means for saving image data in cloud storage" refers to means for storing captured image data in a data storage service on the Internet.

[2566] "Means for automatically posting image data with a timestamp to a social networking platform" refers to a means for automatically publishing and sharing image data with a timestamp attached to a pre-designated social networking service.

[2567] "Means utilizing a generative AI model for image recognition" means means utilizing a generative artificial intelligence model to analyze captured image data and identify objects within the image.

[2568] "Means for identifying object characteristics" refers to means for extracting and identifying characteristics such as size, shape, and color of a specific object from captured image data.

[2569] The "means for detecting new utterances" is a means for automatically identifying and extracting new utterances that are different from conventional ones from recorded audio data.

[2570] The present invention is a system that automatically records the voice and images of a specific person, adds a timestamp, and automatically shares them through a social networking platform. The detailed steps for implementing this system are described below.

[2571] Voice recording function embodiment

[2572] Device (smartphone) operation

[2573] 1. The user launches the app

[2574] The user launches the audio recording app installed on their smartphone and taps the "Start Recording" button to enable the recording function, which puts the device into audio recording mode.

[2575] 2. Configure your device

[2576] The device gets the necessary permissions from the user to operate in the background, including permission to use the microphone and to operate in the background, so the device is always ready to capture audio.

[2577] 3. Capture audio data

[2578] The device uses a microphone to capture voice data in real time, and uses the Google Speech-to-Text API to analyze the captured voice data and detect new utterances with an AI model.

[2579] 4. Adding a timestamp to audio data

[2580] When a new utterance is detected, the audio clip is timestamped with the current date and time information. For example, if the word "Mom" was uttered on 2023-10-01 10:00:00, the audio clip will be timestamped with this time.

[2581] 5. Send the data

[2582] The audio data and timestamp are sent to the cloud server using the HTTPS protocol.

[2583] Server Operation

[2584] 1. Receive data

[2585] The server receives the audio data and timestamps sent from the device, and stores this data in a database such as AWS RDS.

[2586] 2. Add metadata

[2587] The server converts the voice data into text and obtains the text data. It also adds timestamps and named entity recognition metadata. For example, the converted text is saved as "2023-10-01 10:00:00 Mom."

[2588] 3. Post to social media

[2589] The server uses the LINE Messaging API to automatically create and send a post containing the audio clip and metadata to the social media platform.

[2590] Image recording function embodiment

[2591] Device (home camera) operation

[2592] 1. The user installs the camera

[2593] The user simply installs the device in a suitable location and configures the camera via a dedicated smartphone app. Once the settings, including network connection, are complete, the camera goes into standby mode.

[2594] 2. Real-time recognition

[2595] The camera captures the video stream in real time and uses the Google Vision API to identify specific people, which is then used to determine whether a specific person is present in the frame.

[2596] 3. Detecting objects

[2597] The camera detects the object a specific person is holding using the YOLO model, etc. For example, if a child is holding a new toy (a red car), the camera will capture that moment.

[2598] 4. Add a timestamp to your images

[2599] A timestamp based on the current date and time is added to the captured image. For example, a timestamp such as "2023-10-01 15:00:00" is added to the image.

[2600] 5. Send the data

[2601] The captured image data and timestamp are sent to a cloud server using the HTTPS protocol.

[2602] Server Operation

[2603] 1. Receive data

[2604] The server receives the image data and timestamp sent from the device, and stores the data in cloud storage such as Google Cloud Storage.

[2605] 2. Add metadata

[2606] The server performs object recognition processing on the image data and adds the recognition results and a timestamp as metadata. For example, it adds metadata such as "2023-10-01 15:00:00 New toy: Red car."

[2607] 3. Post to social media

[2608] The server automatically creates and sends a post containing image data and metadata to the specified social media platform using the Instagram Graph API.

[2609] Specific examples

[2610] Examples of voice recording functions

[2611] 1. The user launches the app

[2612] The user launches a voice recording app on their smartphone and enables the recording function.

[2613] 2. Capture audio data

[2614] The moment your child says "mom" for the first time, the device detects the new utterance.

[2615] 3. Add a timestamp

[2616] The audio clip will be timestamped with "2023-10-01 10:00:00".

[2617] 4. Send and store data

[2618] The device sends the voice data to the server, which stores it in cloud storage.

[2619] 5. Post to social media

[2620] The server automatically posts the audio clip to LINE with the message "2023-10-01 10:00:00 First comment: Mom."

[2621] Example of image recording function

[2622] 1. The user installs the camera

[2623] The user installs a home camera and configures the recording settings.

[2624] 2. Detect and photograph objects

[2625] The camera detects when a child has a new toy (a red car) and automatically captures the moment.

[2626] 3. Add a timestamp

[2627] The image will be given a timestamp of "2023-10-01 15:00:00".

[2628] 4. Send and store data

[2629] The device sends the image data to the server, which stores it in cloud storage.

[2630] 5. Post to social media

[2631] The server automatically posts the image to Instagram with the message "2023-10-01 15:00:00 New Toy: Red Car."

[2632] This system allows users to effortlessly record important moments of specific people with high accuracy and quickly share them via social media.

[2633] The flow of the identification process in the first embodiment will be described with reference to FIG.

[2634] Voice recording function embodiment

[2635] Device (smartphone) operation

[2636] Step 1:

[2637] The user launches the audio recording app installed on their smartphone. The user taps the "Start Recording" button to enable recording.

[2638] Input: User taps, launched applications

[2639] Output: Recording start trigger

[2640] Specific behavior:

[2641] Tap the application icon.

[2642] The application will launch and a Start Recording button will appear.

[2643] The user taps the start recording button.

[2644] Step 2:

[2645] The device will ask the user for the necessary permissions to operate in the background, such as permission to use the microphone and permission to operate in the background.

[2646] Input: User permission operation

[2647] Output: Allow background operation and microphone usage

[2648] Specific behavior:

[2649] The device will display a popup asking permission to run in the background and use the microphone.

[2650] The user selects "Allow."

[2651] Step 3:

[2652] The device captures audio in real time through a microphone and analyzes it using a generative AI model (Google Speech-to-Text API) to detect new utterances.

[2653] Input: Real-time captured audio data

[2654] Output: Audio clip where new utterances were detected

[2655] Specific behavior:

[2656] The microphone continues to capture audio.

[2657] The audio data is sent to the Google Speech-to-Text API.

[2658] The API analyzes the audio data and detects new utterances.

[2659] Step 4:

[2660] When a new utterance is detected, the audio clip is given a timestamp based on the current date and time.

[2661] Input: Audio data in which new utterances are detected

[2662] Output: Time-stamped audio data

[2663] Specific behavior:

[2664] Get the current date and time information.

[2665] Add timestamps to audio clips.

[2666] Step 5:

[2667] The device sends the time-stamped audio data to the server using the HTTPS protocol.

[2668] Input: Timestamped audio data

[2669] Output: The audio data is sent to the server.

[2670] Specific behavior:

[2671] The data is sent to the server via HTTPS.

[2672] The server receives the data.

[2673] Server Operation

[2674] Step 6:

[2675] The server receives the audio data and timestamp sent from the device and stores it in cloud storage (AWS RDS).

[2676] Input: Audio data with timestamps

[2677] Output: Audio data stored in cloud storage

[2678] Specific behavior:

[2679] The server receives the data.

[2680] Store the audio data in AWS RDS.

[2681] Step 7:

[2682] The server performs text conversion processing on the voice data and adds a timestamp and named entity recognition data to the converted text data.

[2683] Input: Saved audio data

[2684] Output: Audio data with text conversion and metadata

[2685] Specific behavior:

[2686] The server sends the voice data to the generative AI model (voice analysis model).

[2687] Time stamps and named entities are added to the obtained text data.

[2688] Step 8:

[2689] The server automatically creates and sends a post containing the audio clip and the attached metadata to the SNS platform (LINE) using the SNS API.

[2690] Input: Audio data with metadata

[2691] Output: Audio data posted to SNS

[2692] Specific behavior:

[2693] Use the LINE Messaging API to create a message.

[2694] Post the generated message to LINE.

[2695] Image recording function embodiment

[2696] Device (home camera) operation

[2697] Step 1:

[2698] The user installs the home camera in an appropriate location and configures the camera via their smartphone.

[2699] Input: Camera settings operated by the user

[2700] Output: Camera starts operating

[2701] Specific behavior:

[2702] Connect the camera to a power source and position it appropriately.

[2703] Use the smartphone app to configure the camera settings (such as network connection settings).

[2704] Step 2:

[2705] The camera captures the video stream in real time and uses a generative AI model (Google Vision API) to identify specific people.

[2706] Input: Real-time captured video data

[2707] Output: Video data in which a specific person is identified

[2708] Specific behavior:

[2709] The camera continues to capture footage.

[2710] The video data is sent to the Google Vision API.

[2711] The API identifies a specific person.

[2712] Step 3:

[2713] Use an AI model (YOLO) to detect objects held by a specific person. Check if a person is holding a new object.

[2714] Input: Video data of identified person

[2715] Output: Data showing the object being held

[2716] Specific behavior:

[2717] Send the video data to the YOLO model.

[2718] The model detects objects and recognizes what a person is holding.

[2719] Step 4:

[2720] When a new object is detected, the camera automatically captures the moment and adds a timestamp to the image data.

[2721] Input: Video data of a person holding a new object

[2722] Output: Image data with timestamps

[2723] Specific behavior:

[2724] Get the current date and time information.

[2725] Add a timestamp to the image.

[2726] Step 5:

[2727] The device sends the image data and a timestamp to the server using the HTTPS protocol.

[2728] Input: Image data with timestamps

[2729] Output: Image data sent to the server

[2730] Specific behavior:

[2731] Send the data to the server over HTTPS.

[2732] The server receives the data.

[2733] Server Operation

[2734] Step 6:

[2735] The server receives the image data and timestamp sent from the device and stores them in Google Cloud Storage.

[2736] Input: Image data with timestamp

[2737] Output: Image data stored in cloud storage

[2738] Specific behavior:

[2739] The server receives the data.

[2740] Store image data in Google Cloud Storage.

[2741] Step 7:

[2742] The server performs object recognition processing on the image data and adds the recognition results and a timestamp as metadata.

[2743] Input: Saved image data

[2744] Output: Image data with metadata

[2745] Specific behavior:

[2746] The server sends the image data to the Google Vision API.

[2747] The object recognition result and a timestamp are added to the image data.

[2748] Step 8:

[2749] The server automatically creates and sends a post containing image data and attached metadata to a social networking platform (Instagram) using the social networking API.

[2750] Input: Image data with metadata

[2751] Output: Image data posted to SNS

[2752] Specific behavior:

[2753] Use the Instagram Graph API to generate messages.

[2754] Post the generated message to Instagram.

[2755] The above is the flow of the specific processing steps of this system. In this way, users can easily and accurately record important moments of specific people and quickly share them via social media.

[2756] (Application example 1)

[2757] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[2758] Improving safety and security within the home has become an important issue in modern society. Responding quickly to suspicious sounds or movements, especially when the homeowner is away, is difficult. Conventional home security systems require constant monitoring, which is time-consuming and may not provide timely notifications. While recorded data and images are stored in cloud storage, there is no mechanism in place for automatically posting this data as an alert via social networking services (SNS). Therefore, to improve home security, a system that automatically detects suspicious sounds and movements and notifies users in real time is needed.

[2759] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[2760] In this invention, the server includes means for recording the voice of a specific person, means for assigning a timestamp to the voice data, means for storing the voice data in cloud storage, means for automatically posting the time-stamped voice data to a social networking service platform, means for detecting suspicious voices, and means for notifying the social networking service of the detected suspicious voices as an alert, thereby enabling the real-time detection of suspicious voices occurring while the homeowner is away and a prompt response to the detected suspicious voices.

[2761] Furthermore, by including a means for detecting an object held by a specific person, a means for capturing an image of the detected object, a means for adding a timestamp to the image data, a means for storing the image data in cloud storage, a means for automatically posting the time-stamped image data to a social networking service platform, a means for detecting suspicious movements, and a means for notifying the social networking service of the detected suspicious movements as an alert, it is possible to instantly record suspicious movements within the home and automatically send an alert, thereby significantly improving home security.

[2762] A "means for recording the voice of a specific person" is equipment or software for recording the voice spoken by a specific individual as digital data.

[2763] The "means for adding a timestamp to audio data" refers to a device or software for adding a timestamp indicating the date and time of recording to recorded audio data.

[2764] "Means for storing voice data in cloud storage" refers to equipment or software for storing recorded voice data in a remote data storage service on the Internet.

[2765] "Means for automatically posting time-stamped audio data to a social networking service platform" refers to equipment or software for automatically posting time-stamped audio data to a network service such as a social media site.

[2766] "Means for detecting suspicious sounds" refers to equipment or software for detecting abnormal sounds or sounds that indicate a specific danger from recorded audio data.

[2767] "Means for notifying social networking services of detected suspicious audio as an alert" refers to equipment or software for sending information about detected suspicious audio as an alert to services on the network, such as social media sites.

[2768] A "means for detecting an object held by a particular person" is a device or software for recognizing and identifying an item held by a particular individual.

[2769] The "means for taking an image of a detected object" refers to a device or software for taking an image of a detected object using a device such as a camera.

[2770] The "means for adding a timestamp to image data" refers to a device or software for adding a timestamp indicating the date and time of photography to photographed image data.

[2771] "Means for storing image data in cloud storage" refers to equipment or software for storing recorded image data in a remote data storage service on the Internet.

[2772] "Means for automatically posting image data with a timestamp to a social networking service platform" refers to equipment or software for automatically posting image data with a timestamp to a service on a network, such as a social media site.

[2773] "Means for detecting suspicious activity" refers to equipment or software for detecting abnormal or fraudulent behavior from captured video data.

[2774] "Means for notifying social networking services of detected suspicious activity as an alert" refers to equipment or software for sending information about detected suspicious activity as an alert to services on the network, such as social media sites.

[2775] The present invention relates to a home security system that automatically detects suspicious sounds and movements and notifies a social networking service in real time. Specific embodiments of the system will be described in detail below.

[2776] 1. Voice Recording Function

[2777] Device (smartphone) operation

[2778] 1. The user launches the app

[2779] The user launches a voice recording application installed on the smartphone and enables the voice recording function.

[2780] 2. Device Settings

[2781] The device will run in the background and be configured to capture audio constantly using the microphone.

[2782] 3. Capture audio data

[2783] The device uses an AI voice recognition engine (e.g., Google Cloud Speech-to-Text) to analyze voice data in real time and detect suspicious voices.

[2784] 4. Adding a timestamp to audio data

[2785] Any suspicious audio detected is automatically given a timestamp based on the current date and time.

[2786] 5. Data transmission

[2787] The device sends the recorded audio data and a timestamp to the server.

[2788] Server Operation

[2789] 1. Receiving Data

[2790] The server receives the voice data sent from the device and stores it in cloud storage (e.g., AWS S3).

[2791] 2. Adding Metadata

[2792] The server adds metadata to the stored audio data, such as timestamps and a transcription of the audio content.

[2793] 3. Posting to social media

[2794] The server uses social media APIs (e.g., Twitter API) to automatically create and send a warning post containing the suspicious audio clip and metadata.

[2795] Example (voice recording function)

[2796] While the homeowner is away, the moment the sound of a window breaking is detected, the timestamp "2023-10-01 22:30:00" is added. The app automatically posts a message to Twitter saying "2023-10-01 22:30:00 The sound of a window breaking was detected!"

[2797] 2. Image Recording Function

[2798] Device (home camera) operation

[2799] 1. The user installs the camera

[2800] Users install home cameras and configure them to monitor specific areas.

[2801] 2. Real-time recognition

[2802] The device analyzes the live video stream and uses an AI image recognition engine (e.g., AWS Rekognition) to identify suspicious activity in real time.

[2803] 3. Motion Detection

[2804] When suspicious activity is detected, the camera automatically records the moment.

[2805] 4. Add a timestamp to the image

[2806] Recorded images are given a timestamp based on the current date and time.

[2807] 5. Data transmission

[2808] The terminal transmits the image data and a timestamp to the server.

[2809] Server Operation

[2810] 1. Receiving Data

[2811] The server receives the image data sent from the terminal and stores it in cloud storage.

[2812] 2. Adding Metadata

[2813] The server adds metadata to the stored image data, such as timestamps and suspicious movement detection results.

[2814] 3. Posting to social media

[2815] The server uses the SNS API to automatically create and send alerts containing images and metadata of suspicious activity.

[2816] Example (image recording function)

[2817] If an image of a stranger walking around the house while the homeowner is out is detected, it will be given the timestamp "2023-10-02 02:15:00." The app will then automatically post the message "2023-10-02 02:15:00 A stranger is walking around the house!" to Twitter.

[2818] Example prompts for generative AI models

[2819] Audio capture: The sound of a window breaking has been detected. Recorded at 2023-10-01 22:30:00.

[2820] Image Capture: An unkn...

Claims

1. a means for recording the voice of a particular person; means for adding a timestamp to the audio data; A means for storing the audio data in cloud storage; A means to automatically post time-stamped audio data to social media platforms; A system including:

2. a means for detecting an object held by a particular person; means for capturing an image of the detected object; means for adding a timestamp to image data; A means for storing image data in cloud storage; A means to automatically post time-stamped image data to a social media platform; A system including:

3. 10. The system of claim 1, further comprising means for detecting new utterances from the recorded audio.

Images