system

Abstract

We provide the system. [Solution] A proposal means that suggests creative activities in response to user requests based on information obtained from the user's terminal, A guide means that provides detailed information regarding the activities proposed by the proposed means, A recording method for digitizing and recording artwork data obtained as a result of creative activities, A sharing means for managing the artwork data recorded by the aforementioned recording means and providing it in a form that can be shared with others, A system that includes this.

Description

Technical Field

[0001] The technology of the present disclosure relates to a system.

Background Art

[0002] Patent Document 1 discloses a method for controlling a persona chatbot, which is performed by at least one processor, including steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to an explanation of a chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance as a response to the user utterance.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In the family, in order to cultivate the creativity of children from infants to middle school students, there is a lack of appropriate tools and project ideas, and there is also a major problem that the time and resources for parent-child creative activities in a busy family environment are limited. Furthermore, there is also a lack of means for commemorating and sharing the works created by children. Due to such problems, there is a situation where it is difficult to promote creative activities within the family and effectively support the creative growth of children.

Means for Solving the Problems

[0005] To address the above challenges, this system includes a suggestion mechanism that proposes creative activities tailored to the child's age and interests based on information obtained from the user's terminal. Furthermore, it includes a guide mechanism that provides detailed information about the suggested activities, supplying a list of necessary materials and procedures in advance, thereby creating an environment where creative activities can be easily carried out even in busy households. It also includes a recording mechanism for digitizing and recording the works resulting from creative activities, and a sharing mechanism for easily sharing the saved works with others, offering options to preserve memories in the form of photo books and calendars. Through this series of mechanisms, the system comprehensively supports an environment for creative activities at home.

[0006] A "user terminal" refers to a device used by a home or individual to send and receive information and operate applications.

[0007] "Information" refers to all data that is input, generated, or received by users or systems.

[0008] "Creative activities" refer to activities in which children can participate and develop their creativity and imagination through creation and expression.

[0009] "Suggestion methods" refer to functions and mechanisms for selecting and presenting the most suitable creative activities according to the user's situation.

[0010] "Guiding means" refers to a function or mechanism that provides guidance for carrying out a proposed creative activity by showing the detailed steps and necessary materials.

[0011] "Means of recording" refers to functions and mechanisms for saving the results of creative activities as digital data.

[0012] "Sharing means" refers to functions or mechanisms for sharing stored digital data with other users or devices.

[0013] "Cloud storage" refers to a form of storage device that stores data via the internet, making it accessible from remote locations.

[0014] A "photobook" refers to a collection of photographs in book form, created using digital images.

[0015] A "calendar" refers to a format of printed or electronic document that includes digital images along with date and schedule information. [Brief explanation of the drawing]

[0016] [Figure 1] This is a conceptual diagram showing an example of the configuration of a data processing system according to the first embodiment. [Figure 2] This is a conceptual diagram showing an example of the essential functions of a data processing device and a smart device according to the first embodiment. [Figure 3] This is a conceptual diagram showing an example of the configuration of a data processing system according to the second embodiment. [Figure 4] This is a conceptual diagram showing an example of the main functions of a data processing device and smart glasses according to the second embodiment. [Figure 5] This is a conceptual diagram showing an example of the configuration of a data processing system according to the third embodiment. [Figure 6] This is a conceptual diagram showing an example of the main functions of a data processing device and a headset-type terminal according to the third embodiment. [Figure 7] This is a conceptual diagram showing an example of the configuration of a data processing system according to the fourth embodiment. [Figure 8] This is a conceptual diagram showing an example of the main functions of a data processing device and a robot according to the fourth embodiment. [Figure 9] This shows an emotion map where multiple emotions are mapped. [Figure 10] This shows an emotion map where multiple emotions are mapped. [Figure 11] This is a sequence diagram showing the processing flow of the data processing system in Example 1. [Figure 12]It is a sequence diagram showing the processing flow of the data processing system in Application Example 1. [Figure 13] It is a sequence diagram showing the processing flow of the data processing system in Embodiment 2 when combined with an emotion engine. [Figure 14] It is a sequence diagram showing the processing flow of the data processing system in Application Example 2 when combined with an emotion engine.

Mode for Carrying Out the Invention

[0017] Hereinafter, an example of an embodiment of a system according to the technology of the present disclosure will be described with reference to the accompanying drawings.

[0018] First, the terms used in the following description will be explained.

[0019] In the following embodiments, a processor with a reference numeral (hereinafter simply referred to as "processor") may be a single arithmetic unit or a combination of multiple arithmetic units. Also, the processor may be a single type of arithmetic unit or a combination of multiple types of arithmetic units. Examples of arithmetic units include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), an APU (Accelerated Processing Unit), and the like.

[0020] In the following embodiments, a RAM (Random Access Memory) with a reference numeral is a memory in which information is temporarily stored and is used as a work memory by the processor.

[0021] In the following embodiments, the signed storage is one or more non-volatile storage devices that store various programs and various parameters. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), or magnetic tapes.

[0022] In the following embodiments, the signed communication interface (I / F) is an interface that includes a communication processor and an antenna, etc. The communication interface manages communication between multiple computers. Examples of communication standards applicable to the communication interface include wireless communication standards such as 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), or Bluetooth (registered trademark).

[0023] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." That is, "A and / or B" means that it may be A alone, or B alone, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" applies when expressing three or more things linked by "and / or."

[0024] [First Embodiment]

[0025] Figure 1 shows an example of the configuration of the data processing system 10 according to the first embodiment.

[0026] As shown in Figure 1, the data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0027] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0028] The smart device 14 comprises a computer 36, a reception device 38, an output device 40, a camera 42, and a communication interface 44. The computer 36 comprises a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The reception device 38, output device 40, and camera 42 are also connected to the bus 52.

[0029] The reception device 38 is equipped with a touch panel 38A and a microphone 38B, etc., and receives user input. The touch panel 38A receives user input by detecting contact with an object (e.g., a pen or finger). The microphone 38B receives user input by detecting the user's voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0030] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form perceptible to the user 20 (e.g., audio and / or text). The display 40A displays visible information such as text and images according to instructions from the processor 46. The speaker 40B outputs audio according to instructions from the processor 46. The camera 42 is a small digital camera equipped with an optical system such as a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0031] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various types of information between processor 46 and processor 28 via network 54.

[0032] Figure 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0033] As shown in Figure 2, in the data processing device 12, a specific processing is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" related to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0034] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0035] In the smart device 14, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The reception output program 60 is used in conjunction with a specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0036] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0037] The embodiments for carrying out the present invention are described below. This system connects a user terminal and a server via a network and can suggest, guide, record, and share creative activities in the home.

[0038] First, the user launches a dedicated application on their device and enters information about the child's age and interests. This information is sent to a server via the device. The server analyzes the received information and runs an AI model to suggest appropriate creative activities. The suggested activities include a variety of options tailored to the child's interests and age, such as drawing, storytelling, and DIY crafts.

[0039] The suggested activity is displayed on the device, and a detailed guide, along with a list of necessary materials and procedures, is provided. The user can then follow these instructions to collaborate on the creative activity with their child. Additional information and hints are provided as needed during the activity, ensuring smooth progress.

[0040] After completing their creative work, users digitize their artwork using their device's camera. The digitized artwork is then sent from the device to a server and stored in cloud storage. The artwork data is automatically organized and links are generated so that it can be shared with family and friends.

[0041] Furthermore, the server offers the option to create photo books and calendars using the digitized artwork. This allows users to preserve their children's growth and memories in a tangible form. For example, if an 8-year-old child is interested in dinosaurs, the server can suggest a dinosaur-themed clay art project and guide them through the creation process. The finished artwork can then be digitized and compiled into a photo book.

[0042] Thus, the system of the present invention provides a comprehensive support for creative activities at home and offers a series of means for recording and sharing creative works.

[0043] The following describes the processing flow.

[0044] Step 1:

[0045] The user launches a dedicated application on their device and enters information about their child's age and interests. This creates a user profile, and the device prepares to send this information to the server.

[0046] Step 2:

[0047] The terminal sends the entered information to the server, establishing secure communication with the server. Based on the received information, the server uses an AI model to begin suggesting creative activities.

[0048] Step 3:

[0049] The server analyzes the submitted information and selects the most suitable creative project. An AI model generates multiple projects based on the user's child's interests and age. The user then selects the most appropriate project from those generated and prepares its details.

[0050] Step 4:

[0051] The server sends an overview of the selected project, a list of required materials, and a step-by-step guide to the terminal. The terminal receives this information and displays it to the user, assisting them in starting their creative work.

[0052] Step 5:

[0053] Users refer to the guides provided on the device and engage in creative activities with their children. If they encounter any difficulties during the activity, they can refer to additional information and hints displayed on the device.

[0054] Step 6:

[0055] After completing their creative work, the user takes a picture of their work using the device's camera and records it as an image. The device automatically saves this digital data.

[0056] Step 7:

[0057] The device sends the saved artwork data to the server, which then stores the artwork in cloud storage. The server receives the digital data, organizes it securely, and manages it.

[0058] Step 8:

[0059] The server generates a special link to share artwork data stored in the cloud with others. The link is sent to the device and offered to the user as a sharing option.

[0060] Step 9:

[0061] The server displays options for creating photo books and calendars on the terminal, and users can proceed with the order process as needed. The server accepts orders and manages the printing and delivery process.

[0062] (Example 1)

[0063] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0064] There are challenges in appropriately proposing activities to foster children's creativity at home and supporting their safe and smooth implementation. Furthermore, there are challenges regarding the effort and methods involved in long-term recording and sharing of completed works.

[0065] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0066] In this invention, the server includes means for proposing creative activities suitable for the user's requests using a generative artificial intelligence model based on identification information obtained from the user's terminal, guide means for providing a list of materials and specific procedures related to the proposed activity, and means for converting the output of the creative activity into image data and recording it. This makes it possible to easily propose, implement, record, and share creative activities at home.

[0067] A "user terminal" is an electronic device used by users to input information or receive suggestions.

[0068] "Identification information" refers to data necessary to identify a user or their circumstances.

[0069] A "generative artificial intelligence model" is an artificial intelligence program used to generate suggestions based on the user's interests and needs.

[0070] "Creative activity" refers to activities in which users can produce some kind of work or deliverable through their participation.

[0071] A "materials list" is a list of specific materials and tools necessary for carrying out creative activities.

[0072] "Specific procedures" refer to the detailed steps that should be followed in order to carry out creative activities smoothly.

[0073] "Image data" refers to visual information used to digitally preserve the results of creative activities.

[0074] A "communication network" is a general term for the infrastructure and systems used to send and receive information between distant locations.

[0075] A "link" is a URL or connection point used when sharing information with others.

[0076] A "remote storage area" is an electronic space used to store data in a separate location, such as the cloud or an external server.

[0077] "Options that can be output as physical goods" refers to methods and options for providing digital information in a tangible form.

[0078] To implement this invention, the user first launches a dedicated application on their terminal and inputs information about the child's age and interests. This information is transmitted to a server via the Internet protocol. The server receives this information and inputs it into a generative artificial intelligence model for processing. The generative AI model suggests appropriate creative activities based on the user's identification information and sends the suggested content back to the terminal.

[0079] The proposed creative activity is displayed on the user interface of the terminal. This includes a detailed guide, a list of necessary materials, and specific instructions. The user follows the terminal's guide and carries out the activity. The resulting work is photographed using the terminal's camera and digitized as image data. This image data is transmitted to a server via the terminal and stored in a remote storage area.

[0080] The server manages the stored image data and generates links that can be shared with other users via a communication network. Through these links, users can share their creations online with family and friends. Furthermore, the server offers users options for outputting the image data as physical objects. For example, saved works can be printed as photo books or calendars.

[0081] For example, if an 8-year-old child is interested in dinosaurs, the following prompt can be input into the AI ​​model: "Suggest a creative activity using clay for an 8-year-old child who likes dinosaurs." This prompt allows the AI ​​model to suggest specific activities for creating dinosaur shapes, indicating the necessary materials and steps.

[0082] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0083] Step 1:

[0084] The user launches a dedicated application on their device and enters information about their child's age and interests. This input process involves filling in specific details such as "8 years old" and "dinosaurs" in a form. The entered data is formatted in JSON format and ready for submission.

[0085] Step 2:

[0086] The terminal encrypts the entered identification information and sends it to the server via a secure communication protocol (e.g., HTTPS). The terminal's encryption process ensures data security. Once the server receives the data, it is decrypted and stored in an internal database.

[0087] Step 3:

[0088] The server inputs the received information into an artificial intelligence model to generate prompts. For example, using the prompt, "Please suggest a creative activity using clay for an 8-year-old child who likes dinosaurs," the AI ​​model performs analysis. The generated suggestions are then returned to the terminal.

[0089] Step 4:

[0090] The terminal generates creative activity suggestions sent from the server and displays them in the user interface. Here, a detailed list of materials and specific steps required for the activity are provided. The user reviews this and is ready to begin the actual activity.

[0091] Step 5:

[0092] The user follows the instructions on the device and engages in creative activities with their child. Additional hints and advice are provided in real time from the device as needed. Answers to the user's questions are also provided on the device.

[0093] Step 6:

[0094] Once the creative process is complete, the user uses the device's camera to photograph the artwork and generate image data. The device then converts this data to the optimal format and prepares it for transmission to the server.

[0095] Step 7:

[0096] The server receives the image data transmitted by the device and stores it in cloud storage. The server organizes the data and automatically generates a shareable link over the communication network. This link is provided to the user via email or in-app notification.

[0097] Step 8:

[0098] Users can share their creations online with family and friends using a link provided by the server. Furthermore, the server offers users the option to output their work as a photo book or calendar.

[0099] (Application Example 1)

[0100] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0101] Conventional creative activity support systems rely on smartphones and computers for suggestions and guidance, resulting in a lack of support for activities through natural dialogue within the family. Furthermore, the digitization and sharing of artwork can be cumbersome, making it difficult to efficiently share the results of activities with family and friends.

[0102] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0103] In this invention, the server includes a suggestion means that proposes creative activities in response to the user's requests based on information acquired from the user's terminal, a guide means that provides detailed information about the activities proposed by the suggestion means, a recording means that records digital data of the works obtained as a result of the creative activities, and a driving means that is installed in a home-use machine and supports the implementation of creative activities through interaction with the user. This makes it possible to experience creative activities in the home in a more interactive way and to efficiently record and share work data.

[0104] A "user terminal" is a device that a user operates, and is used for inputting information and displaying results.

[0105] A "proposal tool" is a tool that has the function of selecting and proposing appropriate creative activities based on the user's input information.

[0106] A "guiding tool" is a means that has the function of providing users with detailed information and procedures for the proposed creative activity.

[0107] "Means of recording" refers to the means of recording works obtained through creative activity as digital data.

[0108] "Means of sharing" refers to the means of organizing recorded digital data of works and preparing or providing it for sharing with others.

[0109] "Household mechanical devices" refer to robots and mechanical devices used in a home environment that support creative activities through conversational functions.

[0110] "Driving means" refers to the means of assisting in the execution of creative activities by making full use of software and hardware installed in mechanical devices.

[0111] The system for carrying out this invention consists of a network structure including a user terminal, a household mechanical device (e.g., a household robot), and a server.

[0112] The user launches a dedicated application on their device and enters information about the child's age and interests. This information is sent from the user's device to the server. The server analyzes the entered information and uses a generative AI model to suggest creative activities suitable for the child. The generative AI model uses an AI engine that utilizes natural language processing technology (e.g., the GPT model) to generate suggestions through prompt sentences. One example is, "My 7-year-old child is interested in space. Please suggest creative activities appropriate for their age."

[0113] The proposed activities are presented to the user via the display or audio output of a home-use device. The terminal or device is equipped with a digital camera and microphone, allowing for the continued collection of additional information and progress data from the user. This information is used to provide advice and hints as the activity progresses.

[0114] Once a project is completed, the user digitizes it using a camera on their device or machine and sends it to the server. The server stores the digitized work in the cloud storage of the service and automatically generates a link to share it with family and friends.

[0115] The server also offers the option to create photo books and calendars using the saved works. This allows users to preserve their children's growth and memories in a tangible form. This entire process is carried out smoothly within the home through the coordination of the user's terminal and home-based electronic devices.

[0116] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0117] Step 1:

[0118] The user launches a dedicated application on their device and enters information about the child's age and interests. This information is structured in JSON format and sent to the server. This information includes the child's name, age, and interests.

[0119] Step 2:

[0120] The server parses the received JSON data. It extracts keywords related to age and interests obtained from the parsing and passes them to the generative AI model. The generative AI model generates prompt sentences based on the data it receives and inputs them into the AI ​​engine. For example, the prompt sentence "Example: A 7-year-old child is interested in space. Please suggest creative activities appropriate for their age." is generated.

[0121] Step 3:

[0122] The AI ​​engine receives a prompt and uses a generative AI model (e.g., a GPT model) to suggest appropriate creative activities. The suggested results are returned to the server in text format. The suggestions include a specific activity name, a list of required materials, and the steps of the activity.

[0123] Step 4:

[0124] The server transfers the suggested results received from the AI ​​engine to the user terminal and home appliances. The transferred content is then provided to the user through the user terminal's display or the monitor and speakers of the home appliance.

[0125] Step 5:

[0126] The user prepares the materials and begins the creative activity according to the proposed activity. During the activity, the user terminal or home appliance receives questions and requests from the user via voice or text and provides further guidance and hints based on them. Interaction takes place via speech recognition technology and natural language processing.

[0127] Step 6:

[0128] Once the activity is complete, users digitize their artwork using the cameras on their devices or other equipment. The digitized image data is sent to a server. The artwork data is then saved as image files in cloud storage.

[0129] Step 7:

[0130] The server uses the saved artwork data and offers the option to convert it into photo books or calendars, if desired by the user. Based on the user's selection, the server automatically selects the necessary templates and generates layouts. The final output is provided to the user in PDF or printable format.

[0131] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0132] This invention is a system that recognizes the user's emotional state and proposes and supports creative activities accordingly, thereby promoting children's creative activities in the home environment. Specific embodiments are described below.

[0133] This system consists of a user terminal, a server, and an emotion recognition engine. First, the user begins their activity using an application installed on the terminal. The terminal uses its camera and microphone to collect emotion data from the user's face and voice. This data is analyzed by the emotion engine to identify the user's current emotional state.

[0134] The server combines the output of the emotion recognition engine with information about the user's age and interests to suggest the most suitable creative activity using AI. This activity selects a theme and difficulty level appropriate to the user's emotional state and is then provided to the device.

[0135] The device displays activity information received from the server to the user, including detailed guidance. This guidance dynamically adjusts to the user's emotional state, including encouragement and cautionary notes, allowing the user to participate in creative activities with confidence.

[0136] During the activity, users follow instructions from their device and proceed through the process of creating their artwork. Throughout the activity, the device continuously acquires emotional data and sends feedback to the server as needed, which is used to adjust the guide content.

[0137] Once the creative process is complete, the user photographs the finished product with their device and digitizes it. This image data is sent to a server and stored in cloud storage. The server then uses the artwork data to add emotionally relevant messages and makes it shareable. Options to output the work as a photobook or calendar are also provided and are executed based on the user's choice.

[0138] For example, if an 8-year-old child is a little nervous at the start of an activity, the system suggests an activity where they draw animal illustrations in a gentle tone, accompanied by calming music. If the activity is progressing smoothly, based on the emotional data acquired by the device during the activity, the system displays a message on the screen with encouraging comments to boost their confidence. The completed artwork, along with the results of the emotional analysis, is shared in a way that facilitates emotional interaction among family members. In this way, the entire system understands the user's emotions and effectively supports the creative experience.

[0139] The following describes the processing flow.

[0140] Step 1:

[0141] The user launches the application on their device and enters their profile information (age, interests, etc.). The device's camera and microphone are activated, and it is ready to collect the user's emotional information (facial expressions and voice).

[0142] Step 2:

[0143] The device collects emotional data and sends it to the emotion engine in real time. The engine analyzes this data and identifies the user's emotional state. The emotional data is then prepared for transmission to the server.

[0144] Step 3:

[0145] The server receives the output of the emotion recognition engine and user profile information. Based on this data, the AI ​​model prepares to suggest creative activities that match the user's emotions. The specific activity themes and difficulty levels are then determined.

[0146] Step 4:

[0147] The server sends the selected activity to the terminal along with a detailed guide. The guide is tailored to include advice and encouraging messages that take the user's emotions into consideration.

[0148] Step 5:

[0149] The device displays the user with the activity details and guidance it has received. The user follows these instructions and begins their creative activity. During the activity, music and hints are provided to help with relaxation and concentration.

[0150] Step 6:

[0151] As the user progresses through the activity, the device continuously acquires emotional data and sends feedback to the server. Based on the emotional data, the server dynamically updates the guide content and sends it to the device as needed.

[0152] Step 7:

[0153] Once the user completes their creative activity, they take a picture of the finished work with their device's camera and save it as image data. The device then prepares to send this data to the server.

[0154] Step 8:

[0155] The device sends the artwork data to a server, which then manages the data in cloud storage. The server adds emotional messages to the artwork and organizes it in a shareable format.

[0156] Step 9:

[0157] The server provides the terminal with options to create photo books and calendars. The user selects and executes the option to save the work in the desired format. The server confirms the order and prepares it for printing and shipping.

[0158] (Example 2)

[0159] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0160] Existing creative activity support systems have the problem of being prone to a uniform experience because it is difficult to adjust individual suggestions and guides according to the user's emotional state. In addition, there is a lack of means to effectively share the results of creative activities, which means that the deliverables obtained from users' creative activities are not being utilized to their fullest potential.

[0161] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0162] In this invention, the server includes means for analyzing the user's emotional state and suggesting creative activities accordingly, means for dynamically adjusting and providing detailed information according to the suggested activities, and means for continuously monitoring the user's emotional data during creative activities. This enables personalized creative activity suggestions tailored to the user's emotions and effective sharing of creative works.

[0163] A "user terminal" is an electronic device that a user directly operates to input information and display results.

[0164] "Emotional state" refers to the psychological or emotional state a user is experiencing at a particular moment, and is inferred from their facial expressions and voice.

[0165] "Suggested means" refers to a method or apparatus that has the function of selecting and presenting appropriate creative activities based on the user's emotional state.

[0166] A "guiding device" is a method or apparatus that provides detailed instructions and information to assist in the progress of a proposed activity, thereby enriching the user's experience.

[0167] "Recording means" refers to a method or apparatus for digitizing and securely storing data of a work obtained as a result of creative activity.

[0168] "Means of sharing" refers to a method or device that provides recorded artwork data in a format accessible to others, enabling interaction and evaluation.

[0169] "Cloud storage" refers to a remote data storage service that stores data via the internet and allows access from anywhere.

[0170] "Printed material" refers to the process or result of outputting digital data onto a physical medium, making it visually verifiable.

[0171] This invention is a system that analyzes a user's emotional state, suggests creative activities, and enables the recording and sharing of creative data obtained through those activities. The system consists of a user terminal, a server, and an emotion recognition engine.

[0172] First, the user launches an application installed on their device and begins their creative activity. During this process, the device uses its camera and microphone to record the user's face and voice, and to acquire emotional data. This process utilizes OpenCV and PyTorch for facial recognition technology, and natural language processing software for voice data analysis.

[0173] The device transmits this acquired data in real time to the emotion recognition engine on the server. The emotion recognition engine uses a machine learning model to analyze and determine the user's current emotional state. For this purpose, natural language processing models such as Hugging Face are sometimes employed.

[0174] The server uses a generative AI model to suggest optimal creative activities based on the analyzed emotional state and user profile (such as age and interests). For example, a model from OpenAI® is used as this generative AI model. These suggestions include themes and difficulty levels based on the user's emotional state.

[0175] The device displays creative activity information and detailed guides received from the server to the user. These guides are dynamically adjusted according to the user's emotions and include activity steps and encouraging messages. The user can then proceed with their creative activity by following the device's instructions.

[0176] As a concrete example, for a child who is a little nervous at the start of an activity, the system suggests an activity where they draw animal illustrations accompanied by calming music. If the activity is progressing smoothly, based on the emotional data acquired by the device during the activity, comments that boost confidence will be displayed on the screen. In this way, the system supports the creative experience while providing feedback on the user's emotions.

[0177] Example prompt: "An 8-year-old child is feeling a little anxious. Please suggest some calming activities. Also, please provide a detailed guide of activities to help understand and support their emotions."

[0178] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0179] Step 1:

[0180] The user launches an application installed on their device to begin their creative activity. The device uses its camera and microphone to acquire data on the user's face and voice. This input data includes still images and audio. Specifically, OpenCV is used to capture facial expressions, and the Google® Speech-to-Text API is used to convert speech to text. This data serves as foundational data for inferring the user's emotional state.

[0181] Step 2:

[0182] The device sends the acquired emotional data to the emotion recognition engine on the server. The data received as input is analyzed using machine learning algorithms. For example, the Hugging Face Transformer model is applied to classify joy and sadness from facial expressions. As a result of this process, the user's emotional state is output as numerical or categorical data.

[0183] Step 3:

[0184] The server receives the analyzed emotional state and user-registered information such as age and interests (profile data) as input. Using this information, it proposes the most suitable creative activity using a generative AI model. This process uses an OpenAI model to generate prompts and select activities that match the user's emotions. The output includes the title, theme, and difficulty level of the proposed activity.

[0185] Step 4:

[0186] The terminal presents the user with activity suggestions sent from the server. Based on this, the terminal displays detailed guides and procedures in the user interface. These guides are dynamically adjusted to the user's emotions, and messages such as "Please proceed with confidence" may be displayed.

[0187] Step 5:

[0188] The user follows the instructions on the device to create their artwork. During the creation process, the device continuously collects the user's emotional data using the camera and microphone. This data is sent to the server as feedback, and the guide content is adjusted as needed.

[0189] Step 6:

[0190] Users photograph their completed artwork with their device and save it as digital data. This input data is recorded as an image file of the user's artwork. This ensures that the data is securely stored in cloud storage, thus preserving its integrity.

[0191] Step 7:

[0192] The server generates emotion-related messages based on saved artwork data and makes them shareable. This process uses a generative AI model to create messages and offers options to facilitate emotional interaction between users. Output options include the ability to convert the data into printed materials such as photo books and calendars.

[0193] (Application Example 2)

[0194] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as a "server" and the smart device 14 as a "terminal".

[0195] In modern family environments, there is a lack of means to encourage children to understand their own emotions and participate in rich creative activities. Furthermore, there is a need for systems that provide appropriate activities based on children's emotions and effectively share the results.

[0196] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0197] In this invention, the server includes an analysis means for acquiring emotional data and analyzing the user's emotional state, a suggestion means for proposing creative activities suitable for the user's emotional state based on the analysis results, and an adjustment means for continuously acquiring emotional data during the creative activity and adjusting the details of the activity as needed. This makes it possible to propose and support creative activities in accordance with the user's emotional state.

[0198] "Emotional data" refers to fundamental data used to analyze a user's emotional state, based on information obtained from the user's face and voice.

[0199] "Analysis means" refers to a means that analyzes emotional data acquired by a terminal and has the function of identifying the user's emotional state.

[0200] A "suggested means" is a means that, based on the analysis results obtained by the analysis means, presents creative activities that are appropriate to the user's emotional state.

[0201] A "guidance tool" is a tool that provides detailed information about a proposed activity and also dynamically presents encouragement and warnings tailored to the user's emotional state.

[0202] A "modification mechanism" is a means that continuously acquires emotional data during creative activity and dynamically adjusts the detailed information of the activity.

[0203] A "recording device" is a means that has the function of saving the data of a work obtained as a result of creative activity in a digital format.

[0204] A "sharing method" is a means that has the function of providing digitized artwork data, obtained through recording methods, in a form that can be shared with others.

[0205] The system for implementing this invention consists of several components for analyzing emotional data and providing creative activities tailored to the user. The main components include a terminal, a server, an emotional analysis engine, and cloud storage.

[0206] The device is installed on the user and uses a camera and microphone to acquire emotional data such as facial and voice data in real time. This data is sent to an emotion analysis engine. The emotion analysis engine processes the acquired data and identifies the user's emotional state. Software such as OpenCV and NVIDIA's DeepStream can be used for emotion analysis.

[0207] The server receives information about the user's emotional state from the emotion analysis engine and uses a generative AI model to suggest creative activities appropriate to that emotional state. The suggested activities are customized according to the user's age and interests and sent to the device. The device displays a detailed guide to the suggested activities on the screen and dynamically provides encouragement and warnings according to the user's emotional state.

[0208] During the activity, the device continuously acquires emotional data and sends feedback to the server as needed to modify and adjust the activity. This allows the user to enjoy an experience tailored to their state at any given time.

[0209] Once the creative activity is complete, the user records the result digitally and saves it to cloud storage. The server can then use this data to add emotionally relevant messages and output it in a shareable format. Options for outputting it as a photo book or calendar are also provided.

[0210] For example, if an 8-year-old child is a little nervous when starting an activity, the device will play relaxing music and suggest drawing an animal. If the activity goes well, the device will display an encouraging message such as, "Great job! Let's move on to the next step."

[0211] Another example of a prompt for a generative AI model is the text, "Please suggest a method for drawing pictures that children enjoy."

[0212] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0213] Step 1:

[0214] The device uses its camera and microphone to capture the user's facial image and voice. This acquired data is used as input data to analyze the user's emotional state.

[0215] Step 2:

[0216] The emotion analysis engine on the server, upon receiving emotion data transmitted from the terminal, performs image processing and voice analysis to identify the user's emotional state. This involves analyzing facial expressions using image processing libraries such as OpenCV, and analyzing voice tone and pitch. The output provides information about the user's emotional state.

[0217] Step 3:

[0218] The server inputs information about the user's emotional state into a generating AI model and suggests activities appropriate to the user's emotional state. The AI ​​model uses prompts to generate creative activities tailored to the user. The suggested activities include themes and difficulty levels that are appropriate to the user's characteristics.

[0219] Step 4:

[0220] The device displays suggested activities to the user and provides detailed guidance on their content. It offers encouragement and warnings tailored to the user's emotional state to ensure they can proceed with the activities with confidence. During this process, feedback is provided to the user through on-screen displays and audio.

[0221] Step 5:

[0222] While the user is engaged in creative activities, the device periodically retrieves emotional data and feeds it back to the server. Based on this feedback, the server dynamically adjusts the activity guide and sends it back to the device. This enables advice tailored to the user's progress.

[0223] Step 6:

[0224] Once the activity is complete, the user digitizes the output and sends it to cloud storage via their device. The server generates an emotionally relevant message based on the artwork data and provides output options such as photo books and calendars. The output is presented in a shareable format, making it easy to share with family and friends.

[0225] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0226] Data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of data generation model 58 is ChatGPT (registered trademark) (Internet search).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0227] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart device 14.

[0228] [Second Embodiment]

[0229] Figure 3 shows an example of the configuration of the data processing system 210 according to the second embodiment.

[0230] As shown in Figure 3, the data processing system 210 includes a data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0231] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0232] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication interface 44. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, and camera 42 are also connected to the bus 52.

[0233] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0234] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0235] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0236] Figure 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Figure 4, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0237] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0238] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0239] In the smart glasses 214, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0240] Next, the identification processing performed by the identification processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0241] The embodiments for carrying out the present invention are described below. This system connects a user terminal and a server via a network and can suggest, guide, record, and share creative activities in the home.

[0242] First, the user launches a dedicated application on their device and enters information about the child's age and interests. This information is sent to a server via the device. The server analyzes the received information and runs an AI model to suggest appropriate creative activities. The suggested activities include a variety of options tailored to the child's interests and age, such as drawing, storytelling, and DIY crafts.

[0243] The suggested activity is displayed on the device, and a detailed guide, along with a list of necessary materials and procedures, is provided. The user can then follow these instructions to collaborate on the creative activity with their child. Additional information and hints are provided as needed during the activity, ensuring smooth progress.

[0244] After completing their creative work, users digitize their artwork using their device's camera. The digitized artwork is then sent from the device to a server and stored in cloud storage. The artwork data is automatically organized and links are generated so that it can be shared with family and friends.

[0245] Furthermore, the server offers the option to create photo books and calendars using the digitized artwork. This allows users to preserve their children's growth and memories in a tangible form. For example, if an 8-year-old child is interested in dinosaurs, the server can suggest a dinosaur-themed clay art project and guide them through the creation process. The finished artwork can then be digitized and compiled into a photo book.

[0246] Thus, the system of the present invention provides a comprehensive support for creative activities at home and offers a series of means for recording and sharing creative works.

[0247] The following describes the processing flow.

[0248] Step 1:

[0249] The user launches a dedicated application on their device and enters information about their child's age and interests. This creates a user profile, and the device prepares to send this information to the server.

[0250] Step 2:

[0251] The terminal sends the entered information to the server, establishing secure communication with the server. Based on the received information, the server uses an AI model to begin suggesting creative activities.

[0252] Step 3:

[0253] The server analyzes the submitted information and selects the most suitable creative project. An AI model generates multiple projects based on the user's child's interests and age. The user then selects the most appropriate project from those generated and prepares its details.

[0254] Step 4:

[0255] The server sends an overview of the selected project, a list of required materials, and a step-by-step guide to the terminal. The terminal receives this information and displays it to the user, assisting them in starting their creative work.

[0256] Step 5:

[0257] Users refer to the guides provided on the device and engage in creative activities with their children. If they encounter any difficulties during the activity, they can refer to additional information and hints displayed on the device.

[0258] Step 6:

[0259] After completing their creative work, the user takes a picture of their work using the device's camera and records it as an image. The device automatically saves this digital data.

[0260] Step 7:

[0261] The device sends the saved artwork data to the server, which then stores the artwork in cloud storage. The server receives the digital data, organizes it securely, and manages it.

[0262] Step 8:

[0263] The server generates a special link to share artwork data stored in the cloud with others. The link is sent to the device and offered to the user as a sharing option.

[0264] Step 9:

[0265] The server displays options for creating photo books and calendars on the terminal, and users can proceed with the order process as needed. The server accepts orders and manages the printing and delivery process.

[0266] (Example 1)

[0267] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0268] There are challenges in appropriately proposing activities to foster children's creativity at home and supporting their safe and smooth implementation. Furthermore, there are challenges regarding the effort and methods involved in long-term recording and sharing of completed works.

[0269] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0270] In this invention, the server includes means for proposing creative activities suitable for the user's requests using a generative artificial intelligence model based on identification information obtained from the user's terminal, guide means for providing a list of materials and specific procedures related to the proposed activity, and means for converting the output of the creative activity into image data and recording it. This makes it possible to easily propose, implement, record, and share creative activities at home.

[0271] A "user terminal" is an electronic device used by users to input information or receive suggestions.

[0272] "Identification information" refers to data necessary to identify a user or their circumstances.

[0273] A "generative artificial intelligence model" is an artificial intelligence program used to generate suggestions based on the user's interests and needs.

[0274] "Creative activity" refers to activities in which users can produce some kind of work or deliverable through their participation.

[0275] A "materials list" is a list of specific materials and tools necessary for carrying out creative activities.

[0276] "Specific procedures" refer to the detailed steps that should be followed in order to carry out creative activities smoothly.

[0277] "Image data" refers to visual information used to digitally preserve the results of creative activities.

[0278] A "communication network" is a general term for the infrastructure and systems used to send and receive information between distant locations.

[0279] A "link" is a URL or connection point used when sharing information with others.

[0280] A "remote storage area" is an electronic space used to store data in a separate location, such as the cloud or an external server.

[0281] "Options that can be output as physical goods" refers to methods and options for providing digital information in a tangible form.

[0282] To implement this invention, the user first launches a dedicated application on the user terminal and enters information regarding the child's age and interests. This information is transmitted to the server via the Internet protocol. The server receives this information and inputs it into the generative artificial intelligence model for processing. The generative AI model proposes appropriate creative activities based on the user's identification information and returns the proposed content to the terminal.

[0283] The proposed creative activities are displayed on the user interface by the terminal. This includes detailed guides, a list of required materials, and specific procedures. The user follows the terminal's guide to carry out the activities. The works obtained as a result of the activities are photographed using the terminal's camera and digitized as image data. This image data is transmitted to the server through the terminal and stored in a remote storage area.

[0284] The server manages the stored image data and generates a link that can be shared with other users via a communication network. Through this link, the achievements can be shared online with family and friends. Furthermore, the server provides the user with options to output the image data as items. For example, the saved works can be printed as photo books or calendars.

[0285] As a specific example, when an 8-year-old child is interested in dinosaurs, the following prompt sentence can be input into the generative AI model. "Please propose a creative activity using paper clay for an 8-year-old child who likes dinosaurs." Based on this prompt sentence, the AI model can propose specific activities for making the shape of a dinosaur and show the required materials and procedures.

[0286] The flow of the specific process in Example 1 will be described using FIG. 11.

[0287] Step 1:

[0288] The user launches a dedicated application on their device and enters information about their child's age and interests. This input process involves filling in specific details such as "8 years old" and "dinosaurs" in a form. The entered data is formatted in JSON format and ready for submission.

[0289] Step 2:

[0290] The terminal encrypts the entered identification information and sends it to the server via a secure communication protocol (e.g., HTTPS). The terminal's encryption process ensures data security. Once the server receives the data, it is decrypted and stored in an internal database.

[0291] Step 3:

[0292] The server inputs the received information into an artificial intelligence model to generate prompts. For example, using the prompt, "Please suggest a creative activity using clay for an 8-year-old child who likes dinosaurs," the AI ​​model performs analysis. The generated suggestions are then returned to the terminal.

[0293] Step 4:

[0294] The terminal generates creative activity suggestions sent from the server and displays them in the user interface. Here, a detailed list of materials and specific steps required for the activity are provided. The user reviews this and is ready to begin the actual activity.

[0295] Step 5:

[0296] The user follows the instructions on the device and engages in creative activities with their child. Additional hints and advice are provided in real time from the device as needed. Answers to the user's questions are also provided on the device.

[0297] Step 6:

[0298] Once the creative process is complete, the user uses the device's camera to photograph the artwork and generate image data. The device then converts this data to the optimal format and prepares it for transmission to the server.

[0299] Step 7:

[0300] The server receives the image data transmitted by the device and stores it in cloud storage. The server organizes the data and automatically generates a shareable link over the communication network. This link is provided to the user via email or in-app notification.

[0301] Step 8:

[0302] Users can share their creations online with family and friends using a link provided by the server. Furthermore, the server offers users the option to output their work as a photo book or calendar.

[0303] (Application Example 1)

[0304] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0305] Conventional creative activity support systems rely on smartphones and computers for suggestions and guidance, resulting in a lack of support for activities through natural dialogue within the family. Furthermore, the digitization and sharing of artwork can be cumbersome, making it difficult to efficiently share the results of activities with family and friends.

[0306] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0307] In this invention, the server includes: a proposal means for proposing a creative activity according to the user's request based on the information acquired from the user terminal; a guide means for providing detailed information about the activity proposed by the proposal means; a recording means for recording the digital data of the work obtained as a result of the creative activity; and a driving means installed in a household mechanical device to assist in carrying out the creative activity through interaction with the user. Thereby, the creative activity within the home can be experienced more interactively, and efficient recording and sharing of work data becomes possible.

[0308] The "user terminal" is a device for the user to operate, which is a device for inputting information and displaying results.

[0309] The "proposal means" is a means having a function of selecting an appropriate creative activity based on the input information of the user and making a proposal.

[0310] The "guide means" is a means having a function of providing the user with detailed information and progress procedures of the proposed creative activity.

[0311] The "recording means" is a means for recording the work obtained by the creative activity as digital data.

[0312] The "sharing means" is a means for preparing and providing the recorded digital work data for sorting and sharing with others.

[0313] The "household mechanical device" is a robot or mechanical device used in a household environment, which is a device for assisting creative activities through an interactive function.

[0314] The "driving means" is a means for driving the software and hardware installed in the mechanical device to assist in the implementation of creative activities.

[0315] The system for carrying out this invention consists of a network structure including a user terminal, a household mechanical device (e.g., a household robot), and a server.

[0316] The user launches a dedicated application on their device and enters information about the child's age and interests. This information is sent from the user's device to the server. The server analyzes the entered information and uses a generative AI model to suggest creative activities suitable for the child. The generative AI model uses an AI engine that utilizes natural language processing technology (e.g., the GPT model) to generate suggestions through prompt sentences. One example is, "My 7-year-old child is interested in space. Please suggest creative activities appropriate for their age."

[0317] The proposed activities are presented to the user via the display or audio output of a home-use device. The terminal or device is equipped with a digital camera and microphone, allowing for the continued collection of additional information and progress data from the user. This information is used to provide advice and hints as the activity progresses.

[0318] Once a project is completed, the user digitizes it using a camera on their device or machine and sends it to the server. The server stores the digitized work in the cloud storage of the service and automatically generates a link to share it with family and friends.

[0319] The server also offers the option to create photo books and calendars using the saved works. This allows users to preserve their children's growth and memories in a tangible form. This entire process is carried out smoothly within the home through the coordination of the user's terminal and home-based electronic devices.

[0320] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0321] Step 1:

[0322] The user launches a dedicated application on their device and enters information about the child's age and interests. This information is structured in JSON format and sent to the server. This information includes the child's name, age, and interests.

[0323] Step 2:

[0324] The server parses the received JSON data. It extracts keywords related to age and interests obtained from the parsing and passes them to the generative AI model. The generative AI model generates prompt sentences based on the data it receives and inputs them into the AI ​​engine. For example, the prompt sentence "Example: A 7-year-old child is interested in space. Please suggest creative activities appropriate for their age." is generated.

[0325] Step 3:

[0326] The AI ​​engine receives a prompt and uses a generative AI model (e.g., a GPT model) to suggest appropriate creative activities. The suggested results are returned to the server in text format. The suggestions include a specific activity name, a list of required materials, and the steps of the activity.

[0327] Step 4:

[0328] The server transfers the suggested results received from the AI ​​engine to the user terminal and home appliances. The transferred content is then provided to the user through the user terminal's display or the monitor and speakers of the home appliance.

[0329] Step 5:

[0330] The user prepares the materials and begins the creative activity according to the proposed activity. During the activity, the user terminal or home appliance receives questions and requests from the user via voice or text and provides further guidance and hints based on them. Interaction takes place via speech recognition technology and natural language processing.

[0331] Step 6:

[0332] Once the activity is complete, users digitize their artwork using the cameras on their devices or other equipment. The digitized image data is sent to a server. The artwork data is then saved as image files in cloud storage.

[0333] Step 7:

[0334] The server uses the saved artwork data and offers the option to convert it into photo books or calendars, if desired by the user. Based on the user's selection, the server automatically selects the necessary templates and generates layouts. The final output is provided to the user in PDF or printable format.

[0335] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0336] This invention is a system that recognizes the user's emotional state and proposes and supports creative activities accordingly, thereby promoting children's creative activities in the home environment. Specific embodiments are described below.

[0337] This system consists of a user terminal, a server, and an emotion recognition engine. First, the user begins their activity using an application installed on the terminal. The terminal uses its camera and microphone to collect emotion data from the user's face and voice. This data is analyzed by the emotion engine to identify the user's current emotional state.

[0338] The server combines the output of the emotion recognition engine with information about the user's age and interests to suggest the most suitable creative activity using AI. This activity selects a theme and difficulty level appropriate to the user's emotional state and is then provided to the device.

[0339] The device displays activity information received from the server to the user, including detailed guidance. This guidance dynamically adjusts to the user's emotional state, including encouragement and cautionary notes, allowing the user to participate in creative activities with confidence.

[0340] During the activity, users follow instructions from their device and proceed through the process of creating their artwork. Throughout the activity, the device continuously acquires emotional data and sends feedback to the server as needed, which is used to adjust the guide content.

[0341] Once the creative process is complete, the user photographs the finished product with their device and digitizes it. This image data is sent to a server and stored in cloud storage. The server then uses the artwork data to add emotionally relevant messages and makes it shareable. Options to output the work as a photobook or calendar are also provided and are executed based on the user's choice.

[0342] For example, if an 8-year-old child is a little nervous at the start of an activity, the system suggests an activity where they draw animal illustrations in a gentle tone, accompanied by calming music. If the activity is progressing smoothly, based on the emotional data acquired by the device during the activity, the system displays a message on the screen with encouraging comments to boost their confidence. The completed artwork, along with the results of the emotional analysis, is shared in a way that facilitates emotional interaction among family members. In this way, the entire system understands the user's emotions and effectively supports the creative experience.

[0343] The following describes the processing flow.

[0344] Step 1:

[0345] The user launches the application on their device and enters their profile information (age, interests, etc.). The device's camera and microphone are activated, and it is ready to collect the user's emotional information (facial expressions and voice).

[0346] Step 2:

[0347] The device collects emotional data and sends it to the emotion engine in real time. The engine analyzes this data and identifies the user's emotional state. The emotional data is then prepared for transmission to the server.

[0348] Step 3:

[0349] The server receives the output of the emotion recognition engine and user profile information. Based on this data, the AI ​​model prepares to suggest creative activities that match the user's emotions. The specific activity themes and difficulty levels are then determined.

[0350] Step 4:

[0351] The server sends the selected activity to the terminal along with a detailed guide. The guide is tailored to include advice and encouraging messages that take the user's emotions into consideration.

[0352] Step 5:

[0353] The device displays the user with the activity details and guidance it has received. The user follows these instructions and begins their creative activity. During the activity, music and hints are provided to help with relaxation and concentration.

[0354] Step 6:

[0355] As the user progresses through the activity, the device continuously acquires emotional data and sends feedback to the server. Based on the emotional data, the server dynamically updates the guide content and sends it to the device as needed.

[0356] Step 7:

[0357] Once the user completes their creative activity, they take a picture of the finished work with their device's camera and save it as image data. The device then prepares to send this data to the server.

[0358] Step 8:

[0359] The device sends the artwork data to a server, which then manages the data in cloud storage. The server adds emotional messages to the artwork and organizes it in a shareable format.

[0360] Step 9:

[0361] The server provides the terminal with options to create photo books and calendars. The user selects and executes the option to save the work in the desired format. The server confirms the order and prepares it for printing and shipping.

[0362] (Example 2)

[0363] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0364] Existing creative activity support systems have the problem of being prone to a uniform experience because it is difficult to adjust individual suggestions and guides according to the user's emotional state. In addition, there is a lack of means to effectively share the results of creative activities, which means that the deliverables obtained from users' creative activities are not being utilized to their fullest potential.

[0365] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0366] In this invention, the server includes means for analyzing the user's emotional state and suggesting creative activities accordingly, means for dynamically adjusting and providing detailed information according to the suggested activities, and means for continuously monitoring the user's emotional data during creative activities. This enables personalized creative activity suggestions tailored to the user's emotions and effective sharing of creative works.

[0367] A "user terminal" is an electronic device that a user directly operates to input information and display results.

[0368] "Emotional state" refers to the psychological or emotional state a user is experiencing at a particular moment, and is inferred from their facial expressions and voice.

[0369] "Suggested means" refers to a method or apparatus that has the function of selecting and presenting appropriate creative activities based on the user's emotional state.

[0370] A "guiding device" is a method or apparatus that provides detailed instructions and information to assist in the progress of a proposed activity, thereby enriching the user's experience.

[0371] "Recording means" refers to a method or apparatus for digitizing and securely storing data of a work obtained as a result of creative activity.

[0372] "Means of sharing" refers to a method or device that provides recorded artwork data in a format accessible to others, enabling interaction and evaluation.

[0373] "Cloud storage" refers to a remote data storage service that stores data via the internet and allows access from anywhere.

[0374] "Printed material" refers to the process or result of outputting digital data onto a physical medium, making it visually verifiable.

[0375] This invention is a system that analyzes a user's emotional state, suggests creative activities, and enables the recording and sharing of creative data obtained through those activities. The system consists of a user terminal, a server, and an emotion recognition engine.

[0376] First, the user launches an application installed on their device and begins their creative activity. During this process, the device uses its camera and microphone to record the user's face and voice, and to acquire emotional data. This process utilizes OpenCV and PyTorch for facial recognition technology, and natural language processing software for voice data analysis.

[0377] The device transmits this acquired data in real time to the emotion recognition engine on the server. The emotion recognition engine uses a machine learning model to analyze and determine the user's current emotional state. For this purpose, natural language processing models such as Hugging Face are sometimes employed.

[0378] The server uses a generative AI model to suggest optimal creative activities based on the analyzed emotional state and user profile (age, interests, etc.). For example, OpenAI models are used as this generative AI model. These suggestions include themes and difficulty levels based on the user's emotional state.

[0379] The device displays creative activity information and detailed guides received from the server to the user. These guides are dynamically adjusted according to the user's emotions and include activity steps and encouraging messages. The user can then proceed with their creative activity by following the device's instructions.

[0380] As a concrete example, for a child who is a little nervous at the start of an activity, the system suggests an activity where they draw animal illustrations accompanied by calming music. If the activity is progressing smoothly, based on the emotional data acquired by the device during the activity, comments that boost confidence will be displayed on the screen. In this way, the system supports the creative experience while providing feedback on the user's emotions.

[0381] Example prompt: "An 8-year-old child is feeling a little anxious. Please suggest some calming activities. Also, please provide a detailed guide of activities to help understand and support their emotions."

[0382] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0383] Step 1:

[0384] The user launches an application installed on their device to begin their creative activity. The device uses its camera and microphone to acquire data on the user's face and voice. This input data includes still images and audio. Specifically, OpenCV is used to capture facial expressions, and the Google Speech-to-Text API is used to convert speech to text. This data serves as foundational data for inferring the user's emotional state.

[0385] Step 2:

[0386] The device sends the acquired emotional data to the emotion recognition engine on the server. The data received as input is analyzed using machine learning algorithms. For example, the Hugging Face Transformer model is applied to classify joy and sadness from facial expressions. As a result of this process, the user's emotional state is output as numerical or categorical data.

[0387] Step 3:

[0388] The server receives the analyzed emotional state and user-registered information such as age and interests (profile data) as input. Using this information, it proposes the most suitable creative activity using a generative AI model. This process uses an OpenAI model to generate prompts and select activities that match the user's emotions. The output includes the title, theme, and difficulty level of the proposed activity.

[0389] Step 4:

[0390] The terminal presents the user with activity suggestions sent from the server. Based on this, the terminal displays detailed guides and procedures in the user interface. These guides are dynamically adjusted to the user's emotions, and messages such as "Please proceed with confidence" may be displayed.

[0391] Step 5:

[0392] The user follows the instructions on the device to create their artwork. During the creation process, the device continuously collects the user's emotional data using the camera and microphone. This data is sent to the server as feedback, and the guide content is adjusted as needed.

[0393] Step 6:

[0394] Users photograph their completed artwork with their device and save it as digital data. This input data is recorded as an image file of the user's artwork. This ensures that the data is securely stored in cloud storage, thus preserving its integrity.

[0395] Step 7:

[0396] The server generates emotion-related messages based on saved artwork data and makes them shareable. This process uses a generative AI model to create messages and offers options to facilitate emotional interaction between users. Output options include the ability to convert the data into printed materials such as photo books and calendars.

[0397] (Application Example 2)

[0398] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0399] In modern family environments, there is a lack of means to encourage children to understand their own emotions and participate in rich creative activities. Furthermore, there is a need for systems that provide appropriate activities based on children's emotions and effectively share the results.

[0400] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0401] In this invention, the server includes an analysis means for acquiring emotional data and analyzing the user's emotional state, a suggestion means for proposing creative activities suitable for the user's emotional state based on the analysis results, and an adjustment means for continuously acquiring emotional data during the creative activity and adjusting the details of the activity as needed. This makes it possible to propose and support creative activities in accordance with the user's emotional state.

[0402] "Emotional data" refers to fundamental data used to analyze a user's emotional state, based on information obtained from the user's face and voice.

[0403] "Analysis means" refers to a means that analyzes emotional data acquired by a terminal and has the function of identifying the user's emotional state.

[0404] A "suggested means" is a means that, based on the analysis results obtained by the analysis means, presents creative activities that are appropriate to the user's emotional state.

[0405] A "guidance tool" is a tool that provides detailed information about a proposed activity and also dynamically presents encouragement and warnings tailored to the user's emotional state.

[0406] A "modification mechanism" is a means that continuously acquires emotional data during creative activity and dynamically adjusts the detailed information of the activity.

[0407] A "recording device" is a means that has the function of saving the data of a work obtained as a result of creative activity in a digital format.

[0408] A "sharing method" is a means that has the function of providing digitized artwork data, obtained through recording methods, in a form that can be shared with others.

[0409] The system for implementing this invention consists of several components for analyzing emotional data and providing creative activities tailored to the user. The main components include a terminal, a server, an emotional analysis engine, and cloud storage.

[0410] The device is installed on the user and uses a camera and microphone to acquire emotional data such as facial and voice data in real time. This data is sent to an emotion analysis engine. The emotion analysis engine processes the acquired data and identifies the user's emotional state. Software such as OpenCV and NVIDIA's DeepStream can be used for emotion analysis.

[0411] The server receives information about the user's emotional state from the emotion analysis engine and uses a generative AI model to suggest creative activities appropriate to that emotional state. The suggested activities are customized according to the user's age and interests and sent to the device. The device displays a detailed guide to the suggested activities on the screen and dynamically provides encouragement and warnings according to the user's emotional state.

[0412] During the activity, the device continuously acquires emotional data and sends feedback to the server as needed to modify and adjust the activity. This allows the user to enjoy an experience tailored to their state at any given time.

[0413] Once the creative activity is complete, the user records the result digitally and saves it to cloud storage. The server can then use this data to add emotionally relevant messages and output it in a shareable format. Options for outputting it as a photo book or calendar are also provided.

[0414] For example, if an 8-year-old child is a little nervous when starting an activity, the device will play relaxing music and suggest drawing an animal. If the activity goes well, the device will display an encouraging message such as, "Great job! Let's move on to the next step."

[0415] Another example of a prompt for a generative AI model is the text, "Please suggest a method for drawing pictures that children enjoy."

[0416] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0417] Step 1:

[0418] The device uses its camera and microphone to capture the user's facial image and voice. This acquired data is used as input data to analyze the user's emotional state.

[0419] Step 2:

[0420] The emotion analysis engine on the server, upon receiving emotion data transmitted from the terminal, performs image processing and voice analysis to identify the user's emotional state. This involves analyzing facial expressions using image processing libraries such as OpenCV, and analyzing voice tone and pitch. The output provides information about the user's emotional state.

[0421] Step 3:

[0422] The server inputs information about the user's emotional state into a generating AI model and suggests activities appropriate to the user's emotional state. The AI ​​model uses prompts to generate creative activities tailored to the user. The suggested activities include themes and difficulty levels that are appropriate to the user's characteristics.

[0423] Step 4:

[0424] The device displays suggested activities to the user and provides detailed guidance on their content. It offers encouragement and warnings tailored to the user's emotional state to ensure they can proceed with the activities with confidence. During this process, feedback is provided to the user through on-screen displays and audio.

[0425] Step 5:

[0426] While the user is engaged in creative activities, the device periodically retrieves emotional data and feeds it back to the server. Based on this feedback, the server dynamically adjusts the activity guide and sends it back to the device. This enables advice tailored to the user's progress.

[0427] Step 6:

[0428] Once the activity is complete, the user digitizes the output and sends it to cloud storage via their device. The server generates an emotionally relevant message based on the artwork data and provides output options such as photo books and calendars. The output is presented in a shareable format, making it easy to share with family and friends.

[0429] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0430] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0431] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart glasses 214.

[0432] [Third Embodiment]

[0433] Figure 5 shows an example of the configuration of the data processing system 310 according to the third embodiment.

[0434] As shown in Figure 5, the data processing system 310 includes a data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0435] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0436] The headset terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a display 343. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and display 343 are also connected to the bus 52.

[0437] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0438] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0439] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0440] Figure 6 shows an example of the main functions of the data processing device 12 and the headset terminal 314. As shown in Figure 6, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0441] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0442] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0443] In the headset terminal 314, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0444] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the headset terminal 314 will be referred to as the "terminal".

[0445] The embodiments for carrying out the present invention are described below. This system connects a user terminal and a server via a network and can suggest, guide, record, and share creative activities in the home.

[0446] First, the user launches a dedicated application on their device and enters information about the child's age and interests. This information is sent to a server via the device. The server analyzes the received information and runs an AI model to suggest appropriate creative activities. The suggested activities include a variety of options tailored to the child's interests and age, such as drawing, storytelling, and DIY crafts.

[0447] The suggested activity is displayed on the device, and a detailed guide, along with a list of necessary materials and procedures, is provided. The user can then follow these instructions to collaborate on the creative activity with their child. Additional information and hints are provided as needed during the activity, ensuring smooth progress.

[0448] After completing their creative work, users digitize their artwork using their device's camera. The digitized artwork is then sent from the device to a server and stored in cloud storage. The artwork data is automatically organized and links are generated so that it can be shared with family and friends.

[0449] Furthermore, the server offers the option to create photo books and calendars using the digitized artwork. This allows users to preserve their children's growth and memories in a tangible form. For example, if an 8-year-old child is interested in dinosaurs, the server can suggest a dinosaur-themed clay art project and guide them through the creation process. The finished artwork can then be digitized and compiled into a photo book.

[0450] Thus, the system of the present invention provides a comprehensive support for creative activities at home and offers a series of means for recording and sharing creative works.

[0451] The following describes the processing flow.

[0452] Step 1:

[0453] The user launches a dedicated application on their device and enters information about their child's age and interests. This creates a user profile, and the device prepares to send this information to the server.

[0454] Step 2:

[0455] The terminal sends the entered information to the server, establishing secure communication with the server. Based on the received information, the server uses an AI model to begin suggesting creative activities.

[0456] Step 3:

[0457] The server analyzes the submitted information and selects the most suitable creative project. An AI model generates multiple projects based on the user's child's interests and age. The user then selects the most appropriate project from those generated and prepares its details.

[0458] Step 4:

[0459] The server sends an overview of the selected project, a list of required materials, and a step-by-step guide to the terminal. The terminal receives this information and displays it to the user, assisting them in starting their creative work.

[0460] Step 5:

[0461] Users refer to the guides provided on the device and engage in creative activities with their children. If they encounter any difficulties during the activity, they can refer to additional information and hints displayed on the device.

[0462] Step 6:

[0463] After completing their creative work, the user takes a picture of their work using the device's camera and records it as an image. The device automatically saves this digital data.

[0464] Step 7:

[0465] The device sends the saved artwork data to the server, which then stores the artwork in cloud storage. The server receives the digital data, organizes it securely, and manages it.

[0466] Step 8:

[0467] The server generates a special link to share artwork data stored in the cloud with others. The link is sent to the device and offered to the user as a sharing option.

[0468] Step 9:

[0469] The server displays options for creating photo books and calendars on the terminal, and users can proceed with the order process as needed. The server accepts orders and manages the printing and delivery process.

[0470] (Example 1)

[0471] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0472] There are challenges in appropriately proposing activities to foster children's creativity at home and supporting their safe and smooth implementation. Furthermore, there are challenges regarding the effort and methods involved in long-term recording and sharing of completed works.

[0473] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0474] In this invention, the server includes means for proposing creative activities suitable for the user's requests using a generative artificial intelligence model based on identification information obtained from the user's terminal, guide means for providing a list of materials and specific procedures related to the proposed activity, and means for converting the output of the creative activity into image data and recording it. This makes it possible to easily propose, implement, record, and share creative activities at home.

[0475] A "user terminal" is an electronic device used by users to input information or receive suggestions.

[0476] "Identification information" refers to data necessary to identify a user or their circumstances.

[0477] A "generative artificial intelligence model" is an artificial intelligence program used to generate suggestions based on the user's interests and needs.

[0478] "Creative activity" refers to activities in which users can produce some kind of work or deliverable through their participation.

[0479] A "materials list" is a list of specific materials and tools necessary for carrying out creative activities.

[0480] "Specific procedures" refer to the detailed steps that should be followed in order to carry out creative activities smoothly.

[0481] "Image data" refers to visual information used to digitally preserve the results of creative activities.

[0482] A "communication network" is a general term for the infrastructure and systems used to send and receive information between distant locations.

[0483] A "link" is a URL or connection point used when sharing information with others.

[0484] A "remote storage area" is an electronic space used to store data in a separate location, such as the cloud or an external server.

[0485] "Options that can be output as physical goods" refers to methods and options for providing digital information in a tangible form.

[0486] To implement this invention, the user first launches a dedicated application on their terminal and inputs information about the child's age and interests. This information is transmitted to a server via the Internet protocol. The server receives this information and inputs it into a generative artificial intelligence model for processing. The generative AI model suggests appropriate creative activities based on the user's identification information and sends the suggested content back to the terminal.

[0487] The proposed creative activity is displayed on the user interface of the terminal. This includes a detailed guide, a list of necessary materials, and specific instructions. The user follows the terminal's guide and carries out the activity. The resulting work is photographed using the terminal's camera and digitized as image data. This image data is transmitted to a server via the terminal and stored in a remote storage area.

[0488] The server manages the stored image data and generates links that can be shared with other users via a communication network. Through these links, users can share their creations online with family and friends. Furthermore, the server offers users options for outputting the image data as physical objects. For example, saved works can be printed as photo books or calendars.

[0489] For example, if an 8-year-old child is interested in dinosaurs, the following prompt can be input into the AI ​​model: "Suggest a creative activity using clay for an 8-year-old child who likes dinosaurs." This prompt allows the AI ​​model to suggest specific activities for creating dinosaur shapes, indicating the necessary materials and steps.

[0490] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0491] Step 1:

[0492] The user launches a dedicated application on their device and enters information about their child's age and interests. This input process involves filling in specific details such as "8 years old" and "dinosaurs" in a form. The entered data is formatted in JSON format and ready for submission.

[0493] Step 2:

[0494] The terminal encrypts the entered identification information and sends it to the server via a secure communication protocol (e.g., HTTPS). The terminal's encryption process ensures data security. Once the server receives the data, it is decrypted and stored in an internal database.

[0495] Step 3:

[0496] The server inputs the received information into an artificial intelligence model to generate prompts. For example, using the prompt, "Please suggest a creative activity using clay for an 8-year-old child who likes dinosaurs," the AI ​​model performs analysis. The generated suggestions are then returned to the terminal.

[0497] Step 4:

[0498] The terminal generates creative activity suggestions sent from the server and displays them in the user interface. Here, a detailed list of materials and specific steps required for the activity are provided. The user reviews this and is ready to begin the actual activity.

[0499] Step 5:

[0500] The user follows the instructions on the device and engages in creative activities with their child. Additional hints and advice are provided in real time from the device as needed. Answers to the user's questions are also provided on the device.

[0501] Step 6:

[0502] Once the creative process is complete, the user uses the device's camera to photograph the artwork and generate image data. The device then converts this data to the optimal format and prepares it for transmission to the server.

[0503] Step 7:

[0504] The server receives the image data transmitted by the device and stores it in cloud storage. The server organizes the data and automatically generates a shareable link over the communication network. This link is provided to the user via email or in-app notification.

[0505] Step 8:

[0506] Users can share their creations online with family and friends using a link provided by the server. Furthermore, the server offers users the option to output their work as a photo book or calendar.

[0507] (Application Example 1)

[0508] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0509] Conventional creative activity support systems rely on smartphones and computers for suggestions and guidance, resulting in a lack of support for activities through natural dialogue within the family. Furthermore, the digitization and sharing of artwork can be cumbersome, making it difficult to efficiently share the results of activities with family and friends.

[0510] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0511] In this invention, the server includes a suggestion means that proposes creative activities in response to the user's requests based on information acquired from the user's terminal, a guide means that provides detailed information about the activities proposed by the suggestion means, a recording means that records digital data of the works obtained as a result of the creative activities, and a driving means that is installed in a home-use machine and supports the implementation of creative activities through interaction with the user. This makes it possible to experience creative activities in the home in a more interactive way and to efficiently record and share work data.

[0512] A "user terminal" is a device that a user operates, and is used for inputting information and displaying results.

[0513] A "proposal tool" is a tool that has the function of selecting and proposing appropriate creative activities based on the user's input information.

[0514] A "guiding tool" is a means that has the function of providing users with detailed information and procedures for the proposed creative activity.

[0515] "Means of recording" refers to the means of recording works obtained through creative activity as digital data.

[0516] "Means of sharing" refers to the means of organizing recorded digital data of works and preparing or providing it for sharing with others.

[0517] "Household mechanical devices" refer to robots and mechanical devices used in a home environment that support creative activities through conversational functions.

[0518] "Driving means" refers to the means of assisting in the execution of creative activities by making full use of software and hardware installed in mechanical devices.

[0519] The system for carrying out this invention consists of a network structure including a user terminal, a household mechanical device (e.g., a household robot), and a server.

[0520] The user launches a dedicated application on their device and enters information about the child's age and interests. This information is sent from the user's device to the server. The server analyzes the entered information and uses a generative AI model to suggest creative activities suitable for the child. The generative AI model uses an AI engine that utilizes natural language processing technology (e.g., the GPT model) to generate suggestions through prompt sentences. One example is, "My 7-year-old child is interested in space. Please suggest creative activities appropriate for their age."

[0521] The proposed activities are presented to the user via the display or audio output of a home-use device. The terminal or device is equipped with a digital camera and microphone, allowing for the continued collection of additional information and progress data from the user. This information is used to provide advice and hints as the activity progresses.

[0522] Once a project is completed, the user digitizes it using a camera on their device or machine and sends it to the server. The server stores the digitized work in the cloud storage of the service and automatically generates a link to share it with family and friends.

[0523] The server also offers the option to create photo books and calendars using the saved works. This allows users to preserve their children's growth and memories in a tangible form. This entire process is carried out smoothly within the home through the coordination of the user's terminal and home-based electronic devices.

[0524] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0525] Step 1:

[0526] The user launches a dedicated application on their device and enters information about the child's age and interests. This information is structured in JSON format and sent to the server. This information includes the child's name, age, and interests.

[0527] Step 2:

[0528] The server parses the received JSON data. It extracts keywords related to age and interests obtained from the parsing and passes them to the generative AI model. The generative AI model generates prompt sentences based on the data it receives and inputs them into the AI ​​engine. For example, the prompt sentence "Example: A 7-year-old child is interested in space. Please suggest creative activities appropriate for their age." is generated.

[0529] Step 3:

[0530] The AI ​​engine receives a prompt and uses a generative AI model (e.g., a GPT model) to suggest appropriate creative activities. The suggested results are returned to the server in text format. The suggestions include a specific activity name, a list of required materials, and the steps of the activity.

[0531] Step 4:

[0532] The server transfers the suggested results received from the AI ​​engine to the user terminal and home appliances. The transferred content is then provided to the user through the user terminal's display or the monitor and speakers of the home appliance.

[0533] Step 5:

[0534] The user prepares the materials and begins the creative activity according to the proposed activity. During the activity, the user terminal or home appliance receives questions and requests from the user via voice or text and provides further guidance and hints based on them. Interaction takes place via speech recognition technology and natural language processing.

[0535] Step 6:

[0536] Once the activity is complete, users digitize their artwork using the cameras on their devices or other equipment. The digitized image data is sent to a server. The artwork data is then saved as image files in cloud storage.

[0537] Step 7:

[0538] The server uses the saved artwork data and offers the option to convert it into photo books or calendars, if desired by the user. Based on the user's selection, the server automatically selects the necessary templates and generates layouts. The final output is provided to the user in PDF or printable format.

[0539] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0540] This invention is a system that recognizes the user's emotional state and proposes and supports creative activities accordingly, thereby promoting children's creative activities in the home environment. Specific embodiments are described below.

[0541] This system consists of a user terminal, a server, and an emotion recognition engine. First, the user begins their activity using an application installed on the terminal. The terminal uses its camera and microphone to collect emotion data from the user's face and voice. This data is analyzed by the emotion engine to identify the user's current emotional state.

[0542] The server combines the output of the emotion recognition engine with information about the user's age and interests to suggest the most suitable creative activity using AI. This activity selects a theme and difficulty level appropriate to the user's emotional state and is then provided to the device.

[0543] The device displays activity information received from the server to the user, including detailed guidance. This guidance dynamically adjusts to the user's emotional state, including encouragement and cautionary notes, allowing the user to participate in creative activities with confidence.

[0544] During the activity, users follow instructions from their device and proceed through the process of creating their artwork. Throughout the activity, the device continuously acquires emotional data and sends feedback to the server as needed, which is used to adjust the guide content.

[0545] Once the creative process is complete, the user photographs the finished product with their device and digitizes it. This image data is sent to a server and stored in cloud storage. The server then uses the artwork data to add emotionally relevant messages and makes it shareable. Options to output the work as a photobook or calendar are also provided and are executed based on the user's choice.

[0546] For example, if an 8-year-old child is a little nervous at the start of an activity, the system suggests an activity where they draw animal illustrations in a gentle tone, accompanied by calming music. If the activity is progressing smoothly, based on the emotional data acquired by the device during the activity, the system displays a message on the screen with encouraging comments to boost their confidence. The completed artwork, along with the results of the emotional analysis, is shared in a way that facilitates emotional interaction among family members. In this way, the entire system understands the user's emotions and effectively supports the creative experience.

[0547] The following describes the processing flow.

[0548] Step 1:

[0549] The user launches the application on their device and enters their profile information (age, interests, etc.). The device's camera and microphone are activated, and it is ready to collect the user's emotional information (facial expressions and voice).

[0550] Step 2:

[0551] The device collects emotional data and sends it to the emotion engine in real time. The engine analyzes this data and identifies the user's emotional state. The emotional data is then prepared for transmission to the server.

[0552] Step 3:

[0553] The server receives the output of the emotion recognition engine and user profile information. Based on this data, the AI ​​model prepares to suggest creative activities that match the user's emotions. The specific activity themes and difficulty levels are then determined.

[0554] Step 4:

[0555] The server sends the selected activity to the terminal along with a detailed guide. The guide is tailored to include advice and encouraging messages that take the user's emotions into consideration.

[0556] Step 5:

[0557] The device displays the user with the activity details and guidance it has received. The user follows these instructions and begins their creative activity. During the activity, music and hints are provided to help with relaxation and concentration.

[0558] Step 6:

[0559] As the user progresses through the activity, the device continuously acquires emotional data and sends feedback to the server. Based on the emotional data, the server dynamically updates the guide content and sends it to the device as needed.

[0560] Step 7:

[0561] Once the user completes their creative activity, they take a picture of the finished work with their device's camera and save it as image data. The device then prepares to send this data to the server.

[0562] Step 8:

[0563] The device sends the artwork data to a server, which then manages the data in cloud storage. The server adds emotional messages to the artwork and organizes it in a shareable format.

[0564] Step 9:

[0565] The server provides the terminal with options to create photo books and calendars. The user selects and executes the option to save the work in the desired format. The server confirms the order and prepares it for printing and shipping.

[0566] (Example 2)

[0567] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0568] Existing creative activity support systems have the problem of being prone to a uniform experience because it is difficult to adjust individual suggestions and guides according to the user's emotional state. In addition, there is a lack of means to effectively share the results of creative activities, which means that the deliverables obtained from users' creative activities are not being utilized to their fullest potential.

[0569] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0570] In this invention, the server includes means for analyzing the user's emotional state and suggesting creative activities accordingly, means for dynamically adjusting and providing detailed information according to the suggested activities, and means for continuously monitoring the user's emotional data during creative activities. This enables personalized creative activity suggestions tailored to the user's emotions and effective sharing of creative works.

[0571] A "user terminal" is an electronic device that a user directly operates to input information and display results.

[0572] "Emotional state" refers to the psychological or emotional state a user is experiencing at a particular moment, and is inferred from their facial expressions and voice.

[0573] "Suggested means" refers to a method or apparatus that has the function of selecting and presenting appropriate creative activities based on the user's emotional state.

[0574] A "guiding device" is a method or apparatus that provides detailed instructions and information to assist in the progress of a proposed activity, thereby enriching the user's experience.

[0575] "Recording means" refers to a method or apparatus for digitizing and securely storing data of a work obtained as a result of creative activity.

[0576] "Means of sharing" refers to a method or device that provides recorded artwork data in a format accessible to others, enabling interaction and evaluation.

[0577] "Cloud storage" refers to a remote data storage service that stores data via the internet and allows access from anywhere.

[0578] "Printed material" refers to the process or result of outputting digital data onto a physical medium, making it visually verifiable.

[0579] This invention is a system that analyzes a user's emotional state, suggests creative activities, and enables the recording and sharing of creative data obtained through those activities. The system consists of a user terminal, a server, and an emotion recognition engine.

[0580] First, the user launches an application installed on their device and begins their creative activity. During this process, the device uses its camera and microphone to record the user's face and voice, and to acquire emotional data. This process utilizes OpenCV and PyTorch for facial recognition technology, and natural language processing software for voice data analysis.

[0581] The device transmits this acquired data in real time to the emotion recognition engine on the server. The emotion recognition engine uses a machine learning model to analyze and determine the user's current emotional state. For this purpose, natural language processing models such as Hugging Face are sometimes employed.

[0582] The server uses a generative AI model to suggest optimal creative activities based on the analyzed emotional state and user profile (age, interests, etc.). For example, OpenAI models are used as this generative AI model. These suggestions include themes and difficulty levels based on the user's emotional state.

[0583] The device displays creative activity information and detailed guides received from the server to the user. These guides are dynamically adjusted according to the user's emotions and include activity steps and encouraging messages. The user can then proceed with their creative activity by following the device's instructions.

[0584] As a concrete example, for a child who is a little nervous at the start of an activity, the system suggests an activity where they draw animal illustrations accompanied by calming music. If the activity is progressing smoothly, based on the emotional data acquired by the device during the activity, comments that boost confidence will be displayed on the screen. In this way, the system supports the creative experience while providing feedback on the user's emotions.

[0585] Example prompt: "An 8-year-old child is feeling a little anxious. Please suggest some calming activities. Also, please provide a detailed guide of activities to help understand and support their emotions."

[0586] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0587] Step 1:

[0588] The user launches an application installed on their device to begin their creative activity. The device uses its camera and microphone to acquire data on the user's face and voice. This input data includes still images and audio. Specifically, OpenCV is used to capture facial expressions, and the Google Speech-to-Text API is used to convert speech to text. This data serves as foundational data for inferring the user's emotional state.

[0589] Step 2:

[0590] The device sends the acquired emotional data to the emotion recognition engine on the server. The data received as input is analyzed using machine learning algorithms. For example, the Hugging Face Transformer model is applied to classify joy and sadness from facial expressions. As a result of this process, the user's emotional state is output as numerical or categorical data.

[0591] Step 3:

[0592] The server receives the analyzed emotional state and user-registered information such as age and interests (profile data) as input. Using this information, it proposes the most suitable creative activity using a generative AI model. This process uses an OpenAI model to generate prompts and select activities that match the user's emotions. The output includes the title, theme, and difficulty level of the proposed activity.

[0593] Step 4:

[0594] The terminal presents the user with activity suggestions sent from the server. Based on this, the terminal displays detailed guides and procedures in the user interface. These guides are dynamically adjusted to the user's emotions, and messages such as "Please proceed with confidence" may be displayed.

[0595] Step 5:

[0596] The user follows the instructions on the device to create their artwork. During the creation process, the device continuously collects the user's emotional data using the camera and microphone. This data is sent to the server as feedback, and the guide content is adjusted as needed.

[0597] Step 6:

[0598] Users photograph their completed artwork with their device and save it as digital data. This input data is recorded as an image file of the user's artwork. This ensures that the data is securely stored in cloud storage, thus preserving its integrity.

[0599] Step 7:

[0600] The server generates emotion-related messages based on saved artwork data and makes them shareable. This process uses a generative AI model to create messages and offers options to facilitate emotional interaction between users. Output options include the ability to convert the data into printed materials such as photo books and calendars.

[0601] (Application Example 2)

[0602] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0603] In modern family environments, there is a lack of means to encourage children to understand their own emotions and participate in rich creative activities. Furthermore, there is a need for systems that provide appropriate activities based on children's emotions and effectively share the results.

[0604] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0605] In this invention, the server includes an analysis means for acquiring emotional data and analyzing the user's emotional state, a suggestion means for proposing creative activities suitable for the user's emotional state based on the analysis results, and an adjustment means for continuously acquiring emotional data during the creative activity and adjusting the details of the activity as needed. This makes it possible to propose and support creative activities in accordance with the user's emotional state.

[0606] "Emotional data" refers to fundamental data used to analyze a user's emotional state, based on information obtained from the user's face and voice.

[0607] "Analysis means" refers to a means that analyzes emotional data acquired by a terminal and has the function of identifying the user's emotional state.

[0608] A "suggested means" is a means that, based on the analysis results obtained by the analysis means, presents creative activities that are appropriate to the user's emotional state.

[0609] A "guidance tool" is a tool that provides detailed information about a proposed activity and also dynamically presents encouragement and warnings tailored to the user's emotional state.

[0610] A "modification mechanism" is a means that continuously acquires emotional data during creative activity and dynamically adjusts the detailed information of the activity.

[0611] A "recording device" is a means that has the function of saving the data of a work obtained as a result of creative activity in a digital format.

[0612] A "sharing method" is a means that has the function of providing digitized artwork data, obtained through recording methods, in a form that can be shared with others.

[0613] The system for implementing this invention consists of several components for analyzing emotional data and providing creative activities tailored to the user. The main components include a terminal, a server, an emotional analysis engine, and cloud storage.

[0614] The device is installed on the user and uses a camera and microphone to acquire emotional data such as facial and voice data in real time. This data is sent to an emotion analysis engine. The emotion analysis engine processes the acquired data and identifies the user's emotional state. Software such as OpenCV and NVIDIA's DeepStream can be used for emotion analysis.

[0615] The server receives information about the user's emotional state from the emotion analysis engine and uses a generative AI model to suggest creative activities appropriate to that emotional state. The suggested activities are customized according to the user's age and interests and sent to the device. The device displays a detailed guide to the suggested activities on the screen and dynamically provides encouragement and warnings according to the user's emotional state.

[0616] During the activity, the device continuously acquires emotional data and sends feedback to the server as needed to modify and adjust the activity. This allows the user to enjoy an experience tailored to their state at any given time.

[0617] Once the creative activity is complete, the user records the result digitally and saves it to cloud storage. The server can then use this data to add emotionally relevant messages and output it in a shareable format. Options for outputting it as a photo book or calendar are also provided.

[0618] For example, if an 8-year-old child is a little nervous when starting an activity, the device will play relaxing music and suggest drawing an animal. If the activity goes well, the device will display an encouraging message such as, "Great job! Let's move on to the next step."

[0619] Another example of a prompt for a generative AI model is the text, "Please suggest a method for drawing pictures that children enjoy."

[0620] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0621] Step 1:

[0622] The device uses its camera and microphone to capture the user's facial image and voice. This acquired data is used as input data to analyze the user's emotional state.

[0623] Step 2:

[0624] The emotion analysis engine on the server, upon receiving emotion data transmitted from the terminal, performs image processing and voice analysis to identify the user's emotional state. This involves analyzing facial expressions using image processing libraries such as OpenCV, and analyzing voice tone and pitch. The output provides information about the user's emotional state.

[0625] Step 3:

[0626] The server inputs information about the user's emotional state into a generating AI model and suggests activities appropriate to the user's emotional state. The AI ​​model uses prompts to generate creative activities tailored to the user. The suggested activities include themes and difficulty levels that are appropriate to the user's characteristics.

[0627] Step 4:

[0628] The device displays suggested activities to the user and provides detailed guidance on their content. It offers encouragement and warnings tailored to the user's emotional state to ensure they can proceed with the activities with confidence. During this process, feedback is provided to the user through on-screen displays and audio.

[0629] Step 5:

[0630] While the user is engaged in creative activities, the device periodically retrieves emotional data and feeds it back to the server. Based on this feedback, the server dynamically adjusts the activity guide and sends it back to the device. This enables advice tailored to the user's progress.

[0631] Step 6:

[0632] Once the activity is complete, the user digitizes the output and sends it to cloud storage via their device. The server generates an emotionally relevant message based on the artwork data and provides output options such as photo books and calendars. The output is presented in a shareable format, making it easy to share with family and friends.

[0633] The specific processing unit 290 transmits the result of the specific processing to the headset terminal 314. In the headset terminal 314, the control unit 46A causes the speaker 240 and display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0634] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0635] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and specific processing may also be performed by the headset terminal 314.

[0636] [Fourth Embodiment]

[0637] Figure 7 shows an example of the configuration of the data processing system 410 according to the fourth embodiment.

[0638] As shown in Figure 7, the data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[0639] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0640] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a controlled object 443. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and controlled object 443 are also connected to the bus 52.

[0641] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0642] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0643] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0644] The controlled object 443 includes a display device, LEDs in the eyes, and motors that drive the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the robot 414's emotions can be expressed by controlling these motors. Furthermore, the robot 414's facial expressions can also be expressed by controlling the illumination state of the LEDs in its eyes.

[0645] Figure 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Figure 8, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0646] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0647] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0648] In robot 414, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0649] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0650] The embodiments for carrying out the present invention are described below. This system connects a user terminal and a server via a network and can suggest, guide, record, and share creative activities in the home.

[0651] First, the user launches a dedicated application on their device and enters information about the child's age and interests. This information is sent to a server via the device. The server analyzes the received information and runs an AI model to suggest appropriate creative activities. The suggested activities include a variety of options tailored to the child's interests and age, such as drawing, storytelling, and DIY crafts.

[0652] The suggested activity is displayed on the device, and a detailed guide, along with a list of necessary materials and procedures, is provided. The user can then follow these instructions to collaborate on the creative activity with their child. Additional information and hints are provided as needed during the activity, ensuring smooth progress.

[0653] After completing their creative work, users digitize their artwork using their device's camera. The digitized artwork is then sent from the device to a server and stored in cloud storage. The artwork data is automatically organized and links are generated so that it can be shared with family and friends.

[0654] Furthermore, the server offers the option to create photo books and calendars using the digitized artwork. This allows users to preserve their children's growth and memories in a tangible form. For example, if an 8-year-old child is interested in dinosaurs, the server can suggest a dinosaur-themed clay art project and guide them through the creation process. The finished artwork can then be digitized and compiled into a photo book.

[0655] Thus, the system of the present invention provides a comprehensive support for creative activities at home and offers a series of means for recording and sharing creative works.

[0656] The following describes the processing flow.

[0657] Step 1:

[0658] The user launches a dedicated application on their device and enters information about their child's age and interests. This creates a user profile, and the device prepares to send this information to the server.

[0659] Step 2:

[0660] The terminal sends the entered information to the server, establishing secure communication with the server. Based on the received information, the server uses an AI model to begin suggesting creative activities.

[0661] Step 3:

[0662] The server analyzes the submitted information and selects the most suitable creative project. An AI model generates multiple projects based on the user's child's interests and age. The user then selects the most appropriate project from those generated and prepares its details.

[0663] Step 4:

[0664] The server sends an overview of the selected project, a list of required materials, and a step-by-step guide to the terminal. The terminal receives this information and displays it to the user, assisting them in starting their creative work.

[0665] Step 5:

[0666] Users refer to the guides provided on the device and engage in creative activities with their children. If they encounter any difficulties during the activity, they can refer to additional information and hints displayed on the device.

[0667] Step 6:

[0668] After completing their creative work, the user takes a picture of their work using the device's camera and records it as an image. The device automatically saves this digital data.

[0669] Step 7:

[0670] The device sends the saved artwork data to the server, which then stores the artwork in cloud storage. The server receives the digital data, organizes it securely, and manages it.

[0671] Step 8:

[0672] The server generates a special link to share artwork data stored in the cloud with others. The link is sent to the device and offered to the user as a sharing option.

[0673] Step 9:

[0674] The server displays options for creating photo books and calendars on the terminal, and users can proceed with the order process as needed. The server accepts orders and manages the printing and delivery process.

[0675] (Example 1)

[0676] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0677] There are challenges in appropriately proposing activities to foster children's creativity at home and supporting their safe and smooth implementation. Furthermore, there are challenges regarding the effort and methods involved in long-term recording and sharing of completed works.

[0678] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0679] In this invention, the server includes means for proposing creative activities suitable for the user's requests using a generative artificial intelligence model based on identification information obtained from the user's terminal, guide means for providing a list of materials and specific procedures related to the proposed activity, and means for converting the output of the creative activity into image data and recording it. This makes it possible to easily propose, implement, record, and share creative activities at home.

[0680] A "user terminal" is an electronic device used by users to input information or receive suggestions.

[0681] "Identification information" refers to data necessary to identify a user or their circumstances.

[0682] A "generative artificial intelligence model" is an artificial intelligence program used to generate suggestions based on the user's interests and needs.

[0683] "Creative activity" refers to activities in which users can produce some kind of work or deliverable through their participation.

[0684] A "materials list" is a list of specific materials and tools necessary for carrying out creative activities.

[0685] "Specific procedures" refer to the detailed steps that should be followed in order to carry out creative activities smoothly.

[0686] "Image data" refers to visual information used to digitally preserve the results of creative activities.

[0687] A "communication network" is a general term for the infrastructure and systems used to send and receive information between distant locations.

[0688] A "link" is a URL or connection point used when sharing information with others.

[0689] A "remote storage area" is an electronic space used to store data in a separate location, such as the cloud or an external server.

[0690] "Options that can be output as physical goods" refers to methods and options for providing digital information in a tangible form.

[0691] To implement this invention, the user first launches a dedicated application on their terminal and inputs information about the child's age and interests. This information is transmitted to a server via the Internet protocol. The server receives this information and inputs it into a generative artificial intelligence model for processing. The generative AI model suggests appropriate creative activities based on the user's identification information and sends the suggested content back to the terminal.

[0692] The proposed creative activity is displayed on the user interface of the terminal. This includes a detailed guide, a list of necessary materials, and specific instructions. The user follows the terminal's guide and carries out the activity. The resulting work is photographed using the terminal's camera and digitized as image data. This image data is transmitted to a server via the terminal and stored in a remote storage area.

[0693] The server manages the stored image data and generates links that can be shared with other users via a communication network. Through these links, users can share their creations online with family and friends. Furthermore, the server offers users options for outputting the image data as physical objects. For example, saved works can be printed as photo books or calendars.

[0694] For example, if an 8-year-old child is interested in dinosaurs, the following prompt can be input into the AI ​​model: "Suggest a creative activity using clay for an 8-year-old child who likes dinosaurs." This prompt allows the AI ​​model to suggest specific activities for creating dinosaur shapes, indicating the necessary materials and steps.

[0695] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0696] Step 1:

[0697] The user launches a dedicated application on their device and enters information about their child's age and interests. This input process involves filling in specific details such as "8 years old" and "dinosaurs" in a form. The entered data is formatted in JSON format and ready for submission.

[0698] Step 2:

[0699] The terminal encrypts the entered identification information and sends it to the server via a secure communication protocol (e.g., HTTPS). The terminal's encryption process ensures data security. Once the server receives the data, it is decrypted and stored in an internal database.

[0700] Step 3:

[0701] The server inputs the received information into an artificial intelligence model to generate prompts. For example, using the prompt, "Please suggest a creative activity using clay for an 8-year-old child who likes dinosaurs," the AI ​​model performs analysis. The generated suggestions are then returned to the terminal.

[0702] Step 4:

[0703] The terminal generates creative activity suggestions sent from the server and displays them in the user interface. Here, a detailed list of materials and specific steps required for the activity are provided. The user reviews this and is ready to begin the actual activity.

[0704] Step 5:

[0705] The user follows the instructions on the device and engages in creative activities with their child. Additional hints and advice are provided in real time from the device as needed. Answers to the user's questions are also provided on the device.

[0706] Step 6:

[0707] Once the creative process is complete, the user uses the device's camera to photograph the artwork and generate image data. The device then converts this data to the optimal format and prepares it for transmission to the server.

[0708] Step 7:

[0709] The server receives the image data transmitted by the device and stores it in cloud storage. The server organizes the data and automatically generates a shareable link over the communication network. This link is provided to the user via email or in-app notification.

[0710] Step 8:

[0711] Users can share their creations online with family and friends using a link provided by the server. Furthermore, the server offers users the option to output their work as a photo book or calendar.

[0712] (Application Example 1)

[0713] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0714] Conventional creative activity support systems rely on smartphones and computers for suggestions and guidance, resulting in a lack of support for activities through natural dialogue within the family. Furthermore, the digitization and sharing of artwork can be cumbersome, making it difficult to efficiently share the results of activities with family and friends.

[0715] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0716] In this invention, the server includes a suggestion means that proposes creative activities in response to the user's requests based on information acquired from the user's terminal, a guide means that provides detailed information about the activities proposed by the suggestion means, a recording means that records digital data of the works obtained as a result of the creative activities, and a driving means that is installed in a home-use machine and supports the implementation of creative activities through interaction with the user. This makes it possible to experience creative activities in the home in a more interactive way and to efficiently record and share work data.

[0717] A "user terminal" is a device that a user operates, and is used for inputting information and displaying results.

[0718] A "proposal tool" is a tool that has the function of selecting and proposing appropriate creative activities based on the user's input information.

[0719] A "guiding tool" is a means that has the function of providing users with detailed information and procedures for the proposed creative activity.

[0720] "Means of recording" refers to the means of recording works obtained through creative activity as digital data.

[0721] "Means of sharing" refers to the means of organizing recorded digital data of works and preparing or providing it for sharing with others.

[0722] "Household mechanical devices" refer to robots and mechanical devices used in a home environment that support creative activities through conversational functions.

[0723] "Driving means" refers to the means of assisting in the execution of creative activities by making full use of software and hardware installed in mechanical devices.

[0724] The system for carrying out this invention consists of a network structure including a user terminal, a household mechanical device (e.g., a household robot), and a server.

[0725] The user launches a dedicated application on their device and enters information about the child's age and interests. This information is sent from the user's device to the server. The server analyzes the entered information and uses a generative AI model to suggest creative activities suitable for the child. The generative AI model uses an AI engine that utilizes natural language processing technology (e.g., the GPT model) to generate suggestions through prompt sentences. One example is, "My 7-year-old child is interested in space. Please suggest creative activities appropriate for their age."

[0726] The proposed activities are presented to the user via the display or audio output of a home-use device. The terminal or device is equipped with a digital camera and microphone, allowing for the continued collection of additional information and progress data from the user. This information is used to provide advice and hints as the activity progresses.

[0727] Once a project is completed, the user digitizes it using a camera on their device or machine and sends it to the server. The server stores the digitized work in the cloud storage of the service and automatically generates a link to share it with family and friends.

[0728] The server also offers the option to create photo books and calendars using the saved works. This allows users to preserve their children's growth and memories in a tangible form. This entire process is carried out smoothly within the home through the coordination of the user's terminal and home-based electronic devices.

[0729] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0730] Step 1:

[0731] The user launches a dedicated application on their device and enters information about the child's age and interests. This information is structured in JSON format and sent to the server. This information includes the child's name, age, and interests.

[0732] Step 2:

[0733] The server parses the received JSON data. It extracts keywords related to age and interests obtained from the parsing and passes them to the generative AI model. The generative AI model generates prompt sentences based on the data it receives and inputs them into the AI ​​engine. For example, the prompt sentence "Example: A 7-year-old child is interested in space. Please suggest creative activities appropriate for their age." is generated.

[0734] Step 3:

[0735] The AI ​​engine receives a prompt and uses a generative AI model (e.g., a GPT model) to suggest appropriate creative activities. The suggested results are returned to the server in text format. The suggestions include a specific activity name, a list of required materials, and the steps of the activity.

[0736] Step 4:

[0737] The server transfers the suggested results received from the AI ​​engine to the user terminal and home appliances. The transferred content is then provided to the user through the user terminal's display or the monitor and speakers of the home appliance.

[0738] Step 5:

[0739] The user prepares the materials and begins the creative activity according to the proposed activity. During the activity, the user terminal or home appliance receives questions and requests from the user via voice or text and provides further guidance and hints based on them. Interaction takes place via speech recognition technology and natural language processing.

[0740] Step 6:

[0741] Once the activity is complete, users digitize their artwork using the cameras on their devices or other equipment. The digitized image data is sent to a server. The artwork data is then saved as image files in cloud storage.

[0742] Step 7:

[0743] The server uses the saved artwork data and offers the option to convert it into photo books or calendars, if desired by the user. Based on the user's selection, the server automatically selects the necessary templates and generates layouts. The final output is provided to the user in PDF or printable format.

[0744] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0745] This invention is a system that recognizes the user's emotional state and proposes and supports creative activities accordingly, thereby promoting children's creative activities in the home environment. Specific embodiments are described below.

[0746] This system consists of a user terminal, a server, and an emotion recognition engine. First, the user begins their activity using an application installed on the terminal. The terminal uses its camera and microphone to collect emotion data from the user's face and voice. This data is analyzed by the emotion engine to identify the user's current emotional state.

[0747] The server combines the output of the emotion recognition engine with information about the user's age and interests to suggest the most suitable creative activity using AI. This activity selects a theme and difficulty level appropriate to the user's emotional state and is then provided to the device.

[0748] The device displays activity information received from the server to the user, including detailed guidance. This guidance dynamically adjusts to the user's emotional state, including encouragement and cautionary notes, allowing the user to participate in creative activities with confidence.

[0749] During the activity, users follow instructions from their device and proceed through the process of creating their artwork. Throughout the activity, the device continuously acquires emotional data and sends feedback to the server as needed, which is used to adjust the guide content.

[0750] Once the creative process is complete, the user photographs the finished product with their device and digitizes it. This image data is sent to a server and stored in cloud storage. The server then uses the artwork data to add emotionally relevant messages and makes it shareable. Options to output the work as a photobook or calendar are also provided and are executed based on the user's choice.

[0751] For example, if an 8-year-old child is a little nervous at the start of an activity, the system suggests an activity where they draw animal illustrations in a gentle tone, accompanied by calming music. If the activity is progressing smoothly, based on the emotional data acquired by the device during the activity, the system displays a message on the screen with encouraging comments to boost their confidence. The completed artwork, along with the results of the emotional analysis, is shared in a way that facilitates emotional interaction among family members. In this way, the entire system understands the user's emotions and effectively supports the creative experience.

[0752] The following describes the processing flow.

[0753] Step 1:

[0754] The user launches the application on their device and enters their profile information (age, interests, etc.). The device's camera and microphone are activated, and it is ready to collect the user's emotional information (facial expressions and voice).

[0755] Step 2:

[0756] The device collects emotional data and sends it to the emotion engine in real time. The engine analyzes this data and identifies the user's emotional state. The emotional data is then prepared for transmission to the server.

[0757] Step 3:

[0758] The server receives the output of the emotion recognition engine and user profile information. Based on this data, the AI ​​model prepares to suggest creative activities that match the user's emotions. The specific activity themes and difficulty levels are then determined.

[0759] Step 4:

[0760] The server sends the selected activity to the terminal along with a detailed guide. The guide is tailored to include advice and encouraging messages that take the user's emotions into consideration.

[0761] Step 5:

[0762] The device displays the user with the activity details and guidance it has received. The user follows these instructions and begins their creative activity. During the activity, music and hints are provided to help with relaxation and concentration.

[0763] Step 6:

[0764] As the user progresses through the activity, the device continuously acquires emotional data and sends feedback to the server. Based on the emotional data, the server dynamically updates the guide content and sends it to the device as needed.

[0765] Step 7:

[0766] Once the user completes their creative activity, they take a picture of the finished work with their device's camera and save it as image data. The device then prepares to send this data to the server.

[0767] Step 8:

[0768] The device sends the artwork data to a server, which then manages the data in cloud storage. The server adds emotional messages to the artwork and organizes it in a shareable format.

[0769] Step 9:

[0770] The server provides the terminal with options to create photo books and calendars. The user selects and executes the option to save the work in the desired format. The server confirms the order and prepares it for printing and shipping.

[0771] (Example 2)

[0772] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0773] Existing creative activity support systems have the problem of being prone to a uniform experience because it is difficult to adjust individual suggestions and guides according to the user's emotional state. In addition, there is a lack of means to effectively share the results of creative activities, which means that the deliverables obtained from users' creative activities are not being utilized to their fullest potential.

[0774] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0775] In this invention, the server includes means for analyzing the user's emotional state and suggesting creative activities accordingly, means for dynamically adjusting and providing detailed information according to the suggested activities, and means for continuously monitoring the user's emotional data during creative activities. This enables personalized creative activity suggestions tailored to the user's emotions and effective sharing of creative works.

[0776] A "user terminal" is an electronic device that a user directly operates to input information and display results.

[0777] "Emotional state" refers to the psychological or emotional state a user is experiencing at a particular moment, and is inferred from their facial expressions and voice.

[0778] "Suggested means" refers to a method or apparatus that has the function of selecting and presenting appropriate creative activities based on the user's emotional state.

[0779] A "guiding device" is a method or apparatus that provides detailed instructions and information to assist in the progress of a proposed activity, thereby enriching the user's experience.

[0780] "Recording means" refers to a method or apparatus for digitizing and securely storing data of a work obtained as a result of creative activity.

[0781] "Means of sharing" refers to a method or device that provides recorded artwork data in a format accessible to others, enabling interaction and evaluation.

[0782] "Cloud storage" refers to a remote data storage service that stores data via the internet and allows access from anywhere.

[0783] "Printed material" refers to the process or result of outputting digital data onto a physical medium, making it visually verifiable.

[0784] This invention is a system that analyzes a user's emotional state, suggests creative activities, and enables the recording and sharing of creative data obtained through those activities. The system consists of a user terminal, a server, and an emotion recognition engine.

[0785] First, the user launches an application installed on their device and begins their creative activity. During this process, the device uses its camera and microphone to record the user's face and voice, and to acquire emotional data. This process utilizes OpenCV and PyTorch for facial recognition technology, and natural language processing software for voice data analysis.

[0786] The device transmits this acquired data in real time to the emotion recognition engine on the server. The emotion recognition engine uses a machine learning model to analyze and determine the user's current emotional state. For this purpose, natural language processing models such as Hugging Face are sometimes employed.

[0787] The server uses a generative AI model to suggest optimal creative activities based on the analyzed emotional state and user profile (age, interests, etc.). For example, OpenAI models are used as this generative AI model. These suggestions include themes and difficulty levels based on the user's emotional state.

[0788] The device displays creative activity information and detailed guides received from the server to the user. These guides are dynamically adjusted according to the user's emotions and include activity steps and encouraging messages. The user can then proceed with their creative activity by following the device's instructions.

[0789] As a concrete example, for a child who is a little nervous at the start of an activity, the system suggests an activity where they draw animal illustrations accompanied by calming music. If the activity is progressing smoothly, based on the emotional data acquired by the device during the activity, comments that boost confidence will be displayed on the screen. In this way, the system supports the creative experience while providing feedback on the user's emotions.

[0790] Example prompt: "An 8-year-old child is feeling a little anxious. Please suggest some calming activities. Also, please provide a detailed guide of activities to help understand and support their emotions."

[0791] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0792] Step 1:

[0793] The user launches an application installed on their device to begin their creative activity. The device uses its camera and microphone to acquire data on the user's face and voice. This input data includes still images and audio. Specifically, OpenCV is used to capture facial expressions, and the Google Speech-to-Text API is used to convert speech to text. This data serves as foundational data for inferring the user's emotional state.

[0794] Step 2:

[0795] The device sends the acquired emotional data to the emotion recognition engine on the server. The data received as input is analyzed using machine learning algorithms. For example, the Hugging Face Transformer model is applied to classify joy and sadness from facial expressions. As a result of this process, the user's emotional state is output as numerical or categorical data.

[0796] Step 3:

[0797] The server receives the analyzed emotional state and user-registered information such as age and interests (profile data) as input. Using this information, it proposes the most suitable creative activity using a generative AI model. This process uses an OpenAI model to generate prompts and select activities that match the user's emotions. The output includes the title, theme, and difficulty level of the proposed activity.

[0798] Step 4:

[0799] The terminal presents the user with activity suggestions sent from the server. Based on this, the terminal displays detailed guides and procedures in the user interface. These guides are dynamically adjusted to the user's emotions, and messages such as "Please proceed with confidence" may be displayed.

[0800] Step 5:

[0801] The user follows the instructions on the device to create their artwork. During the creation process, the device continuously collects the user's emotional data using the camera and microphone. This data is sent to the server as feedback, and the guide content is adjusted as needed.

[0802] Step 6:

[0803] Users photograph their completed artwork with their device and save it as digital data. This input data is recorded as an image file of the user's artwork. This ensures that the data is securely stored in cloud storage, thus preserving its integrity.

[0804] Step 7:

[0805] The server generates emotion-related messages based on saved artwork data and makes them shareable. This process uses a generative AI model to create messages and offers options to facilitate emotional interaction between users. Output options include the ability to convert the data into printed materials such as photo books and calendars.

[0806] (Application Example 2)

[0807] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0808] In modern family environments, there is a lack of means to encourage children to understand their own emotions and participate in rich creative activities. Furthermore, there is a need for systems that provide appropriate activities based on children's emotions and effectively share the results.

[0809] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0810] In this invention, the server includes an analysis means for acquiring emotional data and analyzing the user's emotional state, a suggestion means for proposing creative activities suitable for the user's emotional state based on the analysis results, and an adjustment means for continuously acquiring emotional data during the creative activity and adjusting the details of the activity as needed. This makes it possible to propose and support creative activities in accordance with the user's emotional state.

[0811] "Emotional data" refers to fundamental data used to analyze a user's emotional state, based on information obtained from the user's face and voice.

[0812] "Analysis means" refers to a means that analyzes emotional data acquired by a terminal and has the function of identifying the user's emotional state.

[0813] A "suggested means" is a means that, based on the analysis results obtained by the analysis means, presents creative activities that are appropriate to the user's emotional state.

[0814] A "guidance tool" is a tool that provides detailed information about a proposed activity and also dynamically presents encouragement and warnings tailored to the user's emotional state.

[0815] A "modification mechanism" is a means that continuously acquires emotional data during creative activity and dynamically adjusts the detailed information of the activity.

[0816] A "recording device" is a means that has the function of saving the data of a work obtained as a result of creative activity in a digital format.

[0817] A "sharing method" is a means that has the function of providing digitized artwork data, obtained through recording methods, in a form that can be shared with others.

[0818] The system for implementing this invention consists of several components for analyzing emotional data and providing creative activities tailored to the user. The main components include a terminal, a server, an emotional analysis engine, and cloud storage.

[0819] The device is installed on the user and uses a camera and microphone to acquire emotional data such as facial and voice data in real time. This data is sent to an emotion analysis engine. The emotion analysis engine processes the acquired data and identifies the user's emotional state. Software such as OpenCV and NVIDIA's DeepStream can be used for emotion analysis.

[0820] The server receives information about the user's emotional state from the emotion analysis engine and uses a generative AI model to suggest creative activities appropriate to that emotional state. The suggested activities are customized according to the user's age and interests and sent to the device. The device displays a detailed guide to the suggested activities on the screen and dynamically provides encouragement and warnings according to the user's emotional state.

[0821] During the activity, the device continuously acquires emotional data and sends feedback to the server as needed to modify and adjust the activity. This allows the user to enjoy an experience tailored to their state at any given time.

[0822] Once the creative activity is complete, the user records the result digitally and saves it to cloud storage. The server can then use this data to add emotionally relevant messages and output it in a shareable format. Options for outputting it as a photo book or calendar are also provided.

[0823] For example, if an 8-year-old child is a little nervous when starting an activity, the device will play relaxing music and suggest drawing an animal. If the activity goes well, the device will display an encouraging message such as, "Great job! Let's move on to the next step."

[0824] Another example of a prompt for a generative AI model is the text, "Please suggest a method for drawing pictures that children enjoy."

[0825] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0826] Step 1:

[0827] The device uses its camera and microphone to capture the user's facial image and voice. This acquired data is used as input data to analyze the user's emotional state.

[0828] Step 2:

[0829] The emotion analysis engine on the server, upon receiving emotion data transmitted from the terminal, performs image processing and voice analysis to identify the user's emotional state. This involves analyzing facial expressions using image processing libraries such as OpenCV, and analyzing voice tone and pitch. The output provides information about the user's emotional state.

[0830] Step 3:

[0831] The server inputs information about the user's emotional state into a generating AI model and suggests activities appropriate to the user's emotional state. The AI ​​model uses prompts to generate creative activities tailored to the user. The suggested activities include themes and difficulty levels that are appropriate to the user's characteristics.

[0832] Step 4:

[0833] The device displays suggested activities to the user and provides detailed guidance on their content. It offers encouragement and warnings tailored to the user's emotional state to ensure they can proceed with the activities with confidence. During this process, feedback is provided to the user through on-screen displays and audio.

[0834] Step 5:

[0835] While the user is engaged in creative activities, the device periodically retrieves emotional data and feeds it back to the server. Based on this feedback, the server dynamically adjusts the activity guide and sends it back to the device. This enables advice tailored to the user's progress.

[0836] Step 6:

[0837] Once the activity is complete, the user digitizes the output and sends it to cloud storage via their device. The server generates an emotionally relevant message based on the artwork data and provides output options such as photo books and calendars. The output is presented in a shareable format, making it easy to share with family and friends.

[0838] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the controlled object 443 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0839] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0840] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the robot 414.

[0841] Furthermore, the emotion identification model 59, acting as an emotion engine, may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to a specific mapping, which is an emotion map (see Figure 9). Similarly, the emotion identification model 59 may also determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[0842] Figure 9 shows an emotion map 400 in which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. The closer to the center of the concentric circles, the more primitive the emotions are located. Further out of the concentric circles, emotions representing states and actions arising from mental states are located. Emotion is a concept that includes feelings and mental states. On the left side of the concentric circles, emotions that are generally generated from reactions occurring in the brain are located. On the right side of the concentric circles, emotions that are generally induced by situational judgment are located. Above and below the concentric circles, emotions that are generally generated from reactions occurring in the brain and induced by situational judgment are located. In addition, the emotion of "pleasure" is located on the upper side of the concentric circles, and the emotion of "displeasure" is located on the lower side. Thus, in the emotion map 400, multiple emotions are mapped based on the structure in which emotions arise, and emotions that are likely to occur simultaneously are mapped close together.

[0843] These emotions are distributed at the 3 o'clock position on the Emotion Map 400, and usually fluctuate between feelings of security and anxiety. In the right half of the Emotion Map 400, situational awareness takes precedence over internal feelings, resulting in a calm impression.

[0844] The inside of the Emotion Map 400 represents inner thoughts, while the outside represents actions. Therefore, the further you go from the outside of the Emotion Map 400, the more visible (expressed in actions) your emotions become.

[0845] Here, human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. Similarly, in robots, cars, motorcycles, etc., emotions can be created based on various balances, such as posture and battery level. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. The emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on a system for analyzing brain physiological signals of speech emotion recognition and emotion, Tokushima University, doctoral dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map contains emotions belonging to a region called "response," where sensation is dominant. The right half of the emotion map contains emotions belonging to a region called "situation," where situational awareness is dominant.

[0846] The emotion map defines two emotions that promote learning. One is the emotion around the middle of the negative "repentance" and "reflection" on the situation side. In other words, it is when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is the emotion around the positive "desire" on the reaction side. In other words, it is when the robot has positive feelings such as "I want more" or "I want to know more."

[0847] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​representing each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple training data sets, which are combinations of user input and emotion values ​​representing each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions located close together have similar values, as shown in the emotion map 900 in Figure 10. Figure 10 shows an example where multiple emotions such as "reassured," "calm," and "confident" have similar emotion values.

[0848] The above description primarily focuses on the functions of the data processing device 12 in relation to this disclosure. However, the system related to this disclosure is not necessarily implemented on a server. The system related to this disclosure may be implemented as a general information processing system. This disclosure may be implemented, for example, as a software program that runs on a personal computer or as an application that runs on a smartphone. The method related to this disclosure may be provided to users in SaaS (Software as a Service) format.

[0849] In the above embodiment, an example was given in which a specific process is performed by a single computer 22. However, the technology of this disclosure is not limited thereto, and a distributed processing of the specific process may be performed by multiple computers, including computer 22. For example, a data generation model 58 may be provided in an external device of the data processing device 12, and the external device may generate data according to the input data.

[0850] In the above embodiment, an example was given in which the specific processing program 56 is stored in the storage 32, but the technology of this disclosure is not limited thereto. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-temporary storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-temporary storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes specific processing according to the specific processing program 56.

[0851] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[0852] Furthermore, it is not necessary to store the entirety of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store the entirety of the specific processing program 56 in the storage 32; it is acceptable to store only a portion of the specific processing program 56.

[0853] The following types of processors can be used as hardware resources to perform specific processing. Examples of processors include a CPU, a general-purpose processor that functions as a hardware resource to perform specific processing by executing software, i.e., a program. Other examples of processors include dedicated electrical circuits, such as FPGAs (Field-Programmable Gate Arrays), PLDs (Programmable Logic Devices), or ASICs (Application Specific Integrated Circuits), which have circuit configurations specifically designed to perform specific processing. All of these processors have built-in or connected memory, and all of them perform specific processing by using memory.

[0854] The hardware resource that performs a specific process may consist of one of these various processors, or it may consist of a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Alternatively, the hardware resource that performs a specific process may consist of a single processor.

[0855] Examples of configurations using a single processor include, firstly, a configuration in which one or more CPUs and software are combined to form a single processor, and this processor functions as a hardware resource that performs a specific process. Secondly, there is a configuration using a processor that realizes the functions of the entire system, including multiple hardware resources that perform a specific process, on a single IC chip, as exemplified by SoCs (System-on-a-chip). In this way, a specific process is realized using one or more of the above types of processors as hardware resources.

[0856] Furthermore, the hardware structure of these various processors can more specifically utilize electrical circuits that combine circuit elements such as semiconductor devices. Also, the specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps can be deleted, new steps added, or the processing order rearranged, as long as it does not deviate from the main purpose.

[0857] The descriptions and illustrations presented above are detailed explanations of the technical aspects of this disclosure and are merely examples of the technical aspects. For example, the above descriptions of the structure, function, operation, and effect are examples of the structure, function, operation, and effect of the technical aspects of this disclosure. Therefore, it goes without saying that you may delete unnecessary parts, add new elements, or replace elements in the descriptions and illustrations presented above, as long as you do not deviate from the essence of the technical aspects of this disclosure. Furthermore, in order to avoid confusion and facilitate understanding of the technical aspects of this disclosure, explanations of common technical knowledge and the like that do not require special explanation to enable the implementation of the technical aspects of this disclosure have been omitted from the descriptions and illustrations presented above.

[0858] All documents, patent applications, and technical standards described herein are incorporated by reference to the same extent as if each individual document, patent application, and technical standard were specifically and individually noted to be incorporated by reference.

[0859] The following is further disclosed regarding the embodiments described above.

[0860] (Claim 1)

[0861] A proposal means that suggests creative activities in response to user requests based on information obtained from the user's terminal,

[0862] A guide means that provides detailed information regarding the activities proposed by the proposed means,

[0863] A recording method for digitizing and recording artwork data obtained as a result of creative activities,

[0864] A sharing means for managing the artwork data recorded by the aforementioned recording means and providing it in a form that can be shared with others,

[0865] A system that includes this.

[0866] (Claim 2)

[0867] The system according to claim 1, wherein the recording means includes means for saving artwork data to cloud storage.

[0868] (Claim 3)

[0869] The system according to claim 1, wherein the sharing means includes means for providing an option to output the saved artwork data as a photo book or calendar.

[0870] "Example 1"

[0871] (Claim 1)

[0872] A means of proposing creative activities suitable for the user's requests using a generative artificial intelligence model based on identification information obtained from the user's terminal,

[0873] A guide that provides a list of materials and specific procedures related to the proposed activity,

[0874] A means of converting the results of creative activities into image data and recording them,

[0875] A means for managing recorded image data and generating links that can be shared with others via a communication network,

[0876] A system that includes this.

[0877] (Claim 2)

[0878] The system according to claim 1, wherein the recording means has the function of storing image data in a remote storage area.

[0879] (Claim 3)

[0880] The system according to claim 1, wherein the sharing means has the function of providing an option to output the stored image data as an item.

[0881] "Application Example 1"

[0882] (Claim 1)

[0883] A proposal means that suggests creative activities in response to user requests based on information obtained from the user's terminal,

[0884] A guide means that provides detailed information regarding the activities proposed by the proposed means,

[0885] A recording method for digitizing and recording artwork data obtained as a result of creative activities,

[0886] A sharing means for managing the artwork data recorded by the aforementioned recording means and providing it in a form that can be shared with others,

[0887] A driving mechanism installed in a household mechanical device that supports the implementation of creative activities through interaction with the user,

[0888] A system that includes this.

[0889] (Claim 2)

[0890] The system according to claim 1, wherein the recording means includes means for saving artwork data to cloud storage.

[0891] (Claim 3)

[0892] The system according to claim 1, wherein the sharing means includes means for providing an option to output the saved artwork data as a photo book or calendar.

[0893] "Example 2 of combining an emotion engine"

[0894] (Claim 1)

[0895] A proposal means that analyzes the user's emotional state based on information obtained from the user's terminal and suggests creative activities accordingly,

[0896] A guide means that dynamically adjusts and provides detailed information about the activities proposed by the proposed means according to the user's emotions,

[0897] A means of continuously monitoring user emotional data during creative activities,

[0898] A recording method for digitizing and recording artwork data obtained as a result of creative activities,

[0899] A sharing means for managing the artwork data recorded by the aforementioned recording means and providing it in a form that can be shared with others,

[0900] A system that includes this.

[0901] (Claim 2)

[0902] The system according to claim 1, wherein the recording means includes means for saving artwork data to cloud storage.

[0903] (Claim 3)

[0904] The system according to claim 1, wherein the sharing means includes means for providing an option to output the saved artwork data as a printed material.

[0905] "Application example 2 when combining with an emotional engine"

[0906] (Claim 1)

[0907] An analytical means for acquiring emotional data and analyzing the user's emotional state,

[0908] Based on the analysis results, a proposal method is provided to suggest creative activities that are appropriate for the user's emotional state.

[0909] A guide means that provides detailed information about the activities proposed by the proposed means and instructs on encouragement and warnings according to the emotional state,

[0910] A means of continuously acquiring emotional data during creative activity and adjusting the details of the activity as needed,

[0911] A recording method for digitizing and recording artwork data obtained as a result of creative activities,

[0912] A sharing means for managing the artwork data recorded by the aforementioned recording means and providing it in a form that can be shared with others,

[0913] A system that includes this.

[0914] (Claim 2)

[0915] The system according to claim 1, wherein the recording means includes means for saving artwork data to cloud storage.

[0916] (Claim 3)

[0917] The system according to claim 1, wherein the sharing means includes means for providing an option to output the saved artwork data as a photo book or calendar by adding a message related to the emotional state. [Explanation of Symbols]

[0918] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Devices 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robots< / url:> < / url:> < / url:> < / url:>

Claims

1. A proposal means that suggests creative activities in response to user requests based on information obtained from the user's terminal, A guide means that provides detailed information regarding the activities proposed by the proposed means, A recording method for digitizing and recording artwork data obtained as a result of creative activities, A sharing means for managing the artwork data recorded by the aforementioned recording means and providing it in a form that can be shared with others, A system that includes this.

2. The system according to claim 1, wherein the recording means includes means for saving artwork data to cloud storage.

3. The system according to claim 1, wherein the sharing means includes means for providing an option to output the saved artwork data as a photo book or calendar.

Images