system

Abstract

We provide the system. [Solution] A means for raising virtual animals based on operations performed by individual users, provided in an information and communication device, A means for generating user operation information and estimating the user's activities by analyzing said operation information, A means of indirectly providing monitoring through an information processing device that transmits the analyzed information to a remote monitor, The information is communicated to the monitor using an existing communication interface. A system that includes this.

Description

Technical Field

[0001] The technology of the present disclosure relates to a system.

Background Art

[0002] Patent Document 1 discloses a method for controlling a persona chatbot, which is performed by at least one processor, including steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to an explanation of the chatbot's character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] Amid the deepening problem of the solitary death of the elderly, although there is a need for monitoring, there are many elderly people who are resistant to being monitored and fear the infringement of privacy. Therefore, there is a need for a means to monitor the elderly in a natural and non-resistant form, but this has not been fully solved by existing monitoring systems. To solve this problem, it is necessary to collect and provide information while considering privacy, while ensuring safety without giving users a sense of being monitored.

Means for Solving the Problems

[0005] This invention provides a system that offers a natural and non-resistant monitoring effect to the elderly through a virtual animal installed in an information and communication device. This system analyzes the operation information generated by the elderly person raising the virtual animal and indirectly estimates the elderly person's activities. Furthermore, it transmits this information to a remote monitor, enabling monitoring of the elderly person's situation via an existing communication interface. In addition, by providing an operation display device that provides visual and audible interaction between the user and the virtual animal, a natural monitoring environment is realized without causing resistance to the user.

[0006] An "information and communication device" is a device that allows users to perform operations to raise virtual animals and to generate and transmit information about those operations.

[0007] A "user" is an individual who uses information and communication equipment to raise a virtual animal, and who is the subject of monitoring by this system.

[0008] A "virtual animal" is a digital pet that is displayed on an information and communication device and whose state changes according to the user's actions.

[0009] "Operation information" refers to data related to a series of operations performed by the user on the virtual animal, including records of its growth status.

[0010] "Analysis" is the process of estimating activity trends and living conditions based on user operation information.

[0011] A "monitoring officer" is an individual or organization whose role is to indirectly understand the user's situation and ensure their safety.

[0012] A "communication interface" refers to an existing communication protocol or platform used to transmit information from an information and communication device to a monitor.

[0013] An "operation display device" is a device that has the function of providing visual and auditory feedback to enable users to intuitively interact with virtual animals. [Brief explanation of the drawing]

[0014] [Figure 1] This is a conceptual diagram showing an example of the configuration of a data processing system according to the first embodiment. [Figure 2] This is a conceptual diagram showing an example of the essential functions of a data processing device and a smart device according to the first embodiment. [Figure 3] This is a conceptual diagram showing an example of the configuration of a data processing system according to the second embodiment. [Figure 4] This is a conceptual diagram showing an example of the main functions of a data processing device and smart glasses according to the second embodiment. [Figure 5] This is a conceptual diagram showing an example of the configuration of a data processing system according to the third embodiment. [Figure 6] This is a conceptual diagram showing an example of the main functions of a data processing device and a headset-type terminal according to the third embodiment. [Figure 7] This is a conceptual diagram showing an example of the configuration of a data processing system according to the fourth embodiment. [Figure 8] This is a conceptual diagram showing an example of the main functions of a data processing device and a robot according to the fourth embodiment. [Figure 9] This shows an emotion map where multiple emotions are mapped. [Figure 10] This shows an emotion map where multiple emotions are mapped. [Figure 11] This is a sequence diagram showing the processing flow of the data processing system in Example 1. [Figure 12] This is a sequence diagram showing the processing flow of the data processing system in Application Example 1. [Figure 13] This is a sequence diagram showing the processing flow of the data processing system in Example 2, which incorporates an emotion engine. [Figure 14]A sequence diagram showing the processing flow of a data processing system in Application Example 2 when combined with an emotion engine.

Embodiments for Carrying Out the Invention

[0015] Hereinafter, an example of an embodiment of a system according to the technology of the present disclosure will be described with reference to the accompanying drawings.

[0016] First, the terms used in the following description will be explained.

[0017] In the following embodiments, a labeled processor (hereinafter simply referred to as "processor") may be a single arithmetic unit or a combination of multiple arithmetic units. Also, the processor may be a single type of arithmetic unit or a combination of multiple types of arithmetic units. Examples of arithmetic units include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), an APU (Accelerated Processing Unit), etc.

[0018] In the following embodiments, a labeled RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a work memory by the processor.

[0019] In the following embodiments, a labeled storage is one or more non-volatile storage devices that store various programs and various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), or magnetic tapes, etc.

[0020] In the following embodiments, the signed communication interface (I / F) is an interface that includes a communication processor and an antenna, etc. The communication interface manages communication between multiple computers. Examples of communication standards applicable to the communication interface include wireless communication standards such as 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), or Bluetooth (registered trademark).

[0021] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." That is, "A and / or B" means that it may be A alone, or B alone, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" applies when expressing three or more things linked by "and / or."

[0022] [First Embodiment]

[0023] Figure 1 shows an example of the configuration of the data processing system 10 according to the first embodiment.

[0024] As shown in Figure 1, the data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0025] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0026] The smart device 14 comprises a computer 36, a reception device 38, an output device 40, a camera 42, and a communication interface 44. The computer 36 comprises a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The reception device 38, output device 40, and camera 42 are also connected to the bus 52.

[0027] The reception device 38 is equipped with a touch panel 38A and a microphone 38B, etc., and receives user input. The touch panel 38A receives user input by detecting contact with an object (e.g., a pen or finger). The microphone 38B receives user input by detecting the user's voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0028] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form perceptible to the user 20 (e.g., audio and / or text). The display 40A displays visible information such as text and images according to instructions from the processor 46. The speaker 40B outputs audio according to instructions from the processor 46. The camera 42 is a small digital camera equipped with an optical system such as a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0029] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various types of information between processor 46 and processor 28 via network 54.

[0030] Figure 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0031] As shown in Figure 2, in the data processing device 12, a specific processing is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" related to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0032] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0033] In the smart device 14, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The reception output program 60 is used in conjunction with a specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0034] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0035] The system based on this invention provides a virtual animal raising function that allows users to be naturally monitored. The specific program processing and operation of each element—server, terminal, and user—are described below.

[0036] Server-side embodiment

[0037] The server receives and analyzes operational information sent by the user. This analysis utilizes an AI model to estimate the user's activity level and identify behavioral patterns specific to the elderly. Based on the analysis results, it generates and sends notifications to the supervisor as needed.

[0038] Specific example: The server detects the possibility that a user is not living a normal life based on information such as "the frequency of feeding virtual animals has decreased significantly," and notifies the monitor.

[0039] Terminal-side embodiment

[0040] The device provides a user interface for users to intuitively interact with virtual animals. This includes features that allow the virtual animals to perform actions in response to taps and voice commands. Interaction data is sent to the server at regular intervals.

[0041] Specific example: On the device screen, when the user presses the "feed" button, an animation of the virtual animal eating food is displayed, showing a satisfied expression.

[0042] User-side embodiment

[0043] Users raise virtual animals through an app installed on their smartphones or tablets. Caring for the virtual animals is integrated into the user's daily activities, allowing for natural interaction as part of their life. As a result, relevant information is accumulated, and the user's status is indirectly communicated to a supervisor.

[0044] Specific example: A user can habitually feed a virtual animal every morning and play with their pet during the day, ensuring their security without realizing that their normal activities are being transmitted to the server as data.

[0045] Thus, this invention makes it possible to provide natural monitoring without causing stress to users and to improve the quality of life for the elderly. Through the collaboration of the server, terminal, and user, it provides a continuous monitoring environment that is not resistant to the user.

[0046] The following describes the processing flow.

[0047] Step 1:

[0048] The user launches a virtual animal app on their information and communication terminal. The user checks the virtual animal's status and begins interacting with it, such as feeding it or selecting toys to play with.

[0049] Step 2:

[0050] The device detects user actions in real time and records those actions as operation information. Specifically, it collects data such as operation type, operation time, and duration.

[0051] Step 3:

[0052] The terminal is set to send collected operation information to the server at regular intervals, and then transmits the information. During this process, the data is encrypted according to a security protocol to ensure secure communication.

[0053] Step 4:

[0054] The server stores the received operation information and performs analysis using an AI model. In this analysis process, user activity patterns and unique behaviors are estimated based on the operation information.

[0055] Step 5:

[0056] Based on the analysis results, the server generates necessary monitoring information tailored to the user's situation. For example, if an anomaly is detected, such as "the virtual animal has not been fed for several days," it generates a notification indicating that attention is needed.

[0057] Step 6:

[0058] The server sends the generated monitoring information to the monitor via an existing communication interface. Typically, the monitor is a family member or caregiver of the user. The notification content is provided in the form of suggestions for appropriate actions based on the analysis results.

[0059] Step 7:

[0060] The monitor reviews notifications received from the server and considers appropriate actions based on the user's situation. Based on this, they decide whether to contact the user directly or provide additional support if necessary.

[0061] (Example 1)

[0062] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0063] In modern society, ensuring the safety of the elderly is a crucial issue. However, direct surveillance can infringe on privacy and cause stress. In particular, there is a need for new methods to naturally observe daily behavior and detect abnormalities early.

[0064] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0065] In this invention, the server includes means for managing a virtual pet based on user operations, means for generating operation data, analyzing it using a generation AI model, and estimating the user's behavioral state, and means for providing a method for generating and transmitting notifications to a remote monitor based on the analysis. This makes it possible to monitor the user naturally without causing them stress.

[0066] An "information processing device" is an electronic device used to receive, process, and generate data, and to monitor for anomalies.

[0067] "User" refers to an individual who manages and operates a virtual pet.

[0068] "Operation data" refers to the record of actions performed by the user on the virtual pet.

[0069] A "generative AI model" refers to artificial intelligence technology used to analyze user behavior data and estimate user behavior patterns.

[0070] "Behavioral status" refers to information indicating the user's daily activity patterns and how they change.

[0071] A "monitoring officer" refers to an individual or group whose role is to receive notifications from the server in order to ensure the safety of users.

[0072] A "virtual pet" refers to a digitized animal character that a user raises and interacts with within an information processing device.

[0073] In this invention, a server, terminal, and user work together to provide a natural monitoring system using a virtual pet. The server receives operation data transmitted from the terminal and analyzes this data using a generated AI model. This makes it possible to estimate the user's behavior and identify behavioral patterns specific to the elderly. Based on the analysis results, the server is responsible for generating and sending notifications to remote monitors as needed.

[0074] The device provides a user interface that allows users to intuitively interact with virtual pets. This includes visual and auditory feedback, with the virtual pet responding to taps and voice commands. The interface can also provide feedback through animations and sounds as the user feeds or plays with the virtual pet.

[0075] Users raise virtual pets through applications installed on devices such as smartphones and tablets, and interact with them as part of their natural daily activities. This ensures safety, as abnormal behavior can be detected without the user noticing.

[0076] As a concrete example, an example of a prompt message is shown below.

[0077] "Please explain in detail how to build an AI model to evaluate safety based on virtual pet raising data from elderly individuals."

[0078] Thus, the system of the present invention can improve the quality of life for the elderly by providing natural monitoring without causing stress to the user.

[0079] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0080] Step 1:

[0081] The server receives operation data sent from the terminal. The input includes a record of the specific interactions the user has had with the virtual pet. This data includes, for example, the time and frequency of feeding, and the content of play. The output is this raw data, ready for use in the next analysis step.

[0082] Step 2:

[0083] The server analyzes the received operation data. This analysis uses a generative AI model to estimate the user's behavior patterns based on the input data. Specific data processing includes processing time-series data and detecting outliers. The output is the estimated results of the user's behavior state and patterns.

[0084] Step 3:

[0085] If the server detects an anomaly based on the analysis results, it generates and sends a notification to a remote monitor. This notification may include a report detailing specific user behavioral abnormalities and suggestions for necessary countermeasures. The input is the analysis results, and the output is the notification data sent to the monitor.

[0086] Step 4:

[0087] The device provides a user interface for users to interact with virtual pets. This interface receives input and outputs visual and auditory feedback. For example, when the user presses the "feed" button, the virtual pet displays an animation of satisfaction.

[0088] Step 5:

[0089] Users raise virtual pets using an application on their devices. The user's input actions, such as "feeding in the morning and playing at noon," are sent as data to the server via the device. This generates real-world activity data, which is used in subsequent analysis steps. The output is an operation history sent to the server.

[0090] (Application Example 1)

[0091] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0092] To improve the customer experience in commercial spaces and enhance visitor convenience within facilities, it is crucial that visitors enjoy their visit while ensuring a safe and comfortable stay. However, simply sending notifications to smartphones or using conventional marketing methods is insufficient to retain visitors' continued interest. Furthermore, current customer management systems are often mechanical and struggle to enrich the individual visitor experience.

[0093] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0094] In this invention, the server includes means for cultivating a virtual organism based on operations performed by individual users on an information processing device, means for generating user operation information and estimating the user's activities by analyzing said operation information, and means for recording the user's visit status in a commercial space and for the virtual organism to react according to the user's current location. This makes it possible to enrich individual experiences while visitors enjoy exploring the commercial space in a natural way, and to provide visitors with a meaningful and safe store environment.

[0095] An "information processing device" is a device that generates user operation information, analyzes that information, and has the function of estimating activity.

[0096] A "virtual organism" is a digital representation that is cultivated on an information processing device and exhibits reactions and growth in response to user input.

[0097] A "processing unit" is a device that has the function of processing analyzed information and transmitting it to a remote monitor.

[0098] A "communication interface" is an existing connection method that transmits and receives data between an information processing device and a monitor.

[0099] A "commercial space" is an area within a store or facility that is visited by users, and it is an environment for recording and analyzing the behavior of visitors.

[0100] This invention is a system for improving the user experience in commercial spaces, allowing users to explore the space while raising virtual creatures. The specific program processing and operation of each element—server, terminal, and user—are described below.

[0101] Server-side embodiment

[0102] The server receives and analyzes operational information transmitted from the user's smartphone. A generative AI model is used for analysis to estimate the user's behavior within the commercial space. Furthermore, it records visitor dwell time data at specific points within the commercial space and instructs a virtual organism to respond in real time based on the user's location. This process utilizes a web server based on the Django framework and a PostgreSQL database.

[0103] Terminal-side embodiment

[0104] The device, in this case the user's smartphone, provides a user interface for intuitive interaction with virtual creatures. Specifically, it has the functionality to make the virtual creatures act in response to screen taps and voice commands. This allows users to interact with virtual creatures while enjoying the commercial space, acquire items in response to specific actions, and participate in special events.

[0105] User-side embodiment

[0106] Users raise virtual creatures through an application installed on their smartphones. As users explore commercial spaces, they can enjoy the reactions of their virtual creatures, experiencing a contactless customer experience in the process. For example, when a user reaches a specific store section, the virtual creature may perform a new dance, which can then earn them a reward.

[0107] This invention utilizes generative AI models to provide users with unexpected and engaging experiences, thereby enhancing user engagement within commercial spaces.

[0108] Specific example:

[0109] "When users reach the 'premium section' within the commercial space, virtual creatures will perform a 'dance of joy,' and users can obtain special items."

[0110] Example of a prompt:

[0111] "Please suggest ways to enrich the expressions of joy when customers interact with virtual animals in the store."

[0112] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0113] Step 1:

[0114] The terminal collects information about the user's interactions with the virtual creature. Input consists of taps and voice commands from the user, and output consists of data packets that organize these operations as numerical and string data and send them to the server. At this stage, the virtual creature's action animation is executed on the terminal.

[0115] Step 2:

[0116] The server receives operation information transmitted from the terminal. The input is data packets sent from the terminal, and the output is the analysis result. A generative AI model is used to analyze the user's operation information and estimate their behavior within the commercial space. The data is stored in a database as a user behavior log.

[0117] Step 3:

[0118] The server determines the actions of virtual creatures based on specific points within a commercial space, using analyzed user behavior data. The input is behavior log data, and the output is virtual creature response data. Based on the location information, a generative AI model calculates the appropriate response for the virtual creature.

[0119] Step 4:

[0120] The server sends the analyzed virtual creature's responses to the terminal. The input is the generated response data, and the output is the virtual creature's actions displayed on the terminal. The server responds to the terminal in real time, instructing the virtual creature to perform actions.

[0121] Step 5:

[0122] Users explore a commercial space while observing the reactions of virtual creatures. Input is action information from the server, and output is the user's observations. Users can acquire items and rewards. This makes visiting the commercial space more appealing.

[0123] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0124] This invention combines an emotion engine with a virtual animal raising system to achieve multifaceted monitoring that includes the user's emotions. The specific processing and operation of the programs in the server, terminal, and user elements are described below.

[0125] Server-side embodiment

[0126] The server receives operation information transmitted from the terminal and emotional data from the emotion engine. The server uses an AI model to comprehensively analyze this data. Through this analysis, it estimates the user's behavioral patterns and emotional state, and notifies the supervisor as needed.

[0127] Specific example: The server determines that the user is not showing much interest in the virtual animal and generates a notification stating "emotional state is low," which is then sent to the monitor.

[0128] Terminal-side embodiment

[0129] The terminal is equipped with an operation display device as an information and communication device, providing visual and auditory feedback for the user to interact with the virtual animal. An emotion engine analyzes the user's voice tone and facial expressions, and transmits the resulting emotion data to the server along with the operation information.

[0130] Specific example: If the device detects that the user's voice is cheerful, the virtual animal will move more actively and display fun animations.

[0131] User-side embodiment

[0132] Users perform the usual virtual animal raising operations through an app that incorporates an emotion engine. During this process, the user's emotional state is naturally analyzed, and the virtual animal's reactions are adjusted accordingly, resulting in a more personalized raising experience.

[0133] Specific example: If the user makes a sad sound while operating the system, the virtual animal will display a message on the screen saying, "Cheer up."

[0134] In this way, the system of the present invention can provide meticulous monitoring that takes into account the user's emotions and support that takes into account the user's psychological and emotional health. The server, terminal, and user all work together to provide a monitoring environment optimized for each individual.

[0135] The following describes the processing flow.

[0136] Step 1:

[0137] The user launches an application on their device. The user interacts with a virtual animal, performing everyday actions such as feeding and playing with it.

[0138] Step 2:

[0139] The device records user interactions in real time and saves data such as the type of operation and time. Simultaneously, the emotion engine analyzes the user's voice and facial expressions to generate data that estimates their emotional state.

[0140] Step 3:

[0141] The device collects operational information and emotional data, which is then sent to a server at regular time intervals or triggered by an event. A secure protocol with data encryption is used for this purpose.

[0142] Step 4:

[0143] The server analyzes the received data and uses an AI model to estimate the user's activity patterns and emotional trends. Based on these estimations, it determines the user's current situation.

[0144] Step 5:

[0145] The server generates and sends a notification to the monitor when it detects a condition requiring special attention (for example, if a user exhibits negative emotions for an extended period). This notification includes the estimated emotional state and recommended actions.

[0146] Step 6:

[0147] The monitor receives notifications from the server and considers appropriate actions based on the user's situation. If necessary, they contact the user directly and provide support.

[0148] Step 7:

[0149] Upon receiving feedback again from the server, the device will display appropriate actions and messages to the user, thereby promoting emotional support for the user.

[0150] Following this process, this system, which includes an emotion engine, will consider and monitor the user's emotions in real time.

[0151] (Example 2)

[0152] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0153] Conventional electronic entity nurturing systems, while progressing based on user input, have the challenge of failing to consider the user's emotional state and providing personalized support. In particular, they are unable to respond appropriately to changes in the user's emotions, resulting in a lack of effective monitoring and support.

[0154] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0155] In this invention, the server includes means for nurturing an electronic entity based on operations performed by an individual user, means for acquiring the user's voice and video and generating emotional information using an emotion analysis device, and means for estimating the user's behavioral patterns and emotional state by analyzing the generated emotional information and the operation information. This enables highly personalized monitoring and support based on the user's emotions.

[0156] An "information processing device" is a general term for electronic devices designed to input, analyze, process, and output data.

[0157] An "electronic entity" refers to a virtual character or object that can interact with the user within a digital environment.

[0158] "User" refers to an individual who operates the system and interacts with electronic entities.

[0159] "Emotional analysis" refers to a technology that identifies or estimates a user's emotional state based on data such as voice tone and facial expressions.

[0160] "Communication structure" is a general term for networks and communication protocols used to transmit data to remote devices or monitors.

[0161] "Behavioral patterns" refer to the tendencies of a user's actions and choices under specific conditions, and based on these, it is possible to predict the user's future behavior.

[0162] "Monitoring" means providing safety and security by monitoring the user's condition and activities, and providing support and notifications as needed.

[0163] This invention is a system that utilizes an information processing device to cultivate an electronic entity based on user operations. The system analyzes the user's emotions and provides appropriate feedback accordingly.

[0164] The server receives operation information sent from the terminal and sentiment data from the sentiment engine. The server uses a generative AI model to comprehensively analyze this data. During the analysis process, it is possible to use data analysis tools such as Python's Pandas and Scikit-learn. Based on the analysis results, the server estimates the user's behavior patterns and emotional state and notifies the monitor as needed. This is done, for example, using a prompt message such as, "Analyze the user's psychological tendencies based on their behavior patterns and sentiment data, and generate a warning if attention is needed."

[0165] The device provides visual and auditory feedback to support user interaction with the electronic entity. For voice tone and facial expression analysis, it can utilize facial recognition software such as OpenCV and the Google® Cloud Speech-to-Text API for speech analysis. The device sends the acquired emotion data to the server as user interaction information. Furthermore, if the user expresses positive emotions, it provides visual effects such as playing an animated sequence in which the electronic entity moves actively.

[0166] Users nurture electronic beings through applications that incorporate an emotion engine. The user's emotional state is analyzed in a natural way, and the electronic being's responses are adjusted accordingly. For example, if the user interacts with the electronic being in a sad voice, it will display "Cheer up" and provide encouraging feedback. In this case, a prompt message such as "If a sad tone of voice is detected, display an encouraging message." is used.

[0167] Through this system, we can provide a personalized monitoring environment based on the individual user's emotions, resulting in a more nurturing and ultimately beneficial parenting experience for the user.

[0168] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0169] Step 1:

[0170] The user initiates interaction with the electronic entity through the device. In this step, the user's voice and video are input via the device's microphone and camera. The device collects this input data and prepares to send it to emotion analysis software. Specifically, when the user speaks to the electronic entity, their voice and facial expressions are captured in real time.

[0171] Step 2:

[0172] The device sends the collected audio and video data to the emotion analysis engine. The emotion analysis engine uses the Google Cloud Speech-to-Text API to perform audio analysis, converting the user's voice tone into text. It also uses OpenCV to analyze the user's facial expressions from the video. This data processing provides an indicator of the user's emotional state. The output is analyzed emotion data. Specifically, if the user is smiling, data indicating that a smile has been detected will be output.

[0173] Step 3:

[0174] The device sends the generated emotion data and the user's current action information to the server. This action information includes specific choices and actions the user has made within the app. As a specific action that the server takes to prepare to receive this input data and save it to its database, the device records the action "the user fed the pet" and sends it along with the emotion data.

[0175] Step 4:

[0176] The server combines emotional data and operational information received from the terminal to perform a comprehensive analysis. This analysis utilizes a generative AI model, using Python's Pandas and Scikit-learn to analyze data trends. By combining the input data, the user's behavior patterns and emotional state are estimated. The output is a prediction of the user's state. For example, the server might predict that "the user is likely excited at the moment."

[0177] Step 5:

[0178] The server determines specific actions based on the analysis results and sends notifications to the monitor as needed. The generated notification includes a prompt that clearly communicates the user's state to the monitor. Specifically, it executes the action, "The user's emotional state is elevated; please notify me by email if attention is required."

[0179] Step 6:

[0180] The device receives instructions from the server and provides feedback to the user. If the user expresses positive emotions, it plays visual animations, such as the electronic entity beginning to dance. Specifically, cheerful music plays and the electronic entity moves around energetically on the screen. This feedback allows the user to experience interactions that match their emotions.

[0181] (Application Example 2)

[0182] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as a "server" and the smart device 14 as a "terminal".

[0183] In modern times, virtual organism breeding systems utilizing information and communication devices are widespread, but many systems rely solely on user operation information, lacking sufficient monitoring and support that takes emotional states into account. Furthermore, there is a growing need to provide personalized user experiences by delivering content tailored to the user's emotions. This invention aims to provide a personalized monitoring environment and emotionally responsive support by analyzing the user's emotional state in real time and providing feedback and content based on that analysis.

[0184] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0185] In this invention, the server includes means for raising a virtual organism based on operations by an individual user, means for estimating the user's activity and emotional state using the user's operation information and emotion analysis functions, and means for indirectly providing monitoring and emotion-based content via an information communication device that transmits the analyzed information to a remote monitor. This enables a monitoring and entertainment experience optimized for each individual user by providing feedback and content suggestions that respond to the user's emotions.

[0186] An "information processing device" is a general term for devices that input, process, and output data, and within a system, it is a device that is responsible for analyzing user operation information and emotional data.

[0187] A "user" refers to an individual who uses this system to raise a virtual creature, and is the subject of emotional analysis.

[0188] A "virtual creature" is a digital pet or character that is artificially generated on a computer, and whose actions and reactions change based on the user's input.

[0189] The "emotion analysis function" is the ability to automatically estimate the user's emotional state using AI technology based on their facial expressions and voice data.

[0190] A "remote monitor" is someone who is physically located away from the user but whose role is to understand the user's status through an information processing device and to provide support and monitoring.

[0191] A "communication interface" is a physical or logical connection means that allows an information processing device to exchange information with other devices or networks.

[0192] "Monitoring" refers to actions or systems that monitor the user's situation and condition and provide appropriate support to maintain their safety and health.

[0193] "Content recommendation" is the act of recommending the most suitable information and entertainment resources based on the user's emotions and preferences.

[0194] The server operates on an information processing device and integrates and analyzes user operation information and emotional data. This system receives operation information and emotional analysis results transmitted from the user's smart device (such as a smartphone or tablet) and processes them using a cloud-based generative AI model. Specifically, it performs data calculations to estimate how the user interacts with the virtual creature and the user's emotional state.

[0195] The terminal serves as an interface for the user, providing visual and auditory interaction with the virtual being. Simultaneously, the terminal's built-in camera and microphone capture the user's facial expressions and voice tone, and this data is converted using emotion analysis. This data is sent to a server as input information for real-time estimation of the user's emotional state. When the user operates through the terminal, the system comprehensively analyzes this operation information and emotion data to provide situation-appropriate monitoring functions and content suggestions.

[0196] As users utilize this system on a daily basis, they can gain personalized experiences through their interactions with virtual beings. For example, if the system analyzes that a user is feeling stressed, relaxing music or videos will be suggested. This suggestion feature utilizes a generative AI model that predicts what kind of content the user is looking for based on their emotional state and makes the optimal selection.

[0197] An example of a prompt message might be, "Create an AI model that analyzes the user's facial expressions and voice to suggest relaxing content." This would enable more accurate content suggestions based on emotions, providing a more personalized entertainment experience.

[0198] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0199] Step 1:

[0200] The device collects user interaction and emotional data. Specifically, it captures the user's facial expressions with its built-in camera and records their voice tone with its microphone. This data is processed by an emotion analysis algorithm to generate information for estimating the user's emotional state. The inputs are camera footage and voice audio, and the output is the estimated emotional data.

[0201] Step 2:

[0202] The terminal transmits collected and analyzed operation information and sentiment data to the server. The transmitted data includes operation logs and sentiment analysis results related to user interactions. This process is a preparatory stage for comprehensive data analysis by the server. The input is locally analyzed operation information and sentiment data, and the output is data transfer to the server.

[0203] Step 3:

[0204] The server integrates the received data and uses a generative AI model to estimate the user's behavioral patterns and emotional states. Data processing includes time-series analysis to analyze temporal behavioral patterns and regression analysis to evaluate emotional states. Inputs are operation information and emotional data from the terminal, while output is estimated behavioral and emotional trend data.

[0205] Step 4:

[0206] The server selects the most suitable content using prompts based on estimated data. It retrieves content from the database that is appropriate for the user's emotional state, as predicted by the generative AI model, and generates suggestions. This step provides a personalized user experience by dynamically recommending content that matches the user's emotions. The input is behavioral and emotional trend data, and the output is content suggestion data.

[0207] Step 5:

[0208] Upon receiving suggestion data from the server, the terminal provides visual and auditory notifications to the user's screen. Specifically, it displays details and options for the suggested content and awaits user action. The input is the content suggestion data from the server, and the output is the content information displayed on the user interface.

[0209] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0210] Data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of data generation model 58 is ChatGPT (registered trademark) (Internet search).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0211] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart device 14.

[0212] [Second Embodiment]

[0213] Figure 3 shows an example of the configuration of the data processing system 210 according to the second embodiment.

[0214] As shown in Figure 3, the data processing system 210 includes a data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0215] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0216] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication interface 44. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, and camera 42 are also connected to the bus 52.

[0217] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0218] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0219] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0220] Figure 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Figure 4, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0221] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0222] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0223] In the smart glasses 214, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0224] Next, the identification processing performed by the identification processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0225] The system based on this invention provides a virtual animal raising function that allows users to be naturally monitored. The specific program processing and operation of each element—server, terminal, and user—are described below.

[0226] Server-side embodiment

[0227] The server receives and analyzes operational information sent by the user. This analysis utilizes an AI model to estimate the user's activity level and identify behavioral patterns specific to the elderly. Based on the analysis results, it generates and sends notifications to the supervisor as needed.

[0228] Specific example: The server detects the possibility that a user is not living a normal life based on information such as "the frequency of feeding virtual animals has decreased significantly," and notifies the monitor.

[0229] Terminal-side embodiment

[0230] The device provides a user interface for users to intuitively interact with virtual animals. This includes features that allow the virtual animals to perform actions in response to taps and voice commands. Interaction data is sent to the server at regular intervals.

[0231] Specific example: On the device screen, when the user presses the "feed" button, an animation of the virtual animal eating food is displayed, showing a satisfied expression.

[0232] User-side embodiment

[0233] Users raise virtual animals through an app installed on their smartphones or tablets. Caring for the virtual animals is integrated into the user's daily activities, allowing for natural interaction as part of their life. As a result, relevant information is accumulated, and the user's status is indirectly communicated to a supervisor.

[0234] Specific example: A user can habitually feed a virtual animal every morning and play with their pet during the day, ensuring their security without realizing that their normal activities are being transmitted to the server as data.

[0235] Thus, this invention makes it possible to provide natural monitoring without causing stress to users and to improve the quality of life for the elderly. Through the collaboration of the server, terminal, and user, it provides a continuous monitoring environment that is not resistant to the user.

[0236] The following describes the processing flow.

[0237] Step 1:

[0238] The user launches a virtual animal app on their information and communication terminal. The user checks the virtual animal's status and begins interacting with it, such as feeding it or selecting toys to play with.

[0239] Step 2:

[0240] The device detects user actions in real time and records those actions as operation information. Specifically, it collects data such as operation type, operation time, and duration.

[0241] Step 3:

[0242] The terminal is set to send collected operation information to the server at regular intervals, and then transmits the information. During this process, the data is encrypted according to a security protocol to ensure secure communication.

[0243] Step 4:

[0244] The server stores the received operation information and performs analysis using an AI model. In this analysis process, user activity patterns and unique behaviors are estimated based on the operation information.

[0245] Step 5:

[0246] Based on the analysis results, the server generates necessary monitoring information tailored to the user's situation. For example, if an anomaly is detected, such as "the virtual animal has not been fed for several days," it generates a notification indicating that attention is needed.

[0247] Step 6:

[0248] The server sends the generated monitoring information to the monitor via an existing communication interface. Typically, the monitor is a family member or caregiver of the user. The notification content is provided in the form of suggestions for appropriate actions based on the analysis results.

[0249] Step 7:

[0250] The monitor reviews notifications received from the server and considers appropriate actions based on the user's situation. Based on this, they decide whether to contact the user directly or provide additional support if necessary.

[0251] (Example 1)

[0252] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0253] In modern society, ensuring the safety of the elderly is a crucial issue. However, direct surveillance can infringe on privacy and cause stress. In particular, there is a need for new methods to naturally observe daily behavior and detect abnormalities early.

[0254] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0255] In this invention, the server includes means for managing a virtual pet based on user operations, means for generating operation data, analyzing it using a generation AI model, and estimating the user's behavioral state, and means for providing a method for generating and transmitting notifications to a remote monitor based on the analysis. This makes it possible to monitor the user naturally without causing them stress.

[0256] An "information processing device" is an electronic device used to receive, process, and generate data, and to monitor for anomalies.

[0257] "User" refers to an individual who manages and operates a virtual pet.

[0258] "Operation data" refers to the record of actions performed by the user on the virtual pet.

[0259] A "generative AI model" refers to artificial intelligence technology used to analyze user behavior data and estimate user behavior patterns.

[0260] "Behavioral status" refers to information indicating the user's daily activity patterns and how they change.

[0261] A "monitoring officer" refers to an individual or group whose role is to receive notifications from the server in order to ensure the safety of users.

[0262] A "virtual pet" refers to a digitized animal character that a user raises and interacts with within an information processing device.

[0263] In this invention, a server, terminal, and user work together to provide a natural monitoring system using a virtual pet. The server receives operation data transmitted from the terminal and analyzes this data using a generated AI model. This makes it possible to estimate the user's behavior and identify behavioral patterns specific to the elderly. Based on the analysis results, the server is responsible for generating and sending notifications to remote monitors as needed.

[0264] The device provides a user interface that allows users to intuitively interact with virtual pets. This includes visual and auditory feedback, with the virtual pet responding to taps and voice commands. The interface can also provide feedback through animations and sounds as the user feeds or plays with the virtual pet.

[0265] Users raise virtual pets through applications installed on devices such as smartphones and tablets, and interact with them as part of their natural daily activities. This ensures safety, as abnormal behavior can be detected without the user noticing.

[0266] As a concrete example, an example of a prompt message is shown below.

[0267] "Please explain in detail how to build an AI model to evaluate safety based on virtual pet raising data from elderly individuals."

[0268] Thus, the system of the present invention can improve the quality of life for the elderly by providing natural monitoring without causing stress to the user.

[0269] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0270] Step 1:

[0271] The server receives operation data sent from the terminal. The input includes a record of the specific interactions the user has had with the virtual pet. This data includes, for example, the time and frequency of feeding, and the content of play. The output is this raw data, ready for use in the next analysis step.

[0272] Step 2:

[0273] The server analyzes the received operation data. This analysis uses a generative AI model to estimate the user's behavior patterns based on the input data. Specific data processing includes processing time-series data and detecting outliers. The output is the estimated results of the user's behavior state and patterns.

[0274] Step 3:

[0275] If the server detects an anomaly based on the analysis results, it generates and sends a notification to a remote monitor. This notification may include a report detailing specific user behavioral abnormalities and suggestions for necessary countermeasures. The input is the analysis results, and the output is the notification data sent to the monitor.

[0276] Step 4:

[0277] The device provides a user interface for users to interact with virtual pets. This interface receives input and outputs visual and auditory feedback. For example, when the user presses the "feed" button, the virtual pet displays an animation of satisfaction.

[0278] Step 5:

[0279] Users raise virtual pets using an application on their devices. The user's input actions, such as "feeding in the morning and playing at noon," are sent as data to the server via the device. This generates real-world activity data, which is used in subsequent analysis steps. The output is an operation history sent to the server.

[0280] (Application Example 1)

[0281] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0282] To improve the customer experience in commercial spaces and enhance visitor convenience within facilities, it is crucial that visitors enjoy their visit while ensuring a safe and comfortable stay. However, simply sending notifications to smartphones or using conventional marketing methods is insufficient to retain visitors' continued interest. Furthermore, current customer management systems are often mechanical and struggle to enrich the individual visitor experience.

[0283] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0284] In this invention, the server includes means for cultivating a virtual creature based on the operations of individual users provided on an information processing device, means for generating operation information of the user and estimating the activities of the user by analyzing the operation information, and means for recording the visiting status of the user in a commercial space and indicating that the virtual creature reacts according to the current position of the user. As a result, it becomes possible to provide a meaningful and safe store environment for visitors while allowing them to enjoy exploring the commercial space in a natural form, enriching their individual experiences.

[0285] An "information processing device" is a device having a function of generating operation information of a user and analyzing the information to estimate activities.

[0286] A "virtual creature" is a digital representation that is cultivated on an information processing device and shows reactions and growth according to the operations of the user.

[0287] An "arithmetic processing unit" is a device having a function of processing the analyzed information and transmitting it to a remote monitor.

[0288] A "communication interface" is an existing connection means for transmitting and receiving data between an information processing device and a monitor.

[0289] A "commercial space" is an area within a store or facility visited by users and is an environment for recording and analyzing the actions of visitors.

[0290] The present invention is a system for improving the user experience in a commercial space, and allows users to explore the space while cultivating virtual creatures. Hereinafter, the specific program processing and its operations in each element of the server, terminal, and user will be described.

[0291] Embodiment on the server side

[0292] The server receives and analyzes operational information transmitted from the user's smartphone. A generative AI model is used for analysis to estimate the user's behavior within the commercial space. Furthermore, it records visitor dwell time data at specific points within the commercial space and instructs a virtual organism to respond in real time based on the user's location. This process utilizes a web server based on the Django framework and a PostgreSQL database.

[0293] Terminal-side embodiment

[0294] The device, in this case the user's smartphone, provides a user interface for intuitive interaction with virtual creatures. Specifically, it has the functionality to make the virtual creatures act in response to screen taps and voice commands. This allows users to interact with virtual creatures while enjoying the commercial space, acquire items in response to specific actions, and participate in special events.

[0295] User-side embodiment

[0296] Users raise virtual creatures through an application installed on their smartphones. As users explore commercial spaces, they can enjoy the reactions of their virtual creatures, experiencing a contactless customer experience in the process. For example, when a user reaches a specific store section, the virtual creature may perform a new dance, which can then earn them a reward.

[0297] This invention utilizes generative AI models to provide users with unexpected and engaging experiences, thereby enhancing user engagement within commercial spaces.

[0298] Specific example:

[0299] "When users reach the 'premium section' within the commercial space, virtual creatures will perform a 'dance of joy,' and users can obtain special items."

[0300] Example of prompt text:

[0301] "Please propose a method to enrich the expression of joy when customers interact with virtual animals in the store."

[0302] The flow of specific processing in Application Example 1 will be described using FIG. 12.

[0303] Step 1:

[0304] The terminal collects operation information when the user interacts with the virtual creature. The input is taps and voice commands from the user, and the output is data packets that organize these operations as numerical data or string data and send them to the server. At this stage, the action animation of the virtual creature is executed on the terminal.

[0305] Step 2:

[0306] The server receives the operation information sent from the terminal. The input is the data packet sent from the terminal, and the output is the analysis result. Using the generation AI model, the operation information of the user is analyzed to estimate the behavior in the commercial space. The data is stored in the database as the user's behavior log.

[0307] Step 3:

[0308] Based on the analyzed behavior data of the user, the server determines the actions of the virtual creature based on specific points in the commercial space. The input is the behavior log data, and the output is the reaction data of the virtual creature. According to the position information, the generation AI model calculates the appropriate reaction of the virtual creature.

[0309] Step 4:

[0310] The server sends the analyzed virtual creature's responses to the terminal. The input is the generated response data, and the output is the virtual creature's actions displayed on the terminal. The server responds to the terminal in real time, instructing the virtual creature to perform actions.

[0311] Step 5:

[0312] Users explore a commercial space while observing the reactions of virtual creatures. Input is action information from the server, and output is the user's observations. Users can acquire items and rewards. This makes visiting the commercial space more appealing.

[0313] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0314] This invention combines an emotion engine with a virtual animal raising system to achieve multifaceted monitoring that includes the user's emotions. The specific processing and operation of the programs in the server, terminal, and user elements are described below.

[0315] Server-side embodiment

[0316] The server receives operation information transmitted from the terminal and emotional data from the emotion engine. The server uses an AI model to comprehensively analyze this data. Through this analysis, it estimates the user's behavioral patterns and emotional state, and notifies the supervisor as needed.

[0317] Specific example: The server determines that the user is not showing much interest in the virtual animal and generates a notification stating "emotional state is low," which is then sent to the monitor.

[0318] Terminal-side embodiment

[0319] The terminal is equipped with an operation display device as an information and communication device, providing visual and auditory feedback for the user to interact with the virtual animal. An emotion engine analyzes the user's voice tone and facial expressions, and transmits the resulting emotion data to the server along with the operation information.

[0320] Specific example: If the device detects that the user's voice is cheerful, the virtual animal will move more actively and display fun animations.

[0321] User-side embodiment

[0322] Users perform the usual virtual animal raising operations through an app that incorporates an emotion engine. During this process, the user's emotional state is naturally analyzed, and the virtual animal's reactions are adjusted accordingly, resulting in a more personalized raising experience.

[0323] Specific example: If the user makes a sad sound while operating the system, the virtual animal will display a message on the screen saying, "Cheer up."

[0324] In this way, the system of the present invention can provide meticulous monitoring that takes into account the user's emotions and support that takes into account the user's psychological and emotional health. The server, terminal, and user all work together to provide a monitoring environment optimized for each individual.

[0325] The following describes the processing flow.

[0326] Step 1:

[0327] The user launches an application on their device. The user interacts with a virtual animal, performing everyday actions such as feeding and playing with it.

[0328] Step 2:

[0329] The device records user interactions in real time and saves data such as the type of operation and time. Simultaneously, the emotion engine analyzes the user's voice and facial expressions to generate data that estimates their emotional state.

[0330] Step 3:

[0331] The device collects operational information and emotional data, which is then sent to a server at regular time intervals or triggered by an event. A secure protocol with data encryption is used for this purpose.

[0332] Step 4:

[0333] The server analyzes the received data and uses an AI model to estimate the user's activity patterns and emotional trends. Based on these estimations, it determines the user's current situation.

[0334] Step 5:

[0335] The server generates and sends a notification to the monitor when it detects a condition requiring special attention (for example, if a user exhibits negative emotions for an extended period). This notification includes the estimated emotional state and recommended actions.

[0336] Step 6:

[0337] The monitor receives notifications from the server and considers appropriate actions based on the user's situation. If necessary, they contact the user directly and provide support.

[0338] Step 7:

[0339] Upon receiving feedback again from the server, the device will display appropriate actions and messages to the user, thereby promoting emotional support for the user.

[0340] Following this process, this system, which includes an emotion engine, will consider and monitor the user's emotions in real time.

[0341] (Example 2)

[0342] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0343] Conventional electronic entity nurturing systems, while progressing based on user input, have the challenge of failing to consider the user's emotional state and providing personalized support. In particular, they are unable to respond appropriately to changes in the user's emotions, resulting in a lack of effective monitoring and support.

[0344] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0345] In this invention, the server includes means for nurturing an electronic entity based on operations performed by an individual user, means for acquiring the user's voice and video and generating emotional information using an emotion analysis device, and means for estimating the user's behavioral patterns and emotional state by analyzing the generated emotional information and the operation information. This enables highly personalized monitoring and support based on the user's emotions.

[0346] An "information processing device" is a general term for electronic devices designed to input, analyze, process, and output data.

[0347] An "electronic entity" refers to a virtual character or object that can interact with the user within a digital environment.

[0348] "User" refers to an individual who operates the system and interacts with electronic entities.

[0349] "Emotional analysis" refers to a technology that identifies or estimates a user's emotional state based on data such as voice tone and facial expressions.

[0350] "Communication structure" is a general term for networks and communication protocols used to transmit data to remote devices or monitors.

[0351] "Behavioral patterns" refer to the tendencies of a user's actions and choices under specific conditions, and based on these, it is possible to predict the user's future behavior.

[0352] "Monitoring" means providing safety and security by monitoring the user's condition and activities, and providing support and notifications as needed.

[0353] This invention is a system that utilizes an information processing device to cultivate an electronic entity based on user operations. The system analyzes the user's emotions and provides appropriate feedback accordingly.

[0354] The server receives operation information sent from the terminal and sentiment data from the sentiment engine. The server uses a generative AI model to comprehensively analyze this data. During the analysis process, it is possible to use data analysis tools such as Python's Pandas and Scikit-learn. Based on the analysis results, the server estimates the user's behavior patterns and emotional state and notifies the monitor as needed. This is done, for example, using a prompt message such as, "Analyze the user's psychological tendencies based on their behavior patterns and sentiment data, and generate a warning if attention is needed."

[0355] The device provides visual and auditory feedback to support user interaction with the electronic entity. For voice tone and facial expression analysis, it can utilize facial recognition software such as OpenCV and the Google Cloud Speech-to-Text API for speech analysis. The device sends the acquired emotional data to the server as user interaction information. Furthermore, if the user expresses positive emotions, it provides visual effects such as playing an animated sequence in which the electronic entity moves actively.

[0356] Users nurture electronic beings through applications that incorporate an emotion engine. The user's emotional state is analyzed in a natural way, and the electronic being's responses are adjusted accordingly. For example, if the user interacts with the electronic being in a sad voice, it will display "Cheer up" and provide encouraging feedback. In this case, a prompt message such as "If a sad tone of voice is detected, display an encouraging message." is used.

[0357] Through this system, we can provide a personalized monitoring environment based on the individual user's emotions, resulting in a more nurturing and ultimately beneficial parenting experience for the user.

[0358] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0359] Step 1:

[0360] The user initiates interaction with the electronic entity through the device. In this step, the user's voice and video are input via the device's microphone and camera. The device collects this input data and prepares to send it to emotion analysis software. Specifically, when the user speaks to the electronic entity, their voice and facial expressions are captured in real time.

[0361] Step 2:

[0362] The device sends the collected audio and video data to the emotion analysis engine. The emotion analysis engine uses the Google Cloud Speech-to-Text API to perform audio analysis, converting the user's voice tone into text. It also uses OpenCV to analyze the user's facial expressions from the video. This data processing provides an indicator of the user's emotional state. The output is analyzed emotion data. Specifically, if the user is smiling, data indicating that a smile has been detected will be output.

[0363] Step 3:

[0364] The device sends the generated emotion data and the user's current action information to the server. This action information includes specific choices and actions the user has made within the app. As a specific action that the server takes to prepare to receive this input data and save it to its database, the device records the action "the user fed the pet" and sends it along with the emotion data.

[0365] Step 4:

[0366] The server combines emotional data and operational information received from the terminal to perform a comprehensive analysis. This analysis utilizes a generative AI model, using Python's Pandas and Scikit-learn to analyze data trends. By combining the input data, the user's behavior patterns and emotional state are estimated. The output is a prediction of the user's state. For example, the server might predict that "the user is likely excited at the moment."

[0367] Step 5:

[0368] The server determines specific actions based on the analysis results and sends notifications to the monitor as needed. The generated notification includes a prompt that clearly communicates the user's state to the monitor. Specifically, it executes the action, "The user's emotional state is elevated; please notify me by email if attention is required."

[0369] Step 6:

[0370] The device receives instructions from the server and provides feedback to the user. If the user expresses positive emotions, it plays visual animations, such as the electronic entity beginning to dance. Specifically, cheerful music plays and the electronic entity moves around energetically on the screen. This feedback allows the user to experience interactions that match their emotions.

[0371] (Application Example 2)

[0372] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0373] In modern times, virtual organism breeding systems utilizing information and communication devices are widespread, but many systems rely solely on user operation information, lacking sufficient monitoring and support that takes emotional states into account. Furthermore, there is a growing need to provide personalized user experiences by delivering content tailored to the user's emotions. This invention aims to provide a personalized monitoring environment and emotionally responsive support by analyzing the user's emotional state in real time and providing feedback and content based on that analysis.

[0374] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0375] In this invention, the server includes means for raising a virtual organism based on operations by an individual user, means for estimating the user's activity and emotional state using the user's operation information and emotion analysis functions, and means for indirectly providing monitoring and emotion-based content via an information communication device that transmits the analyzed information to a remote monitor. This enables a monitoring and entertainment experience optimized for each individual user by providing feedback and content suggestions that respond to the user's emotions.

[0376] An "information processing device" is a general term for devices that input, process, and output data, and within a system, it is a device that is responsible for analyzing user operation information and emotional data.

[0377] A "user" refers to an individual who uses this system to raise a virtual creature, and is the subject of emotional analysis.

[0378] A "virtual creature" is a digital pet or character that is artificially generated on a computer, and whose actions and reactions change based on the user's input.

[0379] The "emotion analysis function" is the ability to automatically estimate the user's emotional state using AI technology based on their facial expressions and voice data.

[0380] A "remote monitor" is someone who is physically located away from the user but whose role is to understand the user's status through an information processing device and to provide support and monitoring.

[0381] A "communication interface" is a physical or logical connection means that allows an information processing device to exchange information with other devices or networks.

[0382] "Monitoring" refers to actions or systems that monitor the user's situation and condition and provide appropriate support to maintain their safety and health.

[0383] "Content recommendation" is the act of recommending the most suitable information and entertainment resources based on the user's emotions and preferences.

[0384] The server operates on an information processing device and integrates and analyzes user operation information and emotional data. This system receives operation information and emotional analysis results transmitted from the user's smart device (such as a smartphone or tablet) and processes them using a cloud-based generative AI model. Specifically, it performs data calculations to estimate how the user interacts with the virtual creature and the user's emotional state.

[0385] The terminal serves as an interface for the user, providing visual and auditory interaction with the virtual being. Simultaneously, the terminal's built-in camera and microphone capture the user's facial expressions and voice tone, and this data is converted using emotion analysis. This data is sent to a server as input information for real-time estimation of the user's emotional state. When the user operates through the terminal, the system comprehensively analyzes this operation information and emotion data to provide situation-appropriate monitoring functions and content suggestions.

[0386] As users utilize this system on a daily basis, they can gain personalized experiences through their interactions with virtual beings. For example, if the system analyzes that a user is feeling stressed, relaxing music or videos will be suggested. This suggestion feature utilizes a generative AI model that predicts what kind of content the user is looking for based on their emotional state and makes the optimal selection.

[0387] An example of a prompt message might be, "Create an AI model that analyzes the user's facial expressions and voice to suggest relaxing content." This would enable more accurate content suggestions based on emotions, providing a more personalized entertainment experience.

[0388] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0389] Step 1:

[0390] The device collects user interaction and emotional data. Specifically, it captures the user's facial expressions with its built-in camera and records their voice tone with its microphone. This data is processed by an emotion analysis algorithm to generate information for estimating the user's emotional state. The inputs are camera footage and voice audio, and the output is the estimated emotional data.

[0391] Step 2:

[0392] The terminal transmits collected and analyzed operation information and sentiment data to the server. The transmitted data includes operation logs and sentiment analysis results related to user interactions. This process is a preparatory stage for comprehensive data analysis by the server. The input is locally analyzed operation information and sentiment data, and the output is data transfer to the server.

[0393] Step 3:

[0394] The server integrates the received data and uses a generative AI model to estimate the user's behavioral patterns and emotional states. Data processing includes time-series analysis to analyze temporal behavioral patterns and regression analysis to evaluate emotional states. Inputs are operation information and emotional data from the terminal, while output is estimated behavioral and emotional trend data.

[0395] Step 4:

[0396] The server selects the most suitable content using prompts based on estimated data. It retrieves content from the database that is appropriate for the user's emotional state, as predicted by the generative AI model, and generates suggestions. This step provides a personalized user experience by dynamically recommending content that matches the user's emotions. The input is behavioral and emotional trend data, and the output is content suggestion data.

[0397] Step 5:

[0398] Upon receiving suggestion data from the server, the terminal provides visual and auditory notifications to the user's screen. Specifically, it displays details and options for the suggested content and awaits user action. The input is the content suggestion data from the server, and the output is the content information displayed on the user interface.

[0399] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0400] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0401] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart glasses 214.

[0402] [Third Embodiment]

[0403] Figure 5 shows an example of the configuration of the data processing system 310 according to the third embodiment.

[0404] As shown in Figure 5, the data processing system 310 includes a data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0405] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0406] The headset terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a display 343. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and display 343 are also connected to the bus 52.

[0407] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0408] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0409] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0410] Figure 6 shows an example of the main functions of the data processing device 12 and the headset terminal 314. As shown in Figure 6, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0411] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0412] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0413] In the headset terminal 314, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0414] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the headset terminal 314 will be referred to as the "terminal".

[0415] The system based on this invention provides a virtual animal raising function that allows users to be naturally monitored. The specific program processing and operation of each element—server, terminal, and user—are described below.

[0416] Server-side embodiment

[0417] The server receives and analyzes operational information sent by the user. This analysis utilizes an AI model to estimate the user's activity level and identify behavioral patterns specific to the elderly. Based on the analysis results, it generates and sends notifications to the supervisor as needed.

[0418] Specific example: The server detects the possibility that a user is not living a normal life based on information such as "the frequency of feeding virtual animals has decreased significantly," and notifies the monitor.

[0419] Terminal-side embodiment

[0420] The device provides a user interface for users to intuitively interact with virtual animals. This includes features that allow the virtual animals to perform actions in response to taps and voice commands. Interaction data is sent to the server at regular intervals.

[0421] Specific example: On the device screen, when the user presses the "feed" button, an animation of the virtual animal eating food is displayed, showing a satisfied expression.

[0422] User-side embodiment

[0423] Users raise virtual animals through an app installed on their smartphones or tablets. Caring for the virtual animals is integrated into the user's daily activities, allowing for natural interaction as part of their life. As a result, relevant information is accumulated, and the user's status is indirectly communicated to a supervisor.

[0424] Specific example: A user can habitually feed a virtual animal every morning and play with their pet during the day, ensuring their security without realizing that their normal activities are being transmitted to the server as data.

[0425] Thus, this invention makes it possible to provide natural monitoring without causing stress to users and to improve the quality of life for the elderly. Through the collaboration of the server, terminal, and user, it provides a continuous monitoring environment that is not resistant to the user.

[0426] The following describes the processing flow.

[0427] Step 1:

[0428] The user launches a virtual animal app on their information and communication terminal. The user checks the virtual animal's status and begins interacting with it, such as feeding it or selecting toys to play with.

[0429] Step 2:

[0430] The device detects user actions in real time and records those actions as operation information. Specifically, it collects data such as operation type, operation time, and duration.

[0431] Step 3:

[0432] The terminal is set to send collected operation information to the server at regular intervals, and then transmits the information. During this process, the data is encrypted according to a security protocol to ensure secure communication.

[0433] Step 4:

[0434] The server stores the received operation information and performs analysis using an AI model. In this analysis process, user activity patterns and unique behaviors are estimated based on the operation information.

[0435] Step 5:

[0436] Based on the analysis results, the server generates necessary monitoring information tailored to the user's situation. For example, if an anomaly is detected, such as "the virtual animal has not been fed for several days," it generates a notification indicating that attention is needed.

[0437] Step 6:

[0438] The server sends the generated monitoring information to the monitor via an existing communication interface. Typically, the monitor is a family member or caregiver of the user. The notification content is provided in the form of suggestions for appropriate actions based on the analysis results.

[0439] Step 7:

[0440] The monitor reviews notifications received from the server and considers appropriate actions based on the user's situation. Based on this, they decide whether to contact the user directly or provide additional support if necessary.

[0441] (Example 1)

[0442] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0443] In modern society, ensuring the safety of the elderly is a crucial issue. However, direct surveillance can infringe on privacy and cause stress. In particular, there is a need for new methods to naturally observe daily behavior and detect abnormalities early.

[0444] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0445] In this invention, the server includes means for managing a virtual pet based on user operations, means for generating operation data, analyzing it using a generation AI model, and estimating the user's behavioral state, and means for providing a method for generating and transmitting notifications to a remote monitor based on the analysis. This makes it possible to monitor the user naturally without causing them stress.

[0446] An "information processing device" is an electronic device used to receive, process, and generate data, and to monitor for anomalies.

[0447] "User" refers to an individual who manages and operates a virtual pet.

[0448] "Operation data" refers to the record of actions performed by the user on the virtual pet.

[0449] A "generative AI model" refers to artificial intelligence technology used to analyze user behavior data and estimate user behavior patterns.

[0450] "Behavioral status" refers to information indicating the user's daily activity patterns and how they change.

[0451] A "monitoring officer" refers to an individual or group whose role is to receive notifications from the server in order to ensure the safety of users.

[0452] A "virtual pet" refers to a digitized animal character that a user raises and interacts with within an information processing device.

[0453] In this invention, a server, terminal, and user work together to provide a natural monitoring system using a virtual pet. The server receives operation data transmitted from the terminal and analyzes this data using a generated AI model. This makes it possible to estimate the user's behavior and identify behavioral patterns specific to the elderly. Based on the analysis results, the server is responsible for generating and sending notifications to remote monitors as needed.

[0454] The device provides a user interface that allows users to intuitively interact with virtual pets. This includes visual and auditory feedback, with the virtual pet responding to taps and voice commands. The interface can also provide feedback through animations and sounds as the user feeds or plays with the virtual pet.

[0455] Users raise virtual pets through applications installed on devices such as smartphones and tablets, and interact with them as part of their natural daily activities. This ensures safety, as abnormal behavior can be detected without the user noticing.

[0456] As a concrete example, an example of a prompt message is shown below.

[0457] "Please explain in detail how to build an AI model to evaluate safety based on virtual pet raising data from elderly individuals."

[0458] Thus, the system of the present invention can improve the quality of life for the elderly by providing natural monitoring without causing stress to the user.

[0459] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0460] Step 1:

[0461] The server receives operation data sent from the terminal. The input includes a record of the specific interactions the user has had with the virtual pet. This data includes, for example, the time and frequency of feeding, and the content of play. The output is this raw data, ready for use in the next analysis step.

[0462] Step 2:

[0463] The server analyzes the received operation data. This analysis uses a generative AI model to estimate the user's behavior patterns based on the input data. Specific data processing includes processing time-series data and detecting outliers. The output is the estimated results of the user's behavior state and patterns.

[0464] Step 3:

[0465] If the server detects an anomaly based on the analysis results, it generates and sends a notification to a remote monitor. This notification may include a report detailing specific user behavioral abnormalities and suggestions for necessary countermeasures. The input is the analysis results, and the output is the notification data sent to the monitor.

[0466] Step 4:

[0467] The device provides a user interface for users to interact with virtual pets. This interface receives input and outputs visual and auditory feedback. For example, when the user presses the "feed" button, the virtual pet displays an animation of satisfaction.

[0468] Step 5:

[0469] Users raise virtual pets using an application on their devices. The user's input actions, such as "feeding in the morning and playing at noon," are sent as data to the server via the device. This generates real-world activity data, which is used in subsequent analysis steps. The output is an operation history sent to the server.

[0470] (Application Example 1)

[0471] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0472] To improve the customer experience in commercial spaces and enhance visitor convenience within facilities, it is crucial that visitors enjoy their visit while ensuring a safe and comfortable stay. However, simply sending notifications to smartphones or using conventional marketing methods is insufficient to retain visitors' continued interest. Furthermore, current customer management systems are often mechanical and struggle to enrich the individual visitor experience.

[0473] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0474] In this invention, the server includes means for cultivating a virtual organism based on operations performed by individual users on an information processing device, means for generating user operation information and estimating the user's activities by analyzing said operation information, and means for recording the user's visit status in a commercial space and for the virtual organism to react according to the user's current location. This makes it possible to enrich individual experiences while visitors enjoy exploring the commercial space in a natural way, and to provide visitors with a meaningful and safe store environment.

[0475] An "information processing device" is a device that generates user operation information, analyzes that information, and has the function of estimating activity.

[0476] A "virtual organism" is a digital representation that is cultivated on an information processing device and exhibits reactions and growth in response to user input.

[0477] A "processing unit" is a device that has the function of processing analyzed information and transmitting it to a remote monitor.

[0478] A "communication interface" is an existing connection method that transmits and receives data between an information processing device and a monitor.

[0479] A "commercial space" is an area within a store or facility that is visited by users, and it is an environment for recording and analyzing the behavior of visitors.

[0480] This invention is a system for improving the user experience in commercial spaces, allowing users to explore the space while raising virtual creatures. The specific program processing and operation of each element—server, terminal, and user—are described below.

[0481] Server-side embodiment

[0482] The server receives and analyzes operational information transmitted from the user's smartphone. A generative AI model is used for analysis to estimate the user's behavior within the commercial space. Furthermore, it records visitor dwell time data at specific points within the commercial space and instructs a virtual organism to respond in real time based on the user's location. This process utilizes a web server based on the Django framework and a PostgreSQL database.

[0483] Terminal-side embodiment

[0484] The device, in this case the user's smartphone, provides a user interface for intuitive interaction with virtual creatures. Specifically, it has the functionality to make the virtual creatures act in response to screen taps and voice commands. This allows users to interact with virtual creatures while enjoying the commercial space, acquire items in response to specific actions, and participate in special events.

[0485] User-side embodiment

[0486] Users raise virtual creatures through an application installed on their smartphones. As users explore commercial spaces, they can enjoy the reactions of their virtual creatures, experiencing a contactless customer experience in the process. For example, when a user reaches a specific store section, the virtual creature may perform a new dance, which can then earn them a reward.

[0487] This invention utilizes generative AI models to provide users with unexpected and engaging experiences, thereby enhancing user engagement within commercial spaces.

[0488] Specific example:

[0489] "When users reach the 'premium section' within the commercial space, virtual creatures will perform a 'dance of joy,' and users can obtain special items."

[0490] Example of a prompt:

[0491] "Please suggest ways to enrich the expressions of joy when customers interact with virtual animals in the store."

[0492] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0493] Step 1:

[0494] The terminal collects information about the user's interactions with the virtual creature. Input consists of taps and voice commands from the user, and output consists of data packets that organize these operations as numerical and string data and send them to the server. At this stage, the virtual creature's action animation is executed on the terminal.

[0495] Step 2:

[0496] The server receives operation information transmitted from the terminal. The input is data packets sent from the terminal, and the output is the analysis result. A generative AI model is used to analyze the user's operation information and estimate their behavior within the commercial space. The data is stored in a database as a user behavior log.

[0497] Step 3:

[0498] The server determines the actions of virtual creatures based on specific points within a commercial space, using analyzed user behavior data. The input is behavior log data, and the output is virtual creature response data. Based on the location information, a generative AI model calculates the appropriate response for the virtual creature.

[0499] Step 4:

[0500] The server sends the analyzed virtual creature's responses to the terminal. The input is the generated response data, and the output is the virtual creature's actions displayed on the terminal. The server responds to the terminal in real time, instructing the virtual creature to perform actions.

[0501] Step 5:

[0502] Users explore a commercial space while observing the reactions of virtual creatures. Input is action information from the server, and output is the user's observations. Users can acquire items and rewards. This makes visiting the commercial space more appealing.

[0503] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0504] This invention combines an emotion engine with a virtual animal raising system to achieve multifaceted monitoring that includes the user's emotions. The specific processing and operation of the programs in the server, terminal, and user elements are described below.

[0505] Server-side embodiment

[0506] The server receives operation information transmitted from the terminal and emotional data from the emotion engine. The server uses an AI model to comprehensively analyze this data. Through this analysis, it estimates the user's behavioral patterns and emotional state, and notifies the supervisor as needed.

[0507] Specific example: The server determines that the user is not showing much interest in the virtual animal and generates a notification stating "emotional state is low," which is then sent to the monitor.

[0508] Terminal-side embodiment

[0509] The terminal is equipped with an operation display device as an information and communication device, providing visual and auditory feedback for the user to interact with the virtual animal. An emotion engine analyzes the user's voice tone and facial expressions, and transmits the resulting emotion data to the server along with the operation information.

[0510] Specific example: If the device detects that the user's voice is cheerful, the virtual animal will move more actively and display fun animations.

[0511] User-side embodiment

[0512] Users perform the usual virtual animal raising operations through an app that incorporates an emotion engine. During this process, the user's emotional state is naturally analyzed, and the virtual animal's reactions are adjusted accordingly, resulting in a more personalized raising experience.

[0513] Specific example: If the user makes a sad sound while operating the system, the virtual animal will display a message on the screen saying, "Cheer up."

[0514] In this way, the system of the present invention can provide meticulous monitoring that takes into account the user's emotions and support that takes into account the user's psychological and emotional health. The server, terminal, and user all work together to provide a monitoring environment optimized for each individual.

[0515] The following describes the processing flow.

[0516] Step 1:

[0517] The user launches an application on their device. The user interacts with a virtual animal, performing everyday actions such as feeding and playing with it.

[0518] Step 2:

[0519] The device records user interactions in real time and saves data such as the type of operation and time. Simultaneously, the emotion engine analyzes the user's voice and facial expressions to generate data that estimates their emotional state.

[0520] Step 3:

[0521] The device collects operational information and emotional data, which is then sent to a server at regular time intervals or triggered by an event. A secure protocol with data encryption is used for this purpose.

[0522] Step 4:

[0523] The server analyzes the received data and uses an AI model to estimate the user's activity patterns and emotional trends. Based on these estimations, it determines the user's current situation.

[0524] Step 5:

[0525] The server generates and sends a notification to the monitor when it detects a condition requiring special attention (for example, if a user exhibits negative emotions for an extended period). This notification includes the estimated emotional state and recommended actions.

[0526] Step 6:

[0527] The monitor receives notifications from the server and considers appropriate actions based on the user's situation. If necessary, they contact the user directly and provide support.

[0528] Step 7:

[0529] Upon receiving feedback again from the server, the device will display appropriate actions and messages to the user, thereby promoting emotional support for the user.

[0530] Following this process, this system, which includes an emotion engine, will consider and monitor the user's emotions in real time.

[0531] (Example 2)

[0532] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0533] Conventional electronic entity nurturing systems, while progressing based on user input, have the challenge of failing to consider the user's emotional state and providing personalized support. In particular, they are unable to respond appropriately to changes in the user's emotions, resulting in a lack of effective monitoring and support.

[0534] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0535] In this invention, the server includes means for nurturing an electronic entity based on operations performed by an individual user, means for acquiring the user's voice and video and generating emotional information using an emotion analysis device, and means for estimating the user's behavioral patterns and emotional state by analyzing the generated emotional information and the operation information. This enables highly personalized monitoring and support based on the user's emotions.

[0536] An "information processing device" is a general term for electronic devices designed to input, analyze, process, and output data.

[0537] An "electronic entity" refers to a virtual character or object that can interact with the user within a digital environment.

[0538] "User" refers to an individual who operates the system and interacts with electronic entities.

[0539] "Emotional analysis" refers to a technology that identifies or estimates a user's emotional state based on data such as voice tone and facial expressions.

[0540] "Communication structure" is a general term for networks and communication protocols used to transmit data to remote devices or monitors.

[0541] "Behavioral patterns" refer to the tendencies of a user's actions and choices under specific conditions, and based on these, it is possible to predict the user's future behavior.

[0542] "Monitoring" means providing safety and security by monitoring the user's condition and activities, and providing support and notifications as needed.

[0543] This invention is a system that utilizes an information processing device to cultivate an electronic entity based on user operations. The system analyzes the user's emotions and provides appropriate feedback accordingly.

[0544] The server receives operation information sent from the terminal and sentiment data from the sentiment engine. The server uses a generative AI model to comprehensively analyze this data. During the analysis process, it is possible to use data analysis tools such as Python's Pandas and Scikit-learn. Based on the analysis results, the server estimates the user's behavior patterns and emotional state and notifies the monitor as needed. This is done, for example, using a prompt message such as, "Analyze the user's psychological tendencies based on their behavior patterns and sentiment data, and generate a warning if attention is needed."

[0545] The device provides visual and auditory feedback to support user interaction with the electronic entity. For voice tone and facial expression analysis, it can utilize facial recognition software such as OpenCV and the Google Cloud Speech-to-Text API for speech analysis. The device sends the acquired emotional data to the server as user interaction information. Furthermore, if the user expresses positive emotions, it provides visual effects such as playing an animated sequence in which the electronic entity moves actively.

[0546] Users nurture electronic beings through applications that incorporate an emotion engine. The user's emotional state is analyzed in a natural way, and the electronic being's responses are adjusted accordingly. For example, if the user interacts with the electronic being in a sad voice, it will display "Cheer up" and provide encouraging feedback. In this case, a prompt message such as "If a sad tone of voice is detected, display an encouraging message." is used.

[0547] Through this system, we can provide a personalized monitoring environment based on the individual user's emotions, resulting in a more nurturing and ultimately beneficial parenting experience for the user.

[0548] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0549] Step 1:

[0550] The user initiates interaction with the electronic entity through the device. In this step, the user's voice and video are input via the device's microphone and camera. The device collects this input data and prepares to send it to emotion analysis software. Specifically, when the user speaks to the electronic entity, their voice and facial expressions are captured in real time.

[0551] Step 2:

[0552] The device sends the collected audio and video data to the emotion analysis engine. The emotion analysis engine uses the Google Cloud Speech-to-Text API to perform audio analysis, converting the user's voice tone into text. It also uses OpenCV to analyze the user's facial expressions from the video. This data processing provides an indicator of the user's emotional state. The output is analyzed emotion data. Specifically, if the user is smiling, data indicating that a smile has been detected will be output.

[0553] Step 3:

[0554] The device sends the generated emotion data and the user's current action information to the server. This action information includes specific choices and actions the user has made within the app. As a specific action that the server takes to prepare to receive this input data and save it to its database, the device records the action "the user fed the pet" and sends it along with the emotion data.

[0555] Step 4:

[0556] The server combines emotional data and operational information received from the terminal to perform a comprehensive analysis. This analysis utilizes a generative AI model, using Python's Pandas and Scikit-learn to analyze data trends. By combining the input data, the user's behavior patterns and emotional state are estimated. The output is a prediction of the user's state. For example, the server might predict that "the user is likely excited at the moment."

[0557] Step 5:

[0558] The server determines specific actions based on the analysis results and sends notifications to the monitor as needed. The generated notification includes a prompt that clearly communicates the user's state to the monitor. Specifically, it executes the action, "The user's emotional state is elevated; please notify me by email if attention is required."

[0559] Step 6:

[0560] The device receives instructions from the server and provides feedback to the user. If the user expresses positive emotions, it plays visual animations, such as the electronic entity beginning to dance. Specifically, cheerful music plays and the electronic entity moves around energetically on the screen. This feedback allows the user to experience interactions that match their emotions.

[0561] (Application Example 2)

[0562] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0563] In modern times, virtual organism breeding systems utilizing information and communication devices are widespread, but many systems rely solely on user operation information, lacking sufficient monitoring and support that takes emotional states into account. Furthermore, there is a growing need to provide personalized user experiences by delivering content tailored to the user's emotions. This invention aims to provide a personalized monitoring environment and emotionally responsive support by analyzing the user's emotional state in real time and providing feedback and content based on that analysis.

[0564] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0565] In this invention, the server includes means for raising a virtual organism based on operations by an individual user, means for estimating the user's activity and emotional state using the user's operation information and emotion analysis functions, and means for indirectly providing monitoring and emotion-based content via an information communication device that transmits the analyzed information to a remote monitor. This enables a monitoring and entertainment experience optimized for each individual user by providing feedback and content suggestions that respond to the user's emotions.

[0566] An "information processing device" is a general term for devices that input, process, and output data, and within a system, it is a device that is responsible for analyzing user operation information and emotional data.

[0567] A "user" refers to an individual who uses this system to raise a virtual creature, and is the subject of emotional analysis.

[0568] A "virtual creature" is a digital pet or character that is artificially generated on a computer, and whose actions and reactions change based on the user's input.

[0569] The "emotion analysis function" is the ability to automatically estimate the user's emotional state using AI technology based on their facial expressions and voice data.

[0570] A "remote monitor" is someone who is physically located away from the user but whose role is to understand the user's status through an information processing device and to provide support and monitoring.

[0571] A "communication interface" is a physical or logical connection means that allows an information processing device to exchange information with other devices or networks.

[0572] "Monitoring" refers to actions or systems that monitor the user's situation and condition and provide appropriate support to maintain their safety and health.

[0573] "Content recommendation" is the act of recommending the most suitable information and entertainment resources based on the user's emotions and preferences.

[0574] The server operates on an information processing device and integrates and analyzes user operation information and emotional data. This system receives operation information and emotional analysis results transmitted from the user's smart device (such as a smartphone or tablet) and processes them using a cloud-based generative AI model. Specifically, it performs data calculations to estimate how the user interacts with the virtual creature and the user's emotional state.

[0575] The terminal serves as an interface for the user, providing visual and auditory interaction with the virtual being. Simultaneously, the terminal's built-in camera and microphone capture the user's facial expressions and voice tone, and this data is converted using emotion analysis. This data is sent to a server as input information for real-time estimation of the user's emotional state. When the user operates through the terminal, the system comprehensively analyzes this operation information and emotion data to provide situation-appropriate monitoring functions and content suggestions.

[0576] As users utilize this system on a daily basis, they can gain personalized experiences through their interactions with virtual beings. For example, if the system analyzes that a user is feeling stressed, relaxing music or videos will be suggested. This suggestion feature utilizes a generative AI model that predicts what kind of content the user is looking for based on their emotional state and makes the optimal selection.

[0577] An example of a prompt message might be, "Create an AI model that analyzes the user's facial expressions and voice to suggest relaxing content." This would enable more accurate content suggestions based on emotions, providing a more personalized entertainment experience.

[0578] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0579] Step 1:

[0580] The device collects user interaction and emotional data. Specifically, it captures the user's facial expressions with its built-in camera and records their voice tone with its microphone. This data is processed by an emotion analysis algorithm to generate information for estimating the user's emotional state. The inputs are camera footage and voice audio, and the output is the estimated emotional data.

[0581] Step 2:

[0582] The terminal transmits collected and analyzed operation information and sentiment data to the server. The transmitted data includes operation logs and sentiment analysis results related to user interactions. This process is a preparatory stage for comprehensive data analysis by the server. The input is locally analyzed operation information and sentiment data, and the output is data transfer to the server.

[0583] Step 3:

[0584] The server integrates the received data and uses a generative AI model to estimate the user's behavioral patterns and emotional states. Data processing includes time-series analysis to analyze temporal behavioral patterns and regression analysis to evaluate emotional states. Inputs are operation information and emotional data from the terminal, while output is estimated behavioral and emotional trend data.

[0585] Step 4:

[0586] The server selects the most suitable content using prompts based on estimated data. It retrieves content from the database that is appropriate for the user's emotional state, as predicted by the generative AI model, and generates suggestions. This step provides a personalized user experience by dynamically recommending content that matches the user's emotions. The input is behavioral and emotional trend data, and the output is content suggestion data.

[0587] Step 5:

[0588] Upon receiving suggestion data from the server, the terminal provides visual and auditory notifications to the user's screen. Specifically, it displays details and options for the suggested content and awaits user action. The input is the content suggestion data from the server, and the output is the content information displayed on the user interface.

[0589] The specific processing unit 290 transmits the result of the specific processing to the headset terminal 314. In the headset terminal 314, the control unit 46A causes the speaker 240 and display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0590] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0591] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and specific processing may also be performed by the headset terminal 314.

[0592] [Fourth Embodiment]

[0593] Figure 7 shows an example of the configuration of the data processing system 410 according to the fourth embodiment.

[0594] As shown in Figure 7, the data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[0595] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0596] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a controlled object 443. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and controlled object 443 are also connected to the bus 52.

[0597] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0598] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0599] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0600] The controlled object 443 includes a display device, LEDs in the eyes, and motors that drive the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the robot 414's emotions can be expressed by controlling these motors. Furthermore, the robot 414's facial expressions can also be expressed by controlling the illumination state of the LEDs in its eyes.

[0601] Figure 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Figure 8, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0602] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0603] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0604] In robot 414, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0605] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0606] The system based on this invention provides a virtual animal raising function that allows users to be naturally monitored. The specific program processing and operation of each element—server, terminal, and user—are described below.

[0607] Server-side embodiment

[0608] The server receives and analyzes operational information sent by the user. This analysis utilizes an AI model to estimate the user's activity level and identify behavioral patterns specific to the elderly. Based on the analysis results, it generates and sends notifications to the supervisor as needed.

[0609] Specific example: The server detects the possibility that a user is not living a normal life based on information such as "the frequency of feeding virtual animals has decreased significantly," and notifies the monitor.

[0610] Terminal-side embodiment

[0611] The device provides a user interface for users to intuitively interact with virtual animals. This includes features that allow the virtual animals to perform actions in response to taps and voice commands. Interaction data is sent to the server at regular intervals.

[0612] Specific example: On the device screen, when the user presses the "feed" button, an animation of the virtual animal eating food is displayed, showing a satisfied expression.

[0613] User-side embodiment

[0614] Users raise virtual animals through an app installed on their smartphones or tablets. Caring for the virtual animals is integrated into the user's daily activities, allowing for natural interaction as part of their life. As a result, relevant information is accumulated, and the user's status is indirectly communicated to a supervisor.

[0615] Specific example: A user can habitually feed a virtual animal every morning and play with their pet during the day, ensuring their security without realizing that their normal activities are being transmitted to the server as data.

[0616] Thus, this invention makes it possible to provide natural monitoring without causing stress to users and to improve the quality of life for the elderly. Through the collaboration of the server, terminal, and user, it provides a continuous monitoring environment that is not resistant to the user.

[0617] The following describes the processing flow.

[0618] Step 1:

[0619] The user launches a virtual animal app on their information and communication terminal. The user checks the virtual animal's status and begins interacting with it, such as feeding it or selecting toys to play with.

[0620] Step 2:

[0621] The device detects user actions in real time and records those actions as operation information. Specifically, it collects data such as operation type, operation time, and duration.

[0622] Step 3:

[0623] The terminal is set to send collected operation information to the server at regular intervals, and then transmits the information. During this process, the data is encrypted according to a security protocol to ensure secure communication.

[0624] Step 4:

[0625] The server stores the received operation information and performs analysis using an AI model. In this analysis process, user activity patterns and unique behaviors are estimated based on the operation information.

[0626] Step 5:

[0627] Based on the analysis results, the server generates necessary monitoring information tailored to the user's situation. For example, if an anomaly is detected, such as "the virtual animal has not been fed for several days," it generates a notification indicating that attention is needed.

[0628] Step 6:

[0629] The server sends the generated monitoring information to the monitor via an existing communication interface. Typically, the monitor is a family member or caregiver of the user. The notification content is provided in the form of suggestions for appropriate actions based on the analysis results.

[0630] Step 7:

[0631] The monitor reviews notifications received from the server and considers appropriate actions based on the user's situation. Based on this, they decide whether to contact the user directly or provide additional support if necessary.

[0632] (Example 1)

[0633] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0634] In modern society, ensuring the safety of the elderly is a crucial issue. However, direct surveillance can infringe on privacy and cause stress. In particular, there is a need for new methods to naturally observe daily behavior and detect abnormalities early.

[0635] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0636] In this invention, the server includes means for managing a virtual pet based on user operations, means for generating operation data, analyzing it using a generation AI model, and estimating the user's behavioral state, and means for providing a method for generating and transmitting notifications to a remote monitor based on the analysis. This makes it possible to monitor the user naturally without causing them stress.

[0637] An "information processing device" is an electronic device used to receive, process, and generate data, and to monitor for anomalies.

[0638] "User" refers to an individual who manages and operates a virtual pet.

[0639] "Operation data" refers to the record of actions performed by the user on the virtual pet.

[0640] A "generative AI model" refers to artificial intelligence technology used to analyze user behavior data and estimate user behavior patterns.

[0641] "Behavioral status" refers to information indicating the user's daily activity patterns and how they change.

[0642] A "monitoring officer" refers to an individual or group whose role is to receive notifications from the server in order to ensure the safety of users.

[0643] A "virtual pet" refers to a digitized animal character that a user raises and interacts with within an information processing device.

[0644] In this invention, a server, terminal, and user work together to provide a natural monitoring system using a virtual pet. The server receives operation data transmitted from the terminal and analyzes this data using a generated AI model. This makes it possible to estimate the user's behavior and identify behavioral patterns specific to the elderly. Based on the analysis results, the server is responsible for generating and sending notifications to remote monitors as needed.

[0645] The device provides a user interface that allows users to intuitively interact with virtual pets. This includes visual and auditory feedback, with the virtual pet responding to taps and voice commands. The interface can also provide feedback through animations and sounds as the user feeds or plays with the virtual pet.

[0646] Users raise virtual pets through applications installed on devices such as smartphones and tablets, and interact with them as part of their natural daily activities. This ensures safety, as abnormal behavior can be detected without the user noticing.

[0647] As a concrete example, an example of a prompt message is shown below.

[0648] "Please explain in detail how to build an AI model to evaluate safety based on virtual pet raising data from elderly individuals."

[0649] Thus, the system of the present invention can improve the quality of life for the elderly by providing natural monitoring without causing stress to the user.

[0650] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0651] Step 1:

[0652] The server receives operation data sent from the terminal. The input includes a record of the specific interactions the user has had with the virtual pet. This data includes, for example, the time and frequency of feeding, and the content of play. The output is this raw data, ready for use in the next analysis step.

[0653] Step 2:

[0654] The server analyzes the received operation data. This analysis uses a generative AI model to estimate the user's behavior patterns based on the input data. Specific data processing includes processing time-series data and detecting outliers. The output is the estimated results of the user's behavior state and patterns.

[0655] Step 3:

[0656] If the server detects an anomaly based on the analysis results, it generates and sends a notification to a remote monitor. This notification may include a report detailing specific user behavioral abnormalities and suggestions for necessary countermeasures. The input is the analysis results, and the output is the notification data sent to the monitor.

[0657] Step 4:

[0658] The device provides a user interface for users to interact with virtual pets. This interface receives input and outputs visual and auditory feedback. For example, when the user presses the "feed" button, the virtual pet displays an animation of satisfaction.

[0659] Step 5:

[0660] Users raise virtual pets using an application on their devices. The user's input actions, such as "feeding in the morning and playing at noon," are sent as data to the server via the device. This generates real-world activity data, which is used in subsequent analysis steps. The output is an operation history sent to the server.

[0661] (Application Example 1)

[0662] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0663] To improve the customer experience in commercial spaces and enhance visitor convenience within facilities, it is crucial that visitors enjoy their visit while ensuring a safe and comfortable stay. However, simply sending notifications to smartphones or using conventional marketing methods is insufficient to retain visitors' continued interest. Furthermore, current customer management systems are often mechanical and struggle to enrich the individual visitor experience.

[0664] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0665] In this invention, the server includes means for cultivating a virtual organism based on operations performed by individual users on an information processing device, means for generating user operation information and estimating the user's activities by analyzing said operation information, and means for recording the user's visit status in a commercial space and for the virtual organism to react according to the user's current location. This makes it possible to enrich individual experiences while visitors enjoy exploring the commercial space in a natural way, and to provide visitors with a meaningful and safe store environment.

[0666] An "information processing device" is a device that generates user operation information, analyzes that information, and has the function of estimating activity.

[0667] A "virtual organism" is a digital representation that is cultivated on an information processing device and exhibits reactions and growth in response to user input.

[0668] A "processing unit" is a device that has the function of processing analyzed information and transmitting it to a remote monitor.

[0669] A "communication interface" is an existing connection method that transmits and receives data between an information processing device and a monitor.

[0670] A "commercial space" is an area within a store or facility that is visited by users, and it is an environment for recording and analyzing the behavior of visitors.

[0671] This invention is a system for improving the user experience in commercial spaces, allowing users to explore the space while raising virtual creatures. The specific program processing and operation of each element—server, terminal, and user—are described below.

[0672] Server-side embodiment

[0673] The server receives and analyzes operational information transmitted from the user's smartphone. A generative AI model is used for analysis to estimate the user's behavior within the commercial space. Furthermore, it records visitor dwell time data at specific points within the commercial space and instructs a virtual organism to respond in real time based on the user's location. This process utilizes a web server based on the Django framework and a PostgreSQL database.

[0674] Terminal-side embodiment

[0675] The device, in this case the user's smartphone, provides a user interface for intuitive interaction with virtual creatures. Specifically, it has the functionality to make the virtual creatures act in response to screen taps and voice commands. This allows users to interact with virtual creatures while enjoying the commercial space, acquire items in response to specific actions, and participate in special events.

[0676] User-side embodiment

[0677] Users raise virtual creatures through an application installed on their smartphones. As users explore commercial spaces, they can enjoy the reactions of their virtual creatures, experiencing a contactless customer experience in the process. For example, when a user reaches a specific store section, the virtual creature may perform a new dance, which can then earn them a reward.

[0678] This invention utilizes generative AI models to provide users with unexpected and engaging experiences, thereby enhancing user engagement within commercial spaces.

[0679] Specific example:

[0680] "When users reach the 'premium section' within the commercial space, virtual creatures will perform a 'dance of joy,' and users can obtain special items."

[0681] Example of a prompt:

[0682] "Please suggest ways to enrich the expressions of joy when customers interact with virtual animals in the store."

[0683] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0684] Step 1:

[0685] The terminal collects information about the user's interactions with the virtual creature. Input consists of taps and voice commands from the user, and output consists of data packets that organize these operations as numerical and string data and send them to the server. At this stage, the virtual creature's action animation is executed on the terminal.

[0686] Step 2:

[0687] The server receives operation information transmitted from the terminal. The input is data packets sent from the terminal, and the output is the analysis result. A generative AI model is used to analyze the user's operation information and estimate their behavior within the commercial space. The data is stored in a database as a user behavior log.

[0688] Step 3:

[0689] The server determines the actions of virtual creatures based on specific points within a commercial space, using analyzed user behavior data. The input is behavior log data, and the output is virtual creature response data. Based on the location information, a generative AI model calculates the appropriate response for the virtual creature.

[0690] Step 4:

[0691] The server sends the analyzed virtual creature's responses to the terminal. The input is the generated response data, and the output is the virtual creature's actions displayed on the terminal. The server responds to the terminal in real time, instructing the virtual creature to perform actions.

[0692] Step 5:

[0693] Users explore a commercial space while observing the reactions of virtual creatures. Input is action information from the server, and output is the user's observations. Users can acquire items and rewards. This makes visiting the commercial space more appealing.

[0694] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0695] This invention combines an emotion engine with a virtual animal raising system to achieve multifaceted monitoring that includes the user's emotions. The specific processing and operation of the programs in the server, terminal, and user elements are described below.

[0696] Server-side embodiment

[0697] The server receives operation information transmitted from the terminal and emotional data from the emotion engine. The server uses an AI model to comprehensively analyze this data. Through this analysis, it estimates the user's behavioral patterns and emotional state, and notifies the supervisor as needed.

[0698] Specific example: The server determines that the user is not showing much interest in the virtual animal and generates a notification stating "emotional state is low," which is then sent to the monitor.

[0699] Terminal-side embodiment

[0700] The terminal is equipped with an operation display device as an information and communication device, providing visual and auditory feedback for the user to interact with the virtual animal. An emotion engine analyzes the user's voice tone and facial expressions, and transmits the resulting emotion data to the server along with the operation information.

[0701] Specific example: If the device detects that the user's voice is cheerful, the virtual animal will move more actively and display fun animations.

[0702] User-side embodiment

[0703] Users perform the usual virtual animal raising operations through an app that incorporates an emotion engine. During this process, the user's emotional state is naturally analyzed, and the virtual animal's reactions are adjusted accordingly, resulting in a more personalized raising experience.

[0704] Specific example: If the user makes a sad sound while operating the system, the virtual animal will display a message on the screen saying, "Cheer up."

[0705] In this way, the system of the present invention can provide meticulous monitoring that takes into account the user's emotions and support that takes into account the user's psychological and emotional health. The server, terminal, and user all work together to provide a monitoring environment optimized for each individual.

[0706] The following describes the processing flow.

[0707] Step 1:

[0708] The user launches an application on their device. The user interacts with a virtual animal, performing everyday actions such as feeding and playing with it.

[0709] Step 2:

[0710] The device records user interactions in real time and saves data such as the type of operation and time. Simultaneously, the emotion engine analyzes the user's voice and facial expressions to generate data that estimates their emotional state.

[0711] Step 3:

[0712] The device collects operational information and emotional data, which is then sent to a server at regular time intervals or triggered by an event. A secure protocol with data encryption is used for this purpose.

[0713] Step 4:

[0714] The server analyzes the received data and uses an AI model to estimate the user's activity patterns and emotional trends. Based on these estimations, it determines the user's current situation.

[0715] Step 5:

[0716] The server generates and sends a notification to the monitor when it detects a condition requiring special attention (for example, if a user exhibits negative emotions for an extended period). This notification includes the estimated emotional state and recommended actions.

[0717] Step 6:

[0718] The monitor receives notifications from the server and considers appropriate actions based on the user's situation. If necessary, they contact the user directly and provide support.

[0719] Step 7:

[0720] Upon receiving feedback again from the server, the device will display appropriate actions and messages to the user, thereby promoting emotional support for the user.

[0721] Following this process, this system, which includes an emotion engine, will consider and monitor the user's emotions in real time.

[0722] (Example 2)

[0723] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0724] Conventional electronic entity nurturing systems, while progressing based on user input, have the challenge of failing to consider the user's emotional state and providing personalized support. In particular, they are unable to respond appropriately to changes in the user's emotions, resulting in a lack of effective monitoring and support.

[0725] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0726] In this invention, the server includes means for nurturing an electronic entity based on operations performed by an individual user, means for acquiring the user's voice and video and generating emotional information using an emotion analysis device, and means for estimating the user's behavioral patterns and emotional state by analyzing the generated emotional information and the operation information. This enables highly personalized monitoring and support based on the user's emotions.

[0727] An "information processing device" is a general term for electronic devices designed to input, analyze, process, and output data.

[0728] An "electronic entity" refers to a virtual character or object that can interact with the user within a digital environment.

[0729] "User" refers to an individual who operates the system and interacts with electronic entities.

[0730] "Emotional analysis" refers to a technology that identifies or estimates a user's emotional state based on data such as voice tone and facial expressions.

[0731] "Communication structure" is a general term for networks and communication protocols used to transmit data to remote devices or monitors.

[0732] "Behavioral patterns" refer to the tendencies of a user's actions and choices under specific conditions, and based on these, it is possible to predict the user's future behavior.

[0733] "Monitoring" means providing safety and security by monitoring the user's condition and activities, and providing support and notifications as needed.

[0734] This invention is a system that utilizes an information processing device to cultivate an electronic entity based on user operations. The system analyzes the user's emotions and provides appropriate feedback accordingly.

[0735] The server receives operation information sent from the terminal and sentiment data from the sentiment engine. The server uses a generative AI model to comprehensively analyze this data. During the analysis process, it is possible to use data analysis tools such as Python's Pandas and Scikit-learn. Based on the analysis results, the server estimates the user's behavior patterns and emotional state and notifies the monitor as needed. This is done, for example, using a prompt message such as, "Analyze the user's psychological tendencies based on their behavior patterns and sentiment data, and generate a warning if attention is needed."

[0736] The device provides visual and auditory feedback to support user interaction with the electronic entity. For voice tone and facial expression analysis, it can utilize facial recognition software such as OpenCV and the Google Cloud Speech-to-Text API for speech analysis. The device sends the acquired emotional data to the server as user interaction information. Furthermore, if the user expresses positive emotions, it provides visual effects such as playing an animated sequence in which the electronic entity moves actively.

[0737] Users nurture electronic beings through applications that incorporate an emotion engine. The user's emotional state is analyzed in a natural way, and the electronic being's responses are adjusted accordingly. For example, if the user interacts with the electronic being in a sad voice, it will display "Cheer up" and provide encouraging feedback. In this case, a prompt message such as "If a sad tone of voice is detected, display an encouraging message." is used.

[0738] Through this system, we can provide a personalized monitoring environment based on the individual user's emotions, resulting in a more nurturing and ultimately beneficial parenting experience for the user.

[0739] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0740] Step 1:

[0741] The user initiates interaction with the electronic entity through the device. In this step, the user's voice and video are input via the device's microphone and camera. The device collects this input data and prepares to send it to emotion analysis software. Specifically, when the user speaks to the electronic entity, their voice and facial expressions are captured in real time.

[0742] Step 2:

[0743] The device sends the collected audio and video data to the emotion analysis engine. The emotion analysis engine uses the Google Cloud Speech-to-Text API to perform audio analysis, converting the user's voice tone into text. It also uses OpenCV to analyze the user's facial expressions from the video. This data processing provides an indicator of the user's emotional state. The output is analyzed emotion data. Specifically, if the user is smiling, data indicating that a smile has been detected will be output.

[0744] Step 3:

[0745] The device sends the generated emotion data and the user's current action information to the server. This action information includes specific choices and actions the user has made within the app. As a specific action that the server takes to prepare to receive this input data and save it to its database, the device records the action "the user fed the pet" and sends it along with the emotion data.

[0746] Step 4:

[0747] The server combines emotional data and operational information received from the terminal to perform a comprehensive analysis. This analysis utilizes a generative AI model, using Python's Pandas and Scikit-learn to analyze data trends. By combining the input data, the user's behavior patterns and emotional state are estimated. The output is a prediction of the user's state. For example, the server might predict that "the user is likely excited at the moment."

[0748] Step 5:

[0749] The server determines specific actions based on the analysis results and sends notifications to the monitor as needed. The generated notification includes a prompt that clearly communicates the user's state to the monitor. Specifically, it executes the action, "The user's emotional state is elevated; please notify me by email if attention is required."

[0750] Step 6:

[0751] The device receives instructions from the server and provides feedback to the user. If the user expresses positive emotions, it plays visual animations, such as the electronic entity beginning to dance. Specifically, cheerful music plays and the electronic entity moves around energetically on the screen. This feedback allows the user to experience interactions that match their emotions.

[0752] (Application Example 2)

[0753] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0754] In modern times, virtual organism breeding systems utilizing information and communication devices are widespread, but many systems rely solely on user operation information, lacking sufficient monitoring and support that takes emotional states into account. Furthermore, there is a growing need to provide personalized user experiences by delivering content tailored to the user's emotions. This invention aims to provide a personalized monitoring environment and emotionally responsive support by analyzing the user's emotional state in real time and providing feedback and content based on that analysis.

[0755] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0756] In this invention, the server includes means for raising a virtual organism based on operations by an individual user, means for estimating the user's activity and emotional state using the user's operation information and emotion analysis functions, and means for indirectly providing monitoring and emotion-based content via an information communication device that transmits the analyzed information to a remote monitor. This enables a monitoring and entertainment experience optimized for each individual user by providing feedback and content suggestions that respond to the user's emotions.

[0757] An "information processing device" is a general term for devices that input, process, and output data, and within a system, it is a device that is responsible for analyzing user operation information and emotional data.

[0758] A "user" refers to an individual who uses this system to raise a virtual creature, and is the subject of emotional analysis.

[0759] A "virtual creature" is a digital pet or character that is artificially generated on a computer, and whose actions and reactions change based on the user's input.

[0760] The "emotion analysis function" is the ability to automatically estimate the user's emotional state using AI technology based on their facial expressions and voice data.

[0761] A "remote monitor" is someone who is physically located away from the user but whose role is to understand the user's status through an information processing device and to provide support and monitoring.

[0762] A "communication interface" is a physical or logical connection means that allows an information processing device to exchange information with other devices or networks.

[0763] "Monitoring" refers to actions or systems that monitor the user's situation and condition and provide appropriate support to maintain their safety and health.

[0764] "Content recommendation" is the act of recommending the most suitable information and entertainment resources based on the user's emotions and preferences.

[0765] The server operates on an information processing device and integrates and analyzes user operation information and emotional data. This system receives operation information and emotional analysis results transmitted from the user's smart device (such as a smartphone or tablet) and processes them using a cloud-based generative AI model. Specifically, it performs data calculations to estimate how the user interacts with the virtual creature and the user's emotional state.

[0766] The terminal serves as an interface for the user, providing visual and auditory interaction with the virtual being. Simultaneously, the terminal's built-in camera and microphone capture the user's facial expressions and voice tone, and this data is converted using emotion analysis. This data is sent to a server as input information for real-time estimation of the user's emotional state. When the user operates through the terminal, the system comprehensively analyzes this operation information and emotion data to provide situation-appropriate monitoring functions and content suggestions.

[0767] As users utilize this system on a daily basis, they can gain personalized experiences through their interactions with virtual beings. For example, if the system analyzes that a user is feeling stressed, relaxing music or videos will be suggested. This suggestion feature utilizes a generative AI model that predicts what kind of content the user is looking for based on their emotional state and makes the optimal selection.

[0768] An example of a prompt message might be, "Create an AI model that analyzes the user's facial expressions and voice to suggest relaxing content." This would enable more accurate content suggestions based on emotions, providing a more personalized entertainment experience.

[0769] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0770] Step 1:

[0771] The device collects user interaction and emotional data. Specifically, it captures the user's facial expressions with its built-in camera and records their voice tone with its microphone. This data is processed by an emotion analysis algorithm to generate information for estimating the user's emotional state. The inputs are camera footage and voice audio, and the output is the estimated emotional data.

[0772] Step 2:

[0773] The terminal transmits collected and analyzed operation information and sentiment data to the server. The transmitted data includes operation logs and sentiment analysis results related to user interactions. This process is a preparatory stage for comprehensive data analysis by the server. The input is locally analyzed operation information and sentiment data, and the output is data transfer to the server.

[0774] Step 3:

[0775] The server integrates the received data and uses a generative AI model to estimate the user's behavioral patterns and emotional states. Data processing includes time-series analysis to analyze temporal behavioral patterns and regression analysis to evaluate emotional states. Inputs are operation information and emotional data from the terminal, while output is estimated behavioral and emotional trend data.

[0776] Step 4:

[0777] The server selects the most suitable content using prompts based on estimated data. It retrieves content from the database that is appropriate for the user's emotional state, as predicted by the generative AI model, and generates suggestions. This step provides a personalized user experience by dynamically recommending content that matches the user's emotions. The input is behavioral and emotional trend data, and the output is content suggestion data.

[0778] Step 5:

[0779] Upon receiving suggestion data from the server, the terminal provides visual and auditory notifications to the user's screen. Specifically, it displays details and options for the suggested content and awaits user action. The input is the content suggestion data from the server, and the output is the content information displayed on the user interface.

[0780] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the controlled object 443 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0781] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0782] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the robot 414.

[0783] Furthermore, the emotion identification model 59, acting as an emotion engine, may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to a specific mapping, which is an emotion map (see Figure 9). Similarly, the emotion identification model 59 may also determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[0784] Figure 9 shows an emotion map 400 in which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. The closer to the center of the concentric circles, the more primitive the emotions are located. Further out of the concentric circles, emotions representing states and actions arising from mental states are located. Emotion is a concept that includes feelings and mental states. On the left side of the concentric circles, emotions that are generally generated from reactions occurring in the brain are located. On the right side of the concentric circles, emotions that are generally induced by situational judgment are located. In the upper and lower directions of the concentric circles, emotions that are generally generated from reactions occurring in the brain and induced by situational judgment are located. Also, the upper side of the concentric circles is where "pleasant" emotions are located, and the lower side is where "unpleasant" emotions are located. In this way, in the emotion map 400, multiple emotions are mapped based on the structure in which emotions arise, and emotions that are likely to occur simultaneously are mapped close together.

[0785] These emotions are distributed at the 3 o'clock position on the Emotion Map 400, and usually fluctuate between feelings of security and anxiety. In the right half of the Emotion Map 400, situational awareness takes precedence over internal feelings, resulting in a calm impression.

[0786] The inside of the Emotion Map 400 represents inner thoughts, while the outside represents actions. Therefore, the further you go from the outside of the Emotion Map 400, the more visible (expressed in actions) your emotions become.

[0787] Here, human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. Similarly, in robots, cars, motorcycles, etc., emotions can be created based on various balances, such as posture and battery level. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. The emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on a system for analyzing brain physiological signals of speech emotion recognition and emotion, Tokushima University, doctoral dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map contains emotions belonging to a region called "response," where sensation is dominant. The right half of the emotion map contains emotions belonging to a region called "situation," where situational awareness is dominant.

[0788] The emotion map defines two emotions that promote learning. One is the emotion around the middle of the negative "repentance" and "reflection" on the situation side. In other words, it is when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is the emotion around the positive "desire" on the reaction side. In other words, it is when the robot has positive feelings such as "I want more" or "I want to know more."

[0789] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​representing each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple training data sets, which are combinations of user input and emotion values ​​representing each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions located close together have similar values, as shown in the emotion map 900 in Figure 10. Figure 10 shows an example where multiple emotions such as "reassured," "calm," and "confident" have similar emotion values.

[0790] The above description primarily focuses on the functions of the data processing device 12 in relation to this disclosure. However, the system related to this disclosure is not necessarily implemented on a server. The system related to this disclosure may be implemented as a general information processing system. This disclosure may be implemented, for example, as a software program that runs on a personal computer or as an application that runs on a smartphone. The method related to this disclosure may be provided to users in SaaS (Software as a Service) format.

[0791] In the above embodiment, an example was given in which a specific process is performed by a single computer 22. However, the technology of this disclosure is not limited thereto, and a distributed processing of the specific process may be performed by multiple computers, including computer 22. For example, a data generation model 58 may be provided in an external device of the data processing device 12, and the external device may generate data according to the input data.

[0792] In the above embodiment, an example was given in which the specific processing program 56 is stored in the storage 32, but the technology of this disclosure is not limited thereto. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-temporary storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-temporary storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes specific processing according to the specific processing program 56.

[0793] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[0794] Furthermore, it is not necessary to store the entirety of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store the entirety of the specific processing program 56 in the storage 32; it is acceptable to store only a portion of the specific processing program 56.

[0795] The following types of processors can be used as hardware resources to perform specific processing. Examples of processors include a CPU, a general-purpose processor that functions as a hardware resource to perform specific processing by executing software, i.e., a program. Other examples of processors include dedicated electrical circuits, such as FPGAs (Field-Programmable Gate Arrays), PLDs (Programmable Logic Devices), or ASICs (Application Specific Integrated Circuits), which have circuit configurations specifically designed to perform specific processing. All of these processors have built-in or connected memory, and all of them perform specific processing by using memory.

[0796] The hardware resource that performs a specific process may consist of one of these various processors, or it may consist of a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Alternatively, the hardware resource that performs a specific process may consist of a single processor.

[0797] Examples of configurations using a single processor include, firstly, a configuration in which one or more CPUs and software are combined to form a single processor, and this processor functions as a hardware resource that performs a specific process. Secondly, there is a configuration using a processor that realizes the functions of the entire system, including multiple hardware resources that perform a specific process, on a single IC chip, as exemplified by SoCs (System-on-a-chip). In this way, a specific process is realized using one or more of the above types of processors as hardware resources.

[0798] Furthermore, the hardware structure of these various processors can more specifically utilize electrical circuits that combine circuit elements such as semiconductor devices. Also, the specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps can be deleted, new steps added, or the processing order rearranged, as long as it does not deviate from the main purpose.

[0799] The descriptions and illustrations presented above are detailed explanations of the technical aspects of this disclosure and are merely examples of the technical aspects. For example, the above descriptions of the structure, function, operation, and effect are examples of the structure, function, operation, and effect of the technical aspects of this disclosure. Therefore, it goes without saying that you may delete unnecessary parts, add new elements, or replace elements in the descriptions and illustrations presented above, as long as you do not deviate from the essence of the technical aspects of this disclosure. Furthermore, in order to avoid confusion and facilitate understanding of the technical aspects of this disclosure, explanations of common technical knowledge and the like that do not require special explanation to enable the implementation of the technical aspects of this disclosure have been omitted from the descriptions and illustrations presented above.

[0800] All documents, patent applications, and technical standards described herein are incorporated by reference to the same extent as if each individual document, patent application, and technical standard were specifically and individually noted to be incorporated by reference.

[0801] The following is further disclosed regarding the embodiments described above.

[0802] (Claim 1)

[0803] A means for raising virtual animals based on operations performed by individual users, provided in an information and communication device,

[0804] A means for generating user operation information and estimating the user's activities by analyzing said operation information,

[0805] A means of indirectly providing monitoring through an information processing device that transmits the analyzed information to a remote monitor,

[0806] The information is communicated to the monitor using an existing communication interface.

[0807] A system that includes this.

[0808] (Claim 2)

[0809] The system according to claim 1, comprising an operation display device for providing visual and auditory interaction between the user and the virtual animal.

[0810] (Claim 3)

[0811] The system according to claim 1, further comprising the information and communication device that periodically monitors the state of the virtual animal and issues a warning to the monitor if an abnormality is detected.

[0812] "Example 1"

[0813] (Claim 1)

[0814] A means for managing virtual pets based on user operations, provided in an information processing device,

[0815] A means for generating user operation data, analyzing said operation data using a generated AI model, and estimating the user's behavioral state,

[0816] A means for providing a method for generating and sending notifications to a remote monitor based on analysis results,

[0817] The information is transmitted to the monitor using existing data communication means.

[0818] A system that includes this.

[0819] (Claim 2)

[0820] The system according to claim 1, further comprising an operation display means for providing visual and auditory interaction between the user and the virtual pet.

[0821] (Claim 3)

[0822] The system according to claim 1, further comprising the information processing device that periodically observes the status of a virtual pet and notifies a monitor if an abnormality is identified.

[0823] "Application Example 1"

[0824] (Claim 1)

[0825] A means for raising virtual organisms based on operations performed by individual users on an information processing device,

[0826] A means for generating user operation information and estimating the user's activities by analyzing said operation information,

[0827] A means of indirectly providing monitoring through a computing device that transmits the analyzed information to a remote monitor,

[0828] A means for informing the monitor of the information using an existing communication interface,

[0829] A means of recording the visitor status of users in a commercial space and having a virtual organism react according to the user's current location,

[0830] A system that includes this.

[0831] (Claim 2)

[0832] The system according to claim 1, comprising an operation display device for providing visual and auditory interaction between the user and the virtual creature.

[0833] (Claim 3)

[0834] The system according to claim 1, further comprising the information processing device that periodically monitors the state of the virtual organism and issues a warning to the monitor if an abnormality is detected.

[0835] "Example 2 of combining an emotion engine"

[0836] (Claim 1)

[0837] A means for nurturing an electronic entity based on operations performed by an individual user, provided in an information processing device,

[0838] A means for acquiring the voice and video of the user and generating emotional information using a device that performs emotion analysis,

[0839] A means for estimating the user's behavioral patterns and emotional state by analyzing the generated emotional information and the operation information,

[0840] A means of indirectly providing monitoring through a data processing device that transmits the analyzed information to a remote monitor,

[0841] A method characterized by using an existing communication structure to inform the monitor of the information,

[0842] A system that includes this.

[0843] (Claim 2)

[0844] The system according to claim 1, comprising an operation display device for providing visual and auditory interaction between a user and the electronic entity.

[0845] (Claim 3)

[0846] The system according to claim 1, comprising the information processing device that periodically monitors the state of the electronic entity and issues a warning to the monitor if an abnormality is detected.

[0847] "Application example 2 when combining with an emotional engine"

[0848] (Claim 1)

[0849] A means for cultivating a virtual organism based on operations performed by an individual user, provided in an information processing device,

[0850] A means for generating user operation information and estimating the user's activity and emotional state using said operation information and emotion analysis function,

[0851] A means of indirectly providing monitoring and emotion-based content via an information and communication device that transmits the analyzed information to a remote monitor,

[0852] This method is characterized by using an existing communication interface to inform the monitor of the information and to provide information appropriate to their emotions.

[0853] A system that includes this.

[0854] (Claim 2)

[0855] The system according to claim 1, comprising an operation display device for providing visual and auditory interaction between the user and the virtual creature, and for presenting information based on the user's emotions.

[0856] (Claim 3)

[0857] The system according to claim 1, further comprising the information processing device that periodically monitors the state of the virtual organism and issues a warning to the monitor if an abnormality is detected or if the user's emotional changes exceed a certain threshold. [Explanation of Symbols]

[0858] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Devices 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robots< / url:> < / url:> < / url:> < / url:>

Claims

1. A means for raising virtual animals based on operations performed by individual users, provided in an information and communication device, A means for generating user operation information and estimating the user's activities by analyzing said operation information, A means of indirectly providing monitoring through an information processing device that transmits the analyzed information to a remote monitor, The information is communicated to the monitor using an existing communication interface. A system that includes this.

2. The system according to claim 1, comprising an operation display device for providing visual and auditory interaction between the user and the virtual animal.

3. The system according to claim 1, further comprising the information and communication device that periodically monitors the state of the virtual animal and issues a warning to the monitor if an abnormality is detected.

Images