System

Abstract

The purpose of the present invention is to analyze a user's action or posture in real time and provide appropriate suggestions. A system according to an embodiment includes an imaging unit, an analysis unit, and a provision unit. The imaging unit captures an action of a user. The analysis unit analyzes the data captured by the imaging unit. The provision unit provides suggestions in real time on the basis of the analysis results obtained by the analysis unit.

Description

TECHNICAL FIELD

[0001] The technology of the present disclosure relates to a system. BACKGROUND

[0002] Patent Document 1 discloses a personalized chatbot control method executed by at least one processor, the method including: a step of receiving a user utterance; a step of adding the user utterance to a prompt containing an instruction of a description related to a chatbot character; a step of encoding the prompt; and a step of inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance.

[0003]

Patent Document 1

[0004] In the related art, there is a problem that a user's action or posture cannot be sufficiently analyzed in real time and appropriate advice cannot be provided.

[0005] The system of the present technology aims to analyze a user's action or posture in real time and provide appropriate advice.

[0006] The system of the present technology includes an imaging unit, an analysis unit, and a providing unit. The imaging unit is configured to capture a user's action. The analysis unit is configured to analyze data captured by the imaging unit. The providing unit is configured to provide advice in real time based on an analysis result obtained by the analysis unit.

[0007] The system of the present technology can analyze a user's action or posture in real time and provide appropriate advice. BRIEF DESCRIPTION OF DRAWINGS

[0008] Figure 1 is a conceptual diagram showing an example of a data processing system configuration of the first embodiment.

[0009] Figure 2 is a conceptual diagram showing an example of main functions of a data processing device and a smart device of the first embodiment.

[0010] Figure 3 is a conceptual diagram showing an example of a data processing system configuration of the second embodiment.

[0011] Figure 4 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses of the second embodiment.

[0012] Figure 5 is a conceptual diagram showing an example of a data processing system configuration of the third embodiment.

[0013] Figure 6is a conceptual diagram showing an example of the main functions of a data processing device and a head-mounted terminal according to the third embodiment.

[0014] Figure 7 is a conceptual diagram showing an example of the configuration of a data processing system according to the fourth embodiment.

[0015] Figure 8 is a conceptual diagram showing an example of the main functions of a data processing device and a robot according to the fourth embodiment.

[0016] Figure 9 is an emotion map in which a plurality of emotions are mapped.

[0017] Figure 10 is an emotion map in which a plurality of emotions are mapped.

[0018] Explanation of Reference Signs

[0019] 10, 210, 310, 410 data processing system

[0020] 12 data processing device

[0021] 14 intelligent device

[0022] 214 intelligent glasses

[0023] 314 head-mounted terminal

[0024] 414 robot DETAILED DESCRIPTION

[0025] Hereinafter, an example of an embodiment of a system to which the present technology is applied will be described with reference to the drawings.

[0026] First, the words used in the following description will be explained.

[0027] In the following embodiment, the processor (hereinafter, simply referred to as "processor") with a reference sign can be one arithmetic device or a combination of a plurality of arithmetic devices. In addition, the processor can be one arithmetic device or a combination of a plurality of arithmetic devices. An example of the arithmetic device includes a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), an APU (Accelerated Processing Unit), or a TPU (Tensor Processing Unit), and the like.

[0028] In the following embodiments, the RAM (Random Access Memory) denoted by the reference sign is a memory that temporarily stores information, and is used as a work memory by a processor.

[0029] In the following embodiments, the memory denoted by the reference sign is one or a plurality of nonvolatile storage devices for storing various programs and various parameters, and the like. An example of the nonvolatile storage device includes a flash memory (SSD (Solid State Drive)), a magnetic disk (for example, a hard disk), a magnetic tape, or the like.

[0030] In the following embodiments, the communication I / F (Interface) denoted by the reference sign is an interface including a communication processor and an antenna, and the like. The communication I / F is responsible for communication between a plurality of computers. An example of the communication standard suitable for the communication I / F includes a wireless communication standard such as 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), or Bluetooth (registered trademark).

[0031] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B". That is, "A and / or B" means that only A can be present, only B can be present, or a combination of A and B can be present. Furthermore, in the present specification, when three or more items are described using "and / or" to connect, the same idea as "A and / or B" is also applied.

[0032] First Embodiment

[0033] Figure 1 An example of the configuration of the data processing system 10 of the first embodiment is shown.

[0034] As shown in Figure 1 , the data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0035] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 includes a processor 28, a RAM 30, and a memory 32. The processor 28, the RAM 30, and the memory 32 are connected to a bus 34. Furthermore, the database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. An example of the network 54 includes a WAN (Wide Area Network) and / or a LAN (Local Area Network), and the like.

[0036] The smart device 14 is provided with a computer 36, a receiving device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 is provided with a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. Further, the receiving device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0037] The receiving device 38 is provided with a touch screen 38A and a microphone 38B or the like, and is used to receive user input. The touch screen 38A receives user input generated by contact of a pointing body (e.g., a pen or a finger or the like) by detecting the contact of the pointing body. The microphone 38B receives user input generated by sound by detecting the sound of the user. The control section 46A transmits data indicating the user input received by the touch screen 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing section 290 (refer to Figure 2 ) acquires the data indicating the user input.

[0038] The output device 40 is provided with a display 40A and a speaker 40B or the like, and presents data to the user by outputting data in a perceptible form (e.g., sound and / or text) to the user. The display 40A displays visual information such as text and images in accordance with an instruction from the processor 46. The speaker 40B outputs sound in accordance with an instruction from the processor 46. The camera 42 is a small digital camera equipped with an optical system such as a lens, an aperture, and a shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0039] The communication I / F 44 is connected to a network 54. The communication I / F 44 and 26 are responsible for the transmission and reception of various information between the processor 46 and the processor 28 via the network 54.

[0040] Figure 2 An example of the main functions of the data processing device 12 and the smart device 14 is shown.

[0041] As shown in Figure 2 , in the data processing device 12, specific processing is performed by the processor 28. The specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of the "program" to which the technology of the present disclosure relates. The processor 28 reads out the specific processing program 56 from the storage 32 and executes the read-out specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 acting as the specific processing section 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0042] The data generation model 58 and the emotion-specific model 59 are stored in the memory 32. The data generation model 58 and the emotion-specific model 59 are used by the specific processing section 290. The specific processing section 290 can use the emotion-specific model 59 to infer the emotion of the user, and perform specific processing using the emotion of the user. The emotion inference function (emotion-specific function) using the emotion-specific model 59 includes, but is not limited to, for example, inference and prediction of the emotion of the user, and analysis (resolution) of the emotion, and the like. The inference and prediction of the emotion include, for example, inference and prediction of the emotion of the user, and analysis (resolution) of the emotion, and the like.

[0043] In the smart device 14, specific processing is performed by the processor 46. The specific processing program 60 is stored in the memory 50. The specific processing program 60 is used by the data processing system 10 in conjunction with the specific processing program 56. The processor 46 reads out the specific processing program 60 from the memory 50, and executes the read-out specific processing program 60 on the RAM 48. The specific processing is achieved by the processor 46 acting as the control section 46A in accordance with the specific processing program 60 executed on the RAM 48. In addition, the smart device 14 can also have the same data generation model and emotion-specific model as the data generation model 58 and the emotion-specific model 59, and perform the same processing as the specific processing section 290 using these models.

[0044] In addition, other devices than the data processing apparatus 12 can also have the data generation model 58. For example, a server apparatus (for example, a generation server) can have the data generation model 58. In this case, the data processing apparatus 12 obtains a processing result (prediction result, etc.) using the data generation model 58 by communicating with the server apparatus having the data generation model 58. In addition, the data processing apparatus 12 can be a server apparatus, or a terminal apparatus (for example, a mobile phone, a robot, a home appliance, etc.) owned by a user. Next, an example of the processing performed by the data processing system 10 of the first embodiment will be described.

[0045] Embodiment 1

[0046] The system of the embodiment of the present invention is a system that enables a user to experience the view of a top athlete or a master artisan (a Japanese official certification title for those who possess top-notch traditional skills) through VR glasses. The system allows the user to wear the VR glasses and experience the view of a top athlete or an artisan. In this process, a camera built into the VR glasses captures the user's movements or posture in real time. Then, a generative AI analyzes the captured data to determine how the user's movements or posture differ from those of the master. Finally, the generative AI provides real-time advice based on the analysis results. As a result, the user can learn the master's techniques and improve their own movements. Through this service, anyone can experience and learn top techniques, which helps to improve sports or artistic skills. For example, the user wears the VR glasses and experiences the view of a top athlete or an artisan. For example, the user can watch a game from the perspective of an athlete or experience the process of creating a work from the perspective of an artisan. In this process, a camera built into the VR glasses captures the user's movements or posture in real time. Then, a generative AI analyzes the captured data to determine how the user's movements or posture differ from those of the master. For example, from the perspective of an athlete, the generative AI analyzes how the user's angle or timing of movements differs from those of the master. Finally, the generative AI provides real-time advice based on the analysis results. For example, the generative AI provides specific advice such as "you can improve the angle of your movements" or "make your hand movements more fluid" to the user. As a result, the user can learn the master's techniques and improve their own movements. Through this service, anyone can experience and learn top techniques, which helps to improve sports or artistic skills. Therefore, the system using VR glasses enables a user to experience and learn top techniques, which helps to improve sports or artistic skills.

[0047] The system of the embodiment of the present invention has a camera unit, an analysis unit, and a provision unit. The camera unit is used to capture the user's movements. The camera unit, for example, can capture the user's movements or posture in real time. For example, the camera unit can capture the user's movements while walking. In addition, the camera unit can also capture the user's movements while running. The camera unit can also capture the user's posture while standing. For example, the camera unit can also capture the user's posture while sitting. The analysis unit is used to analyze the captured data. The analysis unit, for example, can determine how the user's movements or posture differ from those of the master. For example, the analysis unit can analyze how the user's angle or timing of movements differs from those of the master. In addition, the analysis unit can also analyze how the user's speed or rhythm of movements differs from those of the master. The analysis unit can also analyze how the stability or balance of the user's posture differs from that of the master. The provision unit provides real-time advice based on the analysis results. The provision unit, for example, can provide specific advice such as "you can improve the angle of your movements" or "make your hand movements more fluid" to the user. In addition, the provision unit can also provide advice such as "you can speed up your movements" to the user. The provision unit can also provide advice such as "make the balance of your posture more stable" to the user. As a result, the system of the embodiment of the present invention supports the user's skill improvement by capturing, analyzing, and providing advice on the user's movements in real time.

[0048] The imaging section can capture the user's movement or posture in real time. The imaging section can capture the user's movement in real time, for example. The imaging section can capture the user's movement while walking, for example. In addition, the imaging section can also capture the user's movement while running. The imaging section can also capture the user's posture while standing. The imaging section can also capture the user's posture while sitting, for example. Thus, by capturing the user's movement or posture in real time, accurate data can be obtained. Part or all of the above processing in the imaging section can be implemented by AI, or can be performed without using AI. For example, the imaging section can input the captured user movement data to a generative AI, and perform data analysis by the generative AI.

[0049] The analysis section can analyze the captured data and determine how the user's movement or posture differs from that of the master. The analysis section can analyze the captured data and determine how the user's movement or posture differs from that of the master, for example. The analysis section can determine how the angle or timing of the user's movement differs from that of the master, for example. In addition, the analysis section can also determine how the speed or rhythm of the user's movement differs from that of the master. The analysis section can also determine how the stability or balance of the user's posture differs from that of the master. Thus, by determining how the user's movement or posture differs from that of the master, specific points for improvement can be clarified. Part or all of the above processing in the analysis section can be implemented by a generative AI, or can be performed without using a generative AI. For example, the analysis section can input the captured data to a generative AI, and perform data analysis by the generative AI.

[0050] The providing section can provide advice in real time based on the analysis result. The providing section can provide advice in real time based on the analysis result, for example. The providing section can provide the user with specific advice such as "increase the movement angle" or "make the hand movement smoother", for example. In addition, the providing section can also provide the user with advice such as "increase the movement speed". The providing section can also provide the user with advice such as "make the balance of the posture more stable". Thus, by providing advice in real time, the user can immediately grasp the points for improvement. Part or all of the above processing in the providing section can be implemented by a generative AI, or can be performed without using a generative AI. For example, the providing section can input the analysis result to a generative AI, and generate advice by the generative AI.

[0051] The providing unit can provide the user with specific suggestion content. For example, the providing unit can provide the user with specific suggestion content. For example, the providing unit can provide the user with a specific suggestion such as "the action angle can be further improved" or "make the hand movement more fluid". In addition, the providing unit can also provide the user with a suggestion such as "the action speed can be further accelerated". The providing unit can also provide the user with a suggestion such as "make the balance of the posture more stable". In this way, by providing specific suggestions, the user can understand the specific improvement method. Part or all of the above processing in the providing unit can be implemented by generative AI, or generative AI can not be used. For example, the providing unit can input the analysis result into the generative AI, and the generative AI generates the suggestion.

[0052] The system has a camera unit that analyzes the user's past action history and selects an appropriate capture opportunity. For example, the camera unit can analyze the user's past action history and select an appropriate capture opportunity. For example, analyze the timing of the user's past actions, and start capturing when similar actions occur. In addition, the frequency of occurrence of a specific action can also be analyzed according to the user's past action history, and the capture opportunity can be adjusted accordingly. The start and end timing of the action can also be predicted based on the user's past action history, and the optimal capture opportunity can be selected. In this way, by selecting the optimal capture opportunity based on the past action history, important moments can not be missed. Part or all of the above processing in the camera unit can be implemented by AI, or AI can not be used. For example, the camera unit can input the user's past action history into the generative AI, and the generative AI selects the optimal capture opportunity.

[0053] The system has a camera unit that can emphasize a specific action or posture when capturing the user's action. For example, the camera unit can emphasize a specific action or posture when capturing the user's action. For example, when the user performs a specific action, the action can be emphasized by adjusting the camera focus for capture. In addition, when the user adopts a specific posture, the posture can also be emphasized by the zoom function of the camera for capture. The action can also be emphasized by adjusting the camera exposure when the user performs a specific action for capture. In this way, by emphasizing a specific action or posture for capture, important actions can be analyzed in detail. Part or all of the above processing in the camera unit can be implemented by AI, or AI can not be used. For example, the camera unit can input the captured user action data into the generative AI, and the generative AI emphasizes the specific action or posture for capture.

[0054] The system has a camera unit that captures actions with high relevance in consideration of user geographical location information. The camera unit, for example, can capture actions with high relevance in consideration of user geographical location information. For example, when the user is at a specific sports facility, actions related to the sports can be captured preferentially. In addition, when the user is at a specific workshop, technical actions performed in the workshop can also be captured preferentially. When the user is at a specific activity site, actions related to the activity can also be captured preferentially. In this way, by considering geographical location information, actions with high relevance can be captured preferentially. Part or all of the above processing in the camera unit can be implemented by AI, or AI can not be used. For example, the camera unit can input user geographical location information into a generative AI, and the generative AI can select actions with high relevance.

[0055] The system has a camera unit that captures actions related to the user by analyzing user social media activities. The camera unit, for example, can capture actions related to the user by analyzing user social media activities. For example, videos or photos shared by the user on social media can be analyzed to capture related actions. In addition, the activities of accounts followed by the user on social media can also be analyzed to capture related actions. Activities participated in by the user on social media can also be analyzed to capture related actions. In this way, by analyzing social media activities, actions related to the user can be captured. Part or all of the above processing in the camera unit can be implemented by AI, or AI can not be used. For example, the camera unit can input user social media activities into a generative AI, and the generative AI can select related actions.

[0056] The analysis unit can adjust the level of detail of analysis according to the importance of actions when analyzing captured data. The analysis unit, for example, can adjust the level of detail of analysis according to the importance of actions when analyzing captured data. For example, for important actions, detailed analysis is performed to provide detailed analysis results. For actions with lower importance, brief analysis can also be performed to provide basic analysis results. The level of detail of analysis can also be adjusted in stages according to the importance of actions to provide appropriate analysis results. In this way, by adjusting the level of detail of analysis according to the importance of actions, important actions can be analyzed in detail. Part or all of the above processing in the analysis unit can be implemented by generative AI, or AI can not be used. For example, the analysis unit can input captured data into a generative AI, and the generative AI can adjust the level of detail of analysis according to the importance of actions.

[0057] The analysis unit can apply different analysis algorithms according to the category of the action when analyzing the captured data. For example, the analysis unit can apply different analysis algorithms according to the category of the action when analyzing the captured data. For example, for sports actions, a dedicated sports analysis algorithm is applied. For process actions, a dedicated process analysis algorithm can also be applied. For artistic actions, a dedicated artistic analysis algorithm can also be applied. In this way, by applying different analysis algorithms according to the category of the action, more appropriate analysis results can be provided. Part or all of the above processing in the analysis unit can be implemented by generative AI, or generative AI can not be used. For example, the analysis unit can input the captured data into the generative AI, and the generative AI can apply different analysis algorithms according to the category of the action.

[0058] The analysis unit can determine the priority of analysis according to the submission time of the action when analyzing the captured data. For example, the analysis unit can determine the priority of analysis according to the submission time of the action when analyzing the captured data. For example, the most recently captured data is analyzed first, providing the latest analysis results. For data submitted earlier, the analysis priority can also be reduced. The priority of analysis can also be adjusted in stages according to the submission time to provide appropriate analysis results. In this way, by determining the priority of analysis according to the submission time, the latest data can be analyzed first. Part or all of the above processing in the analysis unit can be implemented by generative AI, or generative AI can not be used. For example, the analysis unit can input the captured data into the generative AI, and the generative AI can determine the priority of analysis according to the submission time of the action.

[0059] The analysis unit can adjust the order of analysis according to the relevance of the action when analyzing the captured data. For example, the analysis unit can adjust the order of analysis according to the relevance of the action when analyzing the captured data. For example, actions with high relevance are analyzed first, providing detailed analysis results. For actions with lower relevance, the analysis order can also be moved back. The order of analysis can also be adjusted in stages according to the relevance of the action to provide appropriate analysis results. In this way, by adjusting the order of analysis according to the relevance of the action, actions with high relevance can be analyzed first. Part or all of the above processing in the analysis unit can be implemented by generative AI, or generative AI can not be used. For example, the analysis unit can input the captured data into the generative AI, and the generative AI can adjust the order of analysis according to the relevance of the action.

[0060] The providing unit can adjust the level of detail according to the importance of the suggestion when providing the suggestion based on the analysis result. For example, the providing unit can adjust the level of detail according to the importance of the suggestion when providing the suggestion based on the analysis result. For example, for important suggestions, the suggestion including detailed information is provided. For suggestions with lower importance, the suggestion including brief information can also be provided. The level of detail can also be adjusted in stages according to the importance of the suggestion, and appropriate suggestions are provided. In this way, by adjusting the level of detail according to the importance of the suggestion, important suggestions can be provided in detail. Part or all of the above processes in the providing unit can be implemented by generative AI, or generative AI can not be used. For example, the providing unit can input the analysis result into the generative AI, and the generative AI adjusts the level of detail according to the importance of the suggestion.

[0061] The providing unit can apply different suggestion algorithms according to the category of the action when providing the suggestion based on the analysis result. For example, the providing unit can apply different suggestion algorithms according to the category of the action when providing the suggestion based on the analysis result. For example, for sports actions, a dedicated sports suggestion algorithm is applied. For process actions, a dedicated process suggestion algorithm can also be applied. For artistic actions, a dedicated artistic suggestion algorithm can also be applied. In this way, by applying different suggestion algorithms according to the category of the action, more appropriate suggestions can be provided. Part or all of the above processes in the providing unit can be implemented by generative AI, or generative AI can not be used. For example, the providing unit can input the analysis result into the generative AI, and the generative AI applies different suggestion algorithms according to the category of the action.

[0062] The providing unit can determine the priority of the suggestion according to the submission timing of the action when providing the suggestion based on the analysis result. For example, the providing unit can determine the priority of the suggestion according to the submission timing of the action when providing the suggestion based on the analysis result. For example, suggestions based on recently captured data are provided preferentially. The priority of suggestions corresponding to data with earlier submission timing can also be reduced. The priority of the suggestion can also be adjusted in stages according to the submission timing, and appropriate suggestions are provided. In this way, by determining the priority of the suggestion according to the submission timing, suggestions based on the latest data can be provided preferentially. Part or all of the above processes in the providing unit can be implemented by generative AI, or generative AI can not be used. For example, the providing unit can input the analysis result into the generative AI, and the generative AI determines the priority of the suggestion according to the submission timing of the action.

[0063] The providing unit can adjust the order of the suggestions according to the relevance of the actions when providing the suggestions based on the analysis result. For example, the providing unit can adjust the order of the suggestions according to the relevance of the actions when providing the suggestions based on the analysis result. For example, the suggestions for actions with high relevance can be provided preferentially. The order of the suggestions for actions with low relevance can also be moved backward. The order of the suggestions can also be adjusted in stages according to the relevance of the actions, and appropriate suggestions can be provided. In this way, by adjusting the order of the suggestions according to the relevance of the actions, the suggestions for actions with high relevance can be provided preferentially. Part or all of the above processing in the providing unit can be implemented by generative AI, or generative AI can not be used. For example, the providing unit can input the analysis result into generative AI, and the generative AI can adjust the order of the suggestions according to the relevance of the actions.

[0064] The system of the present embodiment is not limited to the above examples, and various modifications can be made, for example, as follows.

[0065] The system can be provided with a voice capturing unit that simultaneously records environmental sound when capturing the user's actions. For example, the surrounding sound when the user is performing a sports activity is recorded, and the voice data is analyzed together with the action analysis. In addition, the sound of the tool when the user is performing a craftwork can also be recorded, and the accuracy and rhythm of the action can be analyzed. The sound of the musical instrument when the user is playing can also be recorded, and used to improve the playing technique. In this way, by analyzing in combination with the action and voice data, more detailed feedback can be provided.

[0066] The system can be provided with an environmental sensor unit that simultaneously records environmental data such as temperature, humidity, etc. when capturing the user's actions. For example, the surrounding temperature and humidity when the user is performing a sports activity are recorded, and the environmental data is analyzed together with the action analysis. In addition, the temperature and humidity of the working environment when the user is performing a craftwork can also be recorded, and factors affecting the efficiency and accuracy of the work can be analyzed. The temperature and humidity of the playing environment when the user is playing can also be recorded, and used to improve the playing technique. In this way, by analyzing in combination with the action and environmental data, more detailed feedback can be provided.

[0067] The system can be provided with a biological sensor unit that simultaneously records biological data such as heart rate, breathing rate, etc. when capturing the user's actions. For example, the heart rate and breathing rate when the user is performing a sports activity are recorded, and the biological data is analyzed together with the action analysis. In addition, the heart rate and breathing rate when the user is performing a craftwork can also be recorded, and the concentration and fatigue of the work can be analyzed. The heart rate and breathing rate when the user is playing can also be recorded, and used to improve the playing technique. In this way, by analyzing in combination with the action and biological data, more detailed feedback can be provided.

[0068] The system can have a line-of-sight tracking section that records the user's line-of-sight movement at the same time as capturing the user's action. For example, the line-of-sight movement of the user when performing a sports activity is recorded, and the line-of-sight data is analyzed together with the action analysis. In addition, the line-of-sight movement of the user when performing a craft work is also recorded, and the concentration and line-of-sight movement pattern of the work are analyzed. The line-of-sight movement of the user when playing is also recorded, and is used to improve the playing technique. Thus, by analyzing in combination with the action and line-of-sight data, more detailed feedback can be provided.

[0069] The system can have an electromyography sensor section that records the user's electromyography at the same time as capturing the user's action. For example, the electromyography of the user when performing a sports activity is recorded, and the electromyography data is analyzed together with the action analysis. In addition, the electromyography of the user when performing a craft work is also recorded, and the accuracy and force degree of the work are analyzed. The electromyography of the user when playing is also recorded, and is used to improve the playing technique. Thus, by analyzing in combination with the action and electromyography data, more detailed feedback can be provided.

[0070] Hereinafter, the processing flow of Embodiment 1 will be briefly described.

[0071] Step 1: The imaging section captures the user's action. The imaging section can capture the user's action or posture in real time, for example. For example, the imaging section can capture the user's action when walking. In addition, the imaging section can also capture the user's action when running. The imaging section can also capture the user's posture when standing. For example, the imaging section can also capture the user's posture when sitting.

[0072] Step 2: The analysis section analyzes the captured data. The analysis section can determine, for example, how the user's action or posture differs from that of a master. For example, the analysis section can analyze how the angle or timing of the user's action differs from that of a master. In addition, the analysis section can also analyze how the speed or rhythm of the user's action differs from that of a master. The analysis section can also analyze how the stability or balance of the user's posture differs from that of a master.

[0073] Step 3: The providing section provides suggestions in real time based on the analysis results. The providing section can provide specific suggestions such as "the action angle can be further improved" or "the action of the hand can be made smoother" to the user, for example. In addition, the providing section can also provide suggestions such as "the action speed can be further increased" to the user. The providing section can also provide suggestions such as "the balance of the posture can be made more stable" to the user.

[0074] Embodiment 2

[0075] The system of the embodiment of the present invention is a system that enables a user to experience the view of a top athlete or a national treasure through VR glasses. The system allows the user to wear the VR glasses and experience the view of a top athlete or a craftsman. In this process, a camera built into the VR glasses captures the user's movements or posture in real time. Then, a generative AI analyzes the captured data to determine how the user's movements or posture differ from those of the master. Finally, the generative AI provides real-time advice based on the analysis results. As a result, the user can learn the master's techniques and improve their own movements. Through this service, anyone can experience and learn top techniques, which helps to improve sports or artistic skills. For example, the user wears the VR glasses and experiences the view of a top athlete or a craftsman. For example, the user can watch a game from the perspective of an athlete or experience the process of creating a work from the perspective of a craftsman. In this process, a camera built into the VR glasses captures the user's movements or posture in real time. Then, a generative AI analyzes the captured data to determine how the user's movements or posture differ from those of the master. For example, from the perspective of an athlete, the generative AI analyzes how the user's angle or timing of movements differs from those of the master. Finally, the generative AI provides real-time advice based on the analysis results. For example, the generative AI provides specific advice such as "you can improve the angle of your movements" or "make your hand movements more fluid." As a result, the user can learn the master's techniques and improve their own movements. Through this service, anyone can experience and learn top techniques, which helps to improve sports or artistic skills. Therefore, the system using VR glasses enables a user to experience and learn top techniques, which helps to improve sports or artistic skills.

[0076] The system of the embodiment of the present invention has a camera unit, an analysis unit, and a provision unit. The camera unit is used to capture the user's movements. The camera unit, for example, can capture the user's movements or posture in real time. For example, the camera unit can capture the user's movements while walking. In addition, the camera unit can also capture the user's movements while running. The camera unit can also capture the user's posture while standing. For example, the camera unit can also capture the user's posture while sitting. The analysis unit is used to analyze the captured data. The analysis unit, for example, can determine how the user's movements or posture differ from those of the master. For example, the analysis unit can analyze how the user's angle or timing of movements differs from those of the master. In addition, the analysis unit can also analyze how the user's speed or rhythm of movements differs from those of the master. The analysis unit can also analyze how the stability or balance of the user's posture differs from that of the master. The provision unit provides real-time advice based on the analysis results. The provision unit, for example, can provide specific advice such as "you can improve the angle of your movements" or "make your hand movements more fluid" to the user. In addition, the provision unit can also provide advice such as "you can speed up your movements" to the user. The provision unit can also provide advice such as "make the balance of your posture more stable" to the user. As a result, the system of the embodiment of the present invention supports the user's skill improvement by capturing, analyzing, and providing advice on the user's movements in real time.

[0077] The imaging section can capture the user's movement or posture in real time. The imaging section can capture the user's movement in real time, for example. The imaging section can capture the user's movement while walking, for example. In addition, the imaging section can also capture the user's movement while running. The imaging section can also capture the user's posture while standing. The imaging section can also capture the user's posture while sitting, for example. Thus, by capturing the user's movement or posture in real time, accurate data can be obtained. Part or all of the above processing in the imaging section can be implemented by AI, or AI can not be used. For example, the imaging section can input the captured user movement data to a generative AI, and the generative AI can perform data analysis.

[0078] The analysis section can analyze the captured data and determine how the user's movement or posture differs from that of the master. The analysis section can analyze the captured data and determine how the user's movement or posture differs from that of the master, for example. The analysis section can determine how the angle or timing of the user's movement differs from that of the master, for example. In addition, the analysis section can also determine how the speed or rhythm of the user's movement differs from that of the master. The analysis section can also determine how the stability or balance of the user's posture differs from that of the master. Thus, by determining how the user's movement or posture differs from that of the master, specific points for improvement can be made clear. Part or all of the above processing in the analysis section can be implemented by generative AI, or generative AI can not be used. For example, the analysis section can input the captured data to a generative AI, and the generative AI can perform data analysis.

[0079] The provision section can provide advice in real time based on the analysis result. The provision section can provide advice in real time based on the analysis result, for example. The provision section can provide the user with specific advice such as "increase the movement angle" or "make the hand movement smoother", for example. In addition, the provision section can also provide the user with advice such as "increase the movement speed". The provision section can also provide the user with advice such as "make the balance of the posture more stable". Thus, by providing advice in real time, the user can immediately grasp the points for improvement. Part or all of the above processing in the provision section can be implemented by generative AI, or generative AI can not be used. For example, the provision section can input the analysis result to a generative AI, and the generative AI can generate advice.

[0080] The provisioning department can offer specific suggestions to users. For example, it can provide suggestions such as "you can further increase the angle of movement" or "make hand movements smoother." Furthermore, it can offer suggestions like "you can further increase the speed of movement." It can also offer suggestions like "make the posture more stable." Thus, by providing specific suggestions, users can understand concrete methods for improvement. Some or all of the above processing in the provisioning department can be implemented using generative AI, or it can be done without generative AI. For example, the provisioning department can input the analysis results into generative AI, which will then generate suggestions.

[0081] The system includes a camera unit that infers user emotions and adjusts the camera capture angle based on these inferences. For example, when a user is nervous, the camera capture angle is adjusted to a wide-angle to capture overall movement. When the user is relaxed, the camera capture angle can be adjusted to a narrow-angle to capture detailed movements. Furthermore, when a user is focused, the capture angle can be adjusted to focus on specific actions or postures. Thus, by adjusting the capture angle based on user emotions, more suitable data can be obtained. Emotion inference can be achieved, for example, through emotion inference functions such as emotion engines or generative AI. Generative AI can be, for example, text generation AI (such as LLM) or multimodal generation AI, but is not limited to these. Some or all of the above processing in the camera unit can be achieved through AI, or AI can be omitted. For example, the camera unit can input data used to infer user emotions into generative AI, which then infers the emotions and adjusts the capture angle based on the results.

[0082] The system includes a camera unit that analyzes a user's past action history and selects the appropriate capture time. For example, the camera unit can analyze the user's past action history and begin capturing when similar actions occur. Furthermore, it can analyze the frequency of specific actions based on the user's past action history and adjust the capture time accordingly. It can also predict the start and end times of actions based on the user's past action history and select the optimal capture time. Thus, by selecting the optimal capture time based on past action history, important moments cannot be missed. Some or all of the above processing in the camera unit can be implemented using AI, or it can be done without AI. For example, the camera unit can input the user's past action history into a generative AI, which will then select the optimal capture time.

[0083] The system includes a camera unit capable of emphasizing specific actions or postures when capturing user movements. For example, the camera unit can emphasize specific actions or postures during user motion capture. For instance, when a user performs a specific action, the camera focus can be adjusted to emphasize that action. Furthermore, when a user adopts a specific posture, the camera's zoom function can be used to emphasize that posture. The camera exposure can also be adjusted to emphasize that action. Thus, by emphasizing the capture of specific actions or postures, important movements can be analyzed in detail. Some or all of the above processing in the camera unit can be implemented using AI, or it can be done without AI. For example, the camera unit can input the captured user motion data into generative AI, which can then emphasize specific actions or postures for capture.

[0084] The system includes a camera unit that infers user emotions and determines the priority of captured actions based on these inferences. For example, the camera unit can infer user emotions and determine the priority of captured actions accordingly. For instance, when a user is excited, the speed of the movement is prioritized. When a user is relaxed, the smoothness of the movement is prioritized. Furthermore, when a user is focused, specific technical movements are prioritized. Thus, by prioritizing captured actions based on user emotions, important movements can be captured first. Emotion inference is achieved, for example, through emotion inference functions such as emotion engines or generative AI. Generative AI includes, but is not limited to, text generation AI (such as LLM) or multimodal generation AI. Some or all of the above processing in the camera unit can be implemented using AI, or AI can be omitted. For example, the camera unit can input data used to infer user emotions into generative AI, which infers the emotions and determines the priority of captured actions based on the results.

[0085] The system includes a camera unit that prioritizes capturing highly relevant actions while considering the user's geographic location information. For example, the camera unit can prioritize capturing actions related to a specific sport when the user is at that facility. Similarly, when the user is at a specific workshop, actions performed there can be prioritized. Likewise, when the user is at a specific event location, actions related to that event can be prioritized. Thus, by considering geographic location information, highly relevant actions can be prioritized. Some or all of the above processing in the camera unit can be implemented using AI, or it can be done without AI. For example, the camera unit can input the user's geographic location information into a generative AI, which then selects highly relevant actions.

[0086] The system includes a camera unit that analyzes user social media activity and captures related actions. For example, the camera unit can analyze user social media activity and capture related actions. This includes analyzing videos or photos shared by the user on social media and capturing related actions. Furthermore, it can analyze the activity of accounts the user follows on social media and capture related actions. It can also analyze the activities the user participates in on social media and capture related actions. Thus, by analyzing social media activity, it is possible to capture actions relevant to the user. Some or all of the above processing in the camera unit can be achieved using AI, or it can be done without AI. For example, the camera unit can input user social media activity into generative AI, which can then select relevant actions.

[0087] The system has a parsing unit that infers user emotions and adjusts the parsing algorithm based on the inferred emotions. For example, the parsing unit can infer user emotions and adjust the parsing algorithm accordingly. For instance, when the user is nervous, the parsing algorithm is simplified, focusing on the parsing of basic movements. When the user is relaxed, the parsing algorithm can be refined, focusing on the parsing of detailed movements. Furthermore, when the user is focused, the parsing algorithm can be highly refined, focusing on the parsing of technical movements. Thus, by adjusting the parsing algorithm based on user emotions, more suitable parsing results can be provided. Emotion inference can be achieved, for example, through emotion inference functions such as emotion engines or generative AI. Generative AI can be, for example, text generation AI (such as LLM) or multimodal generation AI, but is not limited to these. Some or all of the above processing in the parsing unit can be implemented using generative AI, or it can be done without generative AI. For example, the parsing unit can input data used to infer user emotions into generative AI, which infers the emotions and adjusts the parsing algorithm based on the results.

[0088] The parsing unit can adjust the level of detail in its analysis of captured data based on the importance of the actions. For example, it can perform detailed analysis on important actions, providing comprehensive results. For actions of lower importance, it can perform simplified analysis, providing basic results. It can also adjust the level of detail in stages based on the importance of the actions, providing appropriate results. Thus, by adjusting the level of detail based on the importance of the actions, important actions can be analyzed in detail. Some or all of the above processing in the parsing unit can be implemented using generative AI, or it can be implemented without generative AI. For example, the parsing unit can input the captured data into generative AI, which can then adjust the level of detail in its analysis based on the importance of the actions.

[0089] The parsing unit can apply different parsing algorithms based on the type of action when parsing the captured data. For example, it can apply a dedicated sports parsing algorithm for sports actions, a dedicated craft parsing algorithm for craft actions, and a dedicated art parsing algorithm for artistic actions. Thus, by applying different parsing algorithms based on the type of action, more suitable parsing results can be provided. Some or all of the above processing in the parsing unit can be implemented using generative AI, or it can be done without generative AI. For example, the parsing unit can input the captured data into generative AI, which then applies different parsing algorithms based on the type of action.

[0090] The system includes a parsing unit that infers user emotions and adjusts the display of parsing results based on the inferred emotions. For example, the parsing unit can infer user emotions and adjust the display of parsing results accordingly. For instance, when the user is nervous, a concise and easily identifiable display is provided. When the user is relaxed, a display containing detailed information can be provided. Furthermore, when the user is focused, a display including technical details can be provided. Thus, by adjusting the display of parsing results based on user emotions, more appropriate information can be provided. Emotion inference can be achieved, for example, through emotion inference functions such as emotion engines or generative AI. Generative AI can be, for example, text generation AI (such as LLM) or multimodal generation AI, but is not limited to these. Some or all of the above processing in the parsing unit can be implemented using generative AI, or it can be done without generative AI. For example, the parsing unit can input data used to infer user emotions into generative AI, which infers the emotions and adjusts the display of parsing results based on the results.

[0091] The parsing unit can determine the parsing priority based on the timing of action submission when parsing captured data. For example, it can prioritize parsing the most recently captured data to provide the latest parsing results. Data submitted earlier can have its parsing priority lowered. The parsing priority can also be adjusted in stages based on the submission timing to provide appropriate parsing results. Thus, by determining the parsing priority based on the submission timing, the latest data can be parsed first. Some or all of the above processing in the parsing unit can be implemented using generative AI, or it can be implemented without generative AI. For example, the parsing unit can input the captured data into generative AI, which then determines the parsing priority based on the timing of action submission.

[0092] The parsing unit can adjust the parsing order based on the relevance of actions when parsing the captured data. For example, it can prioritize parsing highly relevant actions, providing detailed parsing results. For actions with lower relevance, the parsing order can be shifted later. The parsing order can also be adjusted in stages based on the relevance of actions, providing appropriate parsing results. Thus, by adjusting the parsing order based on the relevance of actions, highly relevant actions can be parsed first. Some or all of the above processing in the parsing unit can be implemented using generative AI, or it can be implemented without generative AI. For example, the parsing unit can input the captured data into generative AI, which can then adjust the parsing order based on the relevance of actions.

[0093] The system includes a provisioning unit that infers user emotions and adjusts the expression of suggestions based on the inferred emotions. For example, the provisioning unit can infer user emotions and adjust the expression of suggestions accordingly. For instance, when a user is nervous, it provides concise and easily identifiable suggestions. When a user is relaxed, it can provide suggestions with detailed information. Furthermore, when a user is focused, it can provide suggestions with technical details. Thus, by adjusting the expression of suggestions based on user emotions, more appropriate suggestions can be provided. Emotion inference can be achieved, for example, through emotion inference functions such as emotion engines or generative AI. Generative AI can be, for example, text generation AI (such as LLM) or multimodal generation AI, but is not limited to these. Some or all of the above processing in the provisioning unit can be implemented using generative AI, or it can be done without generative AI. For example, the provisioning unit can input data used to infer user emotions into generative AI, which infers the emotions and adjusts the expression of suggestions based on the results.

[0094] The provisioning unit can adjust the level of detail based on the importance of the suggestions when providing recommendations based on the analysis results. For example, it can provide suggestions with detailed information for important suggestions, and suggestions with brief information for less important suggestions. It can also adjust the level of detail in stages according to the importance of the suggestions to provide appropriate recommendations. Thus, by adjusting the level of detail according to the importance of the suggestions, important suggestions can be provided in detail. Some or all of the above processing in the provisioning unit can be implemented using generative AI, or it can be done without generative AI. For example, the provisioning unit can input the analysis results into generative AI, which can then adjust the level of detail according to the importance of the suggestions.

[0095] The provisioning unit can apply different suggestion algorithms based on the category of the action when providing suggestions based on the analysis results. For example, it can apply a dedicated sports suggestion algorithm for sports actions, a dedicated craft suggestion algorithm for craft actions, and a dedicated art suggestion algorithm for artistic actions. Thus, by applying different suggestion algorithms based on the category of the action, more suitable suggestions can be provided. Some or all of the above processing in the provisioning unit can be implemented using generative AI, or it can be implemented without generative AI. For example, the provisioning unit can input the analysis results into generative AI, which can then apply different suggestion algorithms based on the category of the action.

[0096] The system includes a provisioning unit that infers user emotions and adjusts the length of suggestions based on the inferred emotions. For example, the provisioning unit can infer user emotions and adjust the length of suggestions accordingly. For instance, when a user is nervous, it provides concise and to-the-point suggestions. When a user is relaxed, it can provide longer suggestions with detailed explanations. Furthermore, when a user is focused, it can provide suggestions with technical details. Thus, by adjusting the length of suggestions based on user emotions, more appropriate suggestions can be provided. Emotion inference can be achieved, for example, through emotion inference functions such as emotion engines or generative AI. Generative AI can be, for example, text generation AI (such as LLM) or multimodal generation AI, but is not limited to these. Some or all of the above processing in the provisioning unit can be implemented using generative AI, or it can be done without generative AI. For example, the provisioning unit can input data used to infer user emotions into generative AI, which infers the emotions and adjusts the length of suggestions based on the results.

[0097] The provisioning unit can determine the priority of suggestions based on the submission time of actions when providing suggestions based on the analysis results. For example, it can prioritize suggestions based on the most recently captured data. Suggestions corresponding to data submitted earlier can be given lower priority. The priority of suggestions can also be adjusted in stages according to the submission time to provide appropriate suggestions. Thus, by determining the priority of suggestions based on the submission time, suggestions based on the latest data can be prioritized. Some or all of the above processing in the provisioning unit can be implemented using generative AI, or it can be implemented without generative AI. For example, the provisioning unit can input the analysis results into generative AI, which can then determine the priority of suggestions based on the submission time of actions.

[0098] The providing unit can adjust the order of suggestions based on the relevance of actions when providing suggestions based on the analysis results. For example, the providing unit can prioritize providing suggestions for actions with high relevance. Suggestions for actions with low relevance can also be shifted to later in the order. The order of suggestions can also be adjusted in stages according to the relevance of actions to provide appropriate suggestions. Thus, by adjusting the order of suggestions according to the relevance of actions, suggestions for actions with high relevance can be prioritized. Some or all of the above processing in the providing unit can be implemented using generative AI, or it can be implemented without generative AI. For example, the providing unit can input the analysis results into generative AI, which can then adjust the order of suggestions according to the relevance of actions.

[0099] The system described in this embodiment is not limited to the examples above. For example, various modifications can be made as follows.

[0100] The system can include a voice capture unit that records ambient sounds while capturing user movements. For example, it can record surrounding sounds during physical activities and analyze the voice data along with the motion analysis. Furthermore, it can record the sounds of tools used by the user during craft operations, analyzing the precision and rhythm of the movements. It can also record the sounds of musical instruments played by the user to improve their playing technique. Therefore, by combining motion and voice data for analysis, more detailed feedback can be provided.

[0101] The system can be equipped with environmental sensors that simultaneously record ambient data such as temperature and humidity while capturing user movements. For example, it can record the surrounding temperature and humidity when a user is engaged in sports activities and analyze the environmental data along with the motion analysis. Furthermore, it can record the temperature and humidity of the working environment when a user is performing technological tasks, analyzing factors affecting work efficiency and accuracy. It can also record the temperature and humidity of the playing environment when a user is performing, to improve playing technique. Therefore, by combining motion and environmental data for analysis, more detailed feedback can be provided.

[0102] The system can be equipped with biosensors that simultaneously record biological data such as heart rate and respiratory rate while capturing user movements. For example, it can record a user's heart rate and respiratory rate during physical activity and analyze the biological data along with motion analysis. Furthermore, it can record a user's heart rate and respiratory rate during craft work to analyze focus and fatigue levels. It can also record a user's heart rate and respiratory rate while performing music to improve performance technique. Therefore, by combining motion and biological data analysis, more detailed feedback can be provided.

[0103] The system can include an eye-tracking unit that records user eye movements simultaneously with motion capture. For example, it can record eye movements during physical activities and analyze the data along with motion analysis. Furthermore, it can record eye movements during craft work, analyzing focus levels and eye movement patterns. It can also record eye movements while performing music, helping to improve performance techniques. Thus, by combining motion and eye movement data for analysis, more detailed feedback can be provided.

[0104] The system can be equipped with an electromyography (EMG) sensor that records the user's electrical activity (EMG) while capturing their movements. For example, it can record the EMG during physical activity and analyze the data along with motion analysis. Furthermore, it can record the EMG during craft work, analyzing the precision and force applied during the task. It can also record the EMG during musical performances to improve playing technique. Thus, by combining motion and EMG data for analysis, more detailed feedback can be provided.

[0105] The system can anticipate a user's emotions and adjust the difficulty of the exercises accordingly. For example, when a user is tense, the difficulty can be reduced, focusing on basic movements. When a user is relaxed, the difficulty can be increased, focusing on movements requiring advanced skills. Furthermore, when a user is focused, the difficulty can be adjusted to provide an appropriate challenge. Thus, by adjusting the difficulty of exercises based on the user's emotions, more suitable training can be provided.

[0106] The system can anticipate user emotions and adjust the timing of feedback accordingly. For example, when a user is tense, feedback can be delayed to allow them time to relax. When a user is relaxed, feedback can be provided earlier to immediately offer areas for improvement. Furthermore, when a user is focused, feedback timing can be adjusted to provide feedback at the appropriate moment. Thus, by adjusting feedback timing based on user emotions, more effective feedback can be provided.

[0107] The system can infer user emotions and adjust the feedback content accordingly. For example, when a user is nervous, it can provide more positive feedback. When a user is relaxed, it can provide feedback with specific areas for improvement. Furthermore, when a user is focused, it can provide feedback with technical details. Thus, by adjusting the feedback content based on user emotions, more appropriate feedback can be provided.

[0108] The system can predict user emotions and adjust the training frequency accordingly. For example, when a user is tense, the training frequency can be reduced and relaxation time increased. When a user is relaxed, the training frequency can be increased to maintain focus. Furthermore, when a user is focused, the training frequency can be adjusted to provide an appropriate workload. Therefore, by adjusting the training frequency based on user emotions, more effective training can be provided.

[0109] The system can predict user emotions and adjust the types of training provided based on these predictions. For example, when a user is tense, it can provide relaxation-enhancing training. When a user is relaxed, it can provide training to improve concentration. Furthermore, when a user is focused, it can provide training to improve technical skills. Thus, by adjusting the types of training based on user emotions, more suitable training can be provided.

[0110] The following is a brief description of the processing flow of Implementation Method 2.

[0111] Step 1: The camera captures the user's movements. The camera can capture the user's movements or postures in real time. For example, the camera can capture the user's walking movements. Furthermore, the camera can capture the user's running movements. The camera can also capture the user's posture when standing. For example, the camera can also capture the user's posture when sitting.

[0112] Step 2: The analysis unit analyzes the captured data. For example, the analysis unit can determine how the user's movements or postures differ from the master's. For instance, the analysis unit can analyze how the angle or timing of the user's movements differs from the master's. Furthermore, the analysis unit can analyze how the speed or rhythm of the user's movements differs from the master's. The analysis unit can also analyze how the stability or balance of the user's posture differs from the master's.

[0113] Step 3: The provisioning department provides real-time suggestions based on the analysis results. For example, the provisioning department can offer specific suggestions such as "you can further increase the angle of movement" or "make the hand movements smoother." In addition, the provisioning department can also offer suggestions such as "you can further increase the speed of movement." Furthermore, the provisioning department can also offer suggestions such as "make the posture more stable."

[0114] The specific processing unit 290 sends the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires voice representing the user's input to the result of the specific processing. The control unit 46A sends the voice data representing the user's input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[0115] Data generation model 58 is what is known as generative AI (Artificial Intelligence). An example of data generation model 58 includes ChatGPT (registered trademark) (Internet search).<URL:https: / / openai.com / blog / chatgpt> Generative AI, such as data generation model 58, is obtained by deep learning through a neural network. The data generation model 58 is input with a prompt containing instructions, and with inference data such as speech data representing speech, text data representing text, and image data representing images (e.g., data of still images or data of moving images). The data generation model 58 infers the input inference data according to the instructions shown in the prompt, and outputs the inference result in one or more data forms such as speech data, text data, and image data. The data generation model 58 includes, for example, text generation AI, image generation AI, and multimodal generation AI. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization. The specific processing unit 290 performs the above-described specific processing while using the data generation model 58. The data generation model 58 can also be a fine-tuned model to output inference results from prompts without instructions; in this case, the data generation model 58 can output inference results from prompts without instructions. The data processing device 12, etc., includes multiple data generation models 58, including AI other than generative AI. AI other than generative AI includes, but is not limited to, linear regression, logistic regression, decision trees, random forests, support vector machines (SVM), k-means clustering, convolutional neural networks (CNN), recurrent neural networks (RNN), generative adversarial networks (GAN), or Naive Bayes. Furthermore, AI can also act as an AI agent. Additionally, when the processing described above is performed by AI, this processing may be partially or entirely performed by AI, but is not limited to this example. Moreover, processing performed by AI, including generative AI, can be replaced by rule-based processing, and vice versa.

[0116] Furthermore, the processing performed by the aforementioned data processing system 10 is executed by the specific processing unit 290 of the data processing device 12 or the control unit 46A of the smart device 14, but it can also be executed jointly by the specific processing unit 290 of the data processing device 12 and the control unit 46A of the smart device 14. Additionally, the specific processing unit 290 of the data processing device 12 acquires or collects information required for processing from the smart device 14 or external devices, and the smart device 14 acquires or collects information required for processing from the data processing device 12 or external devices.

[0117] Each of the aforementioned elements—the camera unit, the analysis unit, and the providing unit—can be implemented, for example, by at least one of the smart device 14 and the data processing device 12. For instance, the camera unit captures the user's movements or postures in real time via the camera 42 of the smart device 14. The analysis unit analyzes the captured data via the specific processing unit 290 of the data processing device 12 and determines how the user's movements or postures differ from those of the master. The providing unit provides suggestions in real time based on the analysis results via the specific processing unit 290 of the data processing device 12. Each of the camera unit, the analysis unit, and the providing unit can also be implemented via the control unit 46A of the smart device 14. The correspondence between the various units and the device or control unit is not limited to the above examples and can be varied.

[0118] Second Implementation Method

[0119] Figure 3 An example of the configuration of the data processing system 210 according to the second embodiment is shown.

[0120] like Figure 3 As shown, the data processing system 210 includes a data processing device 12 and smart glasses 214. One example of the data processing device 12 is a server.

[0121] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 includes a processor 28, RAM 30, and memory 32. The processor 28, RAM 30, and memory 32 are connected to a bus 34. Furthermore, the database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. An example of the network 54 includes a WAN and / or LAN, etc.

[0122] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, RAM 48, and memory 50. The processor 46, RAM 48, and memory 50 are connected to a bus 52. Furthermore, the microphone 238, speaker 240, and camera 42 are also connected to the bus 52.

[0123] Microphone 238 receives user commands by receiving the user's voice. Microphone 238 captures the user's voice, converts the captured sound into speech data, and outputs it to processor 46. Speaker 240 outputs sound according to commands from processor 46.

[0124] Camera 42 is a small digital camera equipped with an optical system such as a lens, aperture and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge-Coupled Device) image sensor, used to capture the user's surroundings (e.g., the shooting range defined by an angle of view equivalent to the field of vision of an average healthy person).

[0125] Communication I / F 44 is connected to network 54. Communication I / F 44 and 26 are responsible for the transmission and reception of various information between processor 46 and processor 28 via network 54. The transmission and reception of various information between processor 46 and processor 28 using communication I / F 44 and 26 is performed in a secure state.

[0126] Figure 4 An example of the main functions of the data processing device 12 and the smart glasses 214 is shown. Figure 4 As shown, in the data processing device 12, specific processing is performed by the processor 28. The memory 32 stores a specific processing program 56.

[0127] The processor 28 reads a specific processing program 56 from the memory 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is implemented by the processor 28 operating as a specific processing unit 290 based on the specific processing program 56 executed on the RAM 30.

[0128] The memory 32 stores a data generation model 58 and an emotion-specific model 59. The data generation model 58 and the emotion-specific model 59 are used by the specific processing unit 290. The specific processing unit 290 can use the emotion-specific model 59 to infer the user's emotions and perform specific processing based on the user's emotions. The emotion inference function (emotion-specific function) using the emotion-specific model 59 includes various inferences and predictions related to the user's emotions, but is not limited to this example. Furthermore, emotion inference and prediction also include, for example, emotion analysis (analysis).

[0129] In the smart glasses 214, specific processing is performed by the processor 46. A specific processing program 60 is stored in the memory 50. The processor 46 reads the specific processing program 60 from the memory 50 and executes the read specific processing program 60 on the RAM 48. Specific processing is implemented by the processor 46 operating as a control unit 46A based on the specific processing program 60 executed on the RAM 48. Furthermore, the smart glasses 214 may also have the same data generation model and emotion-specific model as the data generation model 58 and the emotion-specific model 59, and use these models to perform the same processing as the specific processing unit 290.

[0130] Furthermore, other devices besides the data processing device 12 may also have the data generation model 58. For example, a server device may have the data generation model 58. In this case, the data processing device 12 obtains the processing results (prediction results, etc.) using the data generation model 58 by communicating with the server device that has the data generation model 58. In addition, the data processing device 12 may be a server device or a user-owned terminal device (e.g., a mobile phone, robot, home appliance, etc.).

[0131] The specific processing unit 290 sends the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires voice input representing the user's input to the specific processing result. The control unit 46A sends the voice data representing the user's input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[0132] Data generation model 58 is a so-called generative AI. An example of data generation model 58 includes generative AIs such as ChatGPT. Data generation model 58 is obtained by performing deep learning on a neural network. A prompt containing instructions is input to data generation model 58, along with inference data such as speech data representing speech, text data representing text, and image data representing images (e.g., data from still images or moving images). Data generation model 58 infers the inference data based on the instructions shown in the prompt and outputs the inference result in one or more data forms such as speech data, text data, and image data. Data generation model 58 includes, for example, text generation AI, image generation AI, and multimodal generation AI. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization. Specific processing unit 290 performs the aforementioned specific processing while using data generation model 58. Data generation model 58 can also be a fine-tuned model to output inference results from prompts without instructions; in this case, data generation model 58 can output inference results from prompts without instructions. The data processing device 12, etc., includes various data generation models 58, which include AI other than generative AI. These AIs include, for example, linear regression, logistic regression, decision trees, random forests, support vector machines (SVM), k-means clustering, convolutional neural networks (CNN), recurrent neural networks (RNN), generative adversarial networks (GAN), or Naive Bayes, and are capable of various processing methods, but are not limited to these examples. Furthermore, AI can also be an AI agent. Additionally, when the processing described above is performed by AI, this processing may be partially or entirely performed by AI, but is not limited to these examples. Moreover, processing performed by AI including generative AI can be replaced by rule-based processing, and vice versa.

[0133] The data processing system 210 of the second embodiment performs the same processing as the data processing system 10 of the first embodiment. The processing performed by the data processing system 210 is executed by the specific processing unit 290 of the data processing device 12 or the control unit 46A of the smart glasses 214, but it can also be executed jointly by the specific processing unit 290 of the data processing device 12 and the control unit 46A of the smart glasses 214. Furthermore, the specific processing unit 290 of the data processing device 12 acquires or collects information required for processing from the smart glasses 214 or external devices, and the smart glasses 214 acquires or collects information required for processing from the data processing device 12 or external devices.

[0134] Each of the aforementioned elements—the camera unit, the analysis unit, and the providing unit—can be implemented, for example, by at least one of the smart glasses 214 and the data processing device 12. For instance, the camera unit captures the user's movements or postures in real time via the camera 42 of the smart glasses 214. The analysis unit analyzes the captured data via the specific processing unit 290 of the data processing device 12 and determines how the user's movements or postures differ from those of the master. The providing unit provides suggestions in real time based on the analysis results via the specific processing unit 290 of the data processing device 12. Each of the camera unit, the analysis unit, and the providing unit can also be implemented via the control unit 46A of the smart glasses 214. The correspondence between the various units and the device or control unit is not limited to the above examples and can be varied.

[0135] Third Implementation Method

[0136] Figure 5 An example of the configuration of the data processing system 310 according to the third embodiment is shown.

[0137] like Figure 5 As shown, the data processing system 310 includes a data processing device 12 and a head-mounted terminal 314. An example of the data processing device 12 is a server.

[0138] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 includes a processor 28, RAM 30, and memory 32. The processor 28, RAM 30, and memory 32 are connected to a bus 34. Furthermore, the database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. An example of the network 54 includes a WAN and / or LAN, etc.

[0139] The head-mounted terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, RAM 48, and memory 50. The processor 46, RAM 48, and memory 50 are connected to a bus 52. Furthermore, the microphone 238, speaker 240, camera 42, and display 343 are also connected to the bus 52.

[0140] Microphone 238 receives user commands by receiving the user's voice. Microphone 238 captures the user's voice, converts the captured sound into speech data, and outputs it to processor 46. Speaker 240 outputs sound according to commands from processor 46.

[0141] Camera 42 is a small digital camera equipped with an optical system such as a lens, aperture and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge-Coupled Device) image sensor, used to capture the user's surroundings (e.g., the shooting range defined by an angle of view equivalent to the field of vision of an average healthy person).

[0142] Communication I / F 44 is connected to network 54. Communication I / F 44 and 26 are responsible for the transmission and reception of various information between processor 46 and processor 28 via network 54. The transmission and reception of various information between processor 46 and processor 28 using communication I / F 44 and 26 is performed in a secure state.

[0143] Figure 6 An example of the main functions of the data processing device 12 and the head-mounted terminal 314 is shown. Figure 6 As shown, in the data processing device 12, specific processing is performed by the processor 28. The memory 32 stores a specific processing program 56.

[0144] The processor 28 reads a specific processing program 56 from the memory 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is implemented by the processor 28 operating as a specific processing unit 290 based on the specific processing program 56 executed on the RAM 30.

[0145] The memory 32 stores a data generation model 58 and an emotion-specific model 59. The data generation model 58 and the emotion-specific model 59 are used by the specific processing unit 290. The specific processing unit 290 can use the emotion-specific model 59 to infer the user's emotions and perform specific processing based on the user's emotions. The emotion inference function (emotion-specific function) using the emotion-specific model 59 includes various inferences and predictions related to the user's emotions, but is not limited to this example. Furthermore, emotion inference and prediction also include, for example, emotion analysis (analysis).

[0146] In the head-mounted terminal 314, specific processing is performed by the processor 46. A specific program 60 is stored in the memory 50. The processor 46 reads the specific program 60 from the memory 50 and executes the read specific program 60 on the RAM 48. Specific processing is implemented by the processor 46 operating as a control unit 46A based on the specific program 60 executed on the RAM 48. Furthermore, the head-mounted terminal 314 may also have the same data generation model and emotion-specific model as the data generation model 58 and the emotion-specific model 59, and use these models to perform the same processing as the specific processing unit 290.

[0147] Furthermore, other devices besides the data processing device 12 may also have the data generation model 58. For example, a server device may have the data generation model 58. In this case, the data processing device 12 obtains the processing results (prediction results, etc.) using the data generation model 58 by communicating with the server device that has the data generation model 58. In addition, the data processing device 12 may be a server device or a user-owned terminal device (e.g., a mobile phone, robot, home appliance, etc.).

[0148] The specific processing unit 290 sends the result of the specific processing to the head-mounted terminal 314. In the head-mounted terminal 314, the control unit 46A causes the speaker 240 and display 343 to output the result of the specific processing. The microphone 238 acquires voice representing the user's input to the specific processing result. The control unit 46A sends the voice data representing the user's input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[0149] Data generation model 58 is a so-called generative AI. An example of data generation model 58 includes generative AIs such as ChatGPT. Data generation model 58 is obtained by performing deep learning on a neural network. A prompt containing instructions is input to data generation model 58, along with inference data such as speech data representing speech, text data representing text, and image data representing images (e.g., data from still images or moving images). Data generation model 58 infers the inference data based on the instructions shown in the prompt and outputs the inference result in one or more data forms such as speech data, text data, and image data. Data generation model 58 includes, for example, text generation AI, image generation AI, and multimodal generation AI. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization. Specific processing unit 290 performs the aforementioned specific processing while using data generation model 58. Data generation model 58 can also be a fine-tuned model to output inference results from prompts without instructions; in this case, data generation model 58 can output inference results from prompts without instructions. The data processing device 12, etc., includes various data generation models 58, which include AI other than generative AI. These AIs include, for example, linear regression, logistic regression, decision trees, random forests, support vector machines (SVM), k-means clustering, convolutional neural networks (CNN), recurrent neural networks (RNN), generative adversarial networks (GAN), or Naive Bayes, and are capable of various processing methods, but are not limited to these examples. Furthermore, AI can also be an AI agent. Additionally, when the processing described above is performed by AI, this processing may be partially or entirely performed by AI, but is not limited to these examples. Moreover, processing performed by AI including generative AI can be replaced by rule-based processing, and vice versa.

[0150] The data processing system 310 of the third embodiment performs the same processing as the data processing system 10 of the first embodiment. The processing performed by the data processing system 310 is executed by the specific processing unit 290 of the data processing device 12 or the control unit 46A of the head-mounted terminal 314, but it can also be executed jointly by the specific processing unit 290 of the data processing device 12 and the control unit 46A of the head-mounted terminal 314. Furthermore, the specific processing unit 290 of the data processing device 12 acquires or collects information required for processing from the head-mounted terminal 314 or external devices, and the head-mounted terminal 314 acquires or collects information required for processing from the data processing device 12 or external devices.

[0151] Each of the aforementioned elements—the camera unit, the analysis unit, and the providing unit—can be implemented, for example, by at least one of the head-mounted terminal 314 and the data processing device 12. For instance, the camera unit captures the user's movements or postures in real time via the camera 42 of the head-mounted terminal 314. The analysis unit analyzes the captured data via the specific processing unit 290 of the data processing device 12 and determines how the user's movements or postures differ from those of the master. The providing unit provides suggestions in real time based on the analysis results via the specific processing unit 290 of the data processing device 12. Each of the camera unit, the analysis unit, and the providing unit can also be implemented via the control unit 46A of the head-mounted terminal 314. The correspondence between the various units and the device or control unit is not limited to the above examples and can be varied.

[0152] Fourth Implementation Method

[0153] Figure 7 An example of the configuration of the data processing system 410 according to the fourth embodiment is shown.

[0154] like Figure 7 As shown, the data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[0155] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 includes a processor 28, RAM 30, and memory 32. The processor 28, RAM 30, and memory 32 are connected to a bus 34. Furthermore, the database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. An example of the network 54 includes a WAN and / or LAN, etc.

[0156] Robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control object 443. The computer 36 includes a processor 46, RAM 48, and memory 50. The processor 46, RAM 48, and memory 50 are connected to a bus 52. Furthermore, the microphone 238, speaker 240, camera 42, and control object 443 are also connected to the bus 52.

[0157] Microphone 238 receives user commands by receiving the user's voice. Microphone 238 captures the user's voice, converts the captured sound into speech data, and outputs it to processor 46. Speaker 240 outputs sound according to commands from processor 46.

[0158] The camera 42 is a small digital camera equipped with an optical system such as a lens, aperture and shutter, and an imaging element such as a CMOS image sensor or a CCD image sensor, used to photograph the user's surroundings (e.g., the shooting range defined by an angle of view equivalent to the field of vision of an average healthy person).

[0159] Communication I / F 44 is connected to network 54. Communication I / F 44 and 26 are responsible for the transmission and reception of various information between processor 46 and processor 28 via network 54. The transmission and reception of various information between processor 46 and processor 28 using communication I / F 44 and 26 is performed in a secure state.

[0160] The controlled object 443 includes a display device, LEDs for the eyes, and motors for driving the arms, hands, and feet. The posture and movements of the robot 414 are controlled by controlling the motors for the arms, hands, and feet. Some of the robot 414's emotions can be expressed by controlling these motors. In addition, facial expressions of the robot 414 can also be expressed by controlling the illumination state of the LEDs for the robot 414's eyes.

[0161] Figure 8 An example of the main functions of the data processing device 12 and the robot 414 is shown. For example... Figure 8 As shown, in the data processing device 12, specific processing is performed by the processor 28. The memory 32 stores a specific processing program 56.

[0162] The processor 28 reads a specific processing program 56 from the memory 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is implemented by the processor 28 operating as a specific processing unit 290 based on the specific processing program 56 executed on the RAM 30.

[0163] The memory 32 stores a data generation model 58 and an emotion-specific model 59. The data generation model 58 and the emotion-specific model 59 are used by the specific processing unit 290. The specific processing unit 290 can use the emotion-specific model 59 to infer the user's emotions and perform specific processing based on the user's emotions. The emotion inference function (emotion-specific function) using the emotion-specific model 59 includes various inferences and predictions related to the user's emotions, but is not limited to this example. Furthermore, emotion inference and prediction also include, for example, emotion analysis (analysis).

[0164] In robot 414, specific processing is performed by processor 46. A specific program 60 is stored in memory 50. Processor 46 reads the specific program 60 from memory 50 and executes the read specific program 60 on RAM 48. Specific processing is achieved by processor 46 acting as control unit 46A based on the specific program 60 executed on RAM 48. Furthermore, robot 414 may also have the same data generation model and emotion-specific model as data generation model 58 and emotion-specific model 59, and use these models to perform the same processing as specific processing unit 290.

[0165] Furthermore, other devices besides the data processing device 12 may also have the data generation model 58. For example, a server device may have the data generation model 58. In this case, the data processing device 12 obtains the processing results (prediction results, etc.) using the data generation model 58 by communicating with the server device that has the data generation model 58. In addition, the data processing device 12 may be a server device or a user-owned terminal device (e.g., a mobile phone, robot, home appliance, etc.).

[0166] The specific processing unit 290 sends the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the controlled object 443 to output the result of the specific processing. The microphone 238 acquires voice representing the user's input regarding the result of the specific processing. The control unit 46A sends the voice data representing the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[0167] Data generation model 58 is a so-called generative AI. An example of data generation model 58 includes generative AIs such as ChatGPT. Data generation model 58 is obtained by performing deep learning on a neural network. A prompt containing instructions is input to data generation model 58, along with inference data such as speech data representing speech, text data representing text, and image data representing images (e.g., data from still images or moving images). Data generation model 58 infers the inference data based on the instructions shown in the prompt and outputs the inference result in one or more data forms such as speech data, text data, and image data. Data generation model 58 includes, for example, text generation AI, image generation AI, and multimodal generation AI. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization. Specific processing unit 290 performs the aforementioned specific processing while using data generation model 58. Data generation model 58 can also be a fine-tuned model to output inference results from prompts without instructions; in this case, data generation model 58 can output inference results from prompts without instructions. The data processing device 12, etc., includes various data generation models 58, which include AI other than generative AI. These AIs include, for example, linear regression, logistic regression, decision trees, random forests, support vector machines (SVM), k-means clustering, convolutional neural networks (CNN), recurrent neural networks (RNN), generative adversarial networks (GAN), or Naive Bayes, and are capable of various processing methods, but are not limited to these examples. Furthermore, AI can also be an AI agent. Additionally, when the processing described above is performed by AI, this processing may be partially or entirely performed by AI, but is not limited to these examples. Moreover, processing performed by AI including generative AI can be replaced by rule-based processing, and vice versa.

[0168] The data processing system 410 of the fourth embodiment performs the same processing as the data processing system 10 of the first embodiment. The processing performed by the data processing system 410 is executed by the specific processing unit 290 of the data processing device 12 or the control unit 46A of the robot 414, but it can also be executed jointly by the specific processing unit 290 of the data processing device 12 and the control unit 46A of the robot 414. Furthermore, the specific processing unit 290 of the data processing device 12 acquires or collects information required for processing from the robot 414 or external devices, and the robot 414 acquires or collects information required for processing from the data processing device 12 or external devices.

[0169] Each of the aforementioned elements—the camera unit, the analysis unit, and the providing unit—can be implemented, for example, by at least one of the robot 414 and the data processing device 12. For instance, the camera unit captures the user's movements or postures in real time via the camera 42 of the robot 414. The analysis unit analyzes the captured data via a specific processing unit 290 of the data processing device 12 and determines how the user's movements or postures differ from those of the master. The providing unit provides suggestions in real time based on the analysis results via the specific processing unit 290 of the data processing device 12. Each of the camera unit, the analysis unit, and the providing unit can also be implemented via the control unit 46A of the robot 414. The correspondence between the various units and the device or control unit is not limited to the above examples and can be varied.

[0170] Furthermore, the emotion-specific model 59, serving as an emotion engine, can determine the user's emotion based on a specific mapping. Specifically, the emotion-specific model 59 can determine the user's emotion based on an emotion graph that serves as a specific mapping (see...). Figure 9 The robot's emotions can be determined by the emotion-specific model 59. In addition, the emotion-specific model 59 can also determine the robot's emotions in the same way, and the specific processing unit 290 can also perform specific processing using the robot's emotions.

[0171] Figure 9 This is a diagram representing an emotion map 400 that maps various emotions. In the emotion map 400, emotions are arranged radially from the center in concentric circles. The closer to the center of the concentric circles, the more primitive the emotion is. Further out on the concentric circles, emotions are arranged representing states or actions arising from mood. Emotion is a concept that includes both feelings and mental states. To the left of the concentric circles, emotions generated by reactions occurring in the brain are arranged roughly. To the right of the concentric circles, emotions guided by situational judgments are arranged roughly. Above and below the concentric circles, emotions generated by reactions occurring in the brain and guided by situational judgments are arranged roughly. Furthermore, the emotion of "pleasure" is arranged above the concentric circles, and the emotion of "unpleasantness" is arranged below. Thus, in the emotion map 400, various emotions are mapped according to the structure of emotion generation, while easily generated emotions are mapped nearby.

[0172] These emotions are distributed at the 3 o'clock position on the Emotion Chart 400, and usually fluctuate between peace and unease. In the right half of the Emotion Chart 400, because situational awareness is more dominant than internal feelings, it gives a sense of calm.

[0173] The inner side of the emotion diagram 400 represents the mind, and the outer side of the emotion diagram 400 represents actions. Therefore, the further you go to the outer side of the emotion diagram 400, the more the emotion can be seen (manifested in actions).

[0174] Here, human emotions are based on a balance of various factors such as posture and blood sugar levels. When these balances deviate from the ideal, it indicates unhappiness; when they approach the ideal, it indicates pleasure. In robots, cars, and motorcycles, emotions can also be created based on a balance of factors such as posture and remaining battery power. When these balances deviate from the ideal, it indicates unhappiness; when they approach the ideal, it indicates pleasure. Emotion maps can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on a Brain Physiological Signal Analysis System for Speech Emotion Recognition and Emotion, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). In the left half of the emotion map, emotions belonging to the "response" domain, where sensation is dominant, are arranged. Furthermore, in the right half of the emotion map, emotions belonging to the "situation" domain, where situational cognition is dominant, are arranged.

[0175] The emotion map defines two types of emotions that promote learning. One is a negative emotion located near the middle of "repentance" or "reflection" on the situation side. That is, when the robot experiences negative emotions such as "I never want to feel this way again" or "I never want to be scolded again." The other is a positive emotion located near "desire" on the response side. That is, when the robot experiences positive feelings such as "wanting more" or "wanting to know more."

[0176] The emotion-specific model 59 feeds user input into a pre-learned neural network to obtain emotion values ​​representing each emotion shown in the emotion graph 400, and determines the user's emotion. This neural network is pre-learned based on multiple learning data sets that combine user input with emotion values ​​representing each emotion shown in the emotion graph 400. Furthermore, this neural network is learned to... Figure 10 As shown in sentiment graph 900, sentiment values ​​in nearby configurations are similar to each other. Figure 10 Examples show that multiple emotions such as "peace of mind", "stability", and "reassurance" have similar emotional values.

[0177] In the above embodiments, a specific processing is described by a single computer 22, but the technology disclosed herein is not limited to this, and distributed processing by multiple computers, including computer 22, is also possible.

[0178] In the above embodiments, an example of storing a specific processing program 56 in memory 32 is illustrated, but the technology disclosed herein is not limited thereto. For example, the specific processing program 56 may also be stored in a portable computer-readable non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed into the computer 22 of the data processing device 12. The processor 28 performs specific processing according to the specific processing program 56.

[0179] Alternatively, the specific processing program 56 can be stored in a storage device such as a server connected to the data processing device 12 via a network 54, and the specific processing program 56 can be downloaded and installed into the computer 22 upon request from the data processing device 12.

[0180] Furthermore, it is not necessary to store the entire specific process 56 in a storage device such as a server connected to the data processing device 12 via the network 54, nor is it necessary to store the entire specific process 56 in the memory 32; a portion of the specific process 56 may also be stored.

[0181] As a hardware resource for performing specific processing, various processors can be used. For example, a CPU is a general-purpose processor that functions as a hardware resource for performing specific processing by executing software, i.e., programs. Additionally, a dedicated circuit can be listed as a processor; it is a processor with a circuit structure specifically designed for performing specific processing, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit). Every processor has built-in or connected memory, and every processor executes specific processing by using memory.

[0182] The hardware resources for performing a specific process can consist of one of these various processors, or a combination of two or more processors of the same or different types (e.g., a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Furthermore, the hardware resources for performing a specific process can also be a single processor.

[0183] As an example of a single processor, the first type consists of a combination of one or more CPUs and software, which functions as a hardware resource to perform specific processing. The second type uses a processor, such as a System-on-a-chip (SoC), which implements the entire system functionality, including multiple hardware resources for performing specific processing, using a single IC chip. In this case, the specific processing is implemented using one or more of the aforementioned processors as hardware resources.

[0184] Furthermore, as the hardware architecture of these various processors, more specifically, circuits combining semiconductor elements and other circuit components can be used. Moreover, the specific process described above is merely an example. Therefore, it goes without saying that, without departing from the main point, unnecessary steps can be removed, new steps can be added, or the processing order can be changed.

[0185] Furthermore, although the above examples have been described in terms of first to fourth embodiments, some or all of these embodiments can be combined. Additionally, the smart device 14, smart glasses 214, head-mounted terminal 314, and robot 414 are just examples and can be combined separately, or other devices may be used. Furthermore, although the above examples have been described in terms of morphological example 1 and morphological example 2, these can also be combined.

[0186] The foregoing descriptions and illustrations are detailed explanations of the parts covered by this disclosure and are merely one example of this disclosure. For instance, the descriptions of the above-described structure, function, role, and effect are just one example of the structure, function, role, and effect of the parts covered by this disclosure. Therefore, it goes without saying that, without departing from the spirit of this disclosure, unnecessary parts can be deleted, new elements can be added, or replacements can be made to the foregoing descriptions and illustrations. Furthermore, to avoid confusion and facilitate understanding of the parts covered by this disclosure, explanations of technical common sense that does not require special explanation for implementing this disclosure have been omitted from the foregoing descriptions and illustrations.

[0187] All documents, patent applications and technical standards described in this specification are incorporated herein by reference as if they were specifically and individually described as incorporated by reference.

[0188] [Postscript 1]

[0189] A system characterized by comprising:

[0190] The camera unit is used to capture the user's movements;

[0191] The analysis unit is used to analyze the data captured by the camera unit;

[0192] The provisioning unit provides suggestions in real time based on the analysis results obtained by the analysis unit.

[0193] [Postscript 2]

[0194] The system as described in Appendix 1 is characterized in that the camera unit is capable of capturing the user's actions or postures in real time.

[0195] [Postscript 3]

[0196] The system as described in Appendix 1 is characterized in that the parsing unit parses the captured data and determines how the user's actions or postures differ from those of the master.

[0197] [Postscript 4]

[0198] The system as described in Appendix 1 is characterized in that the providing unit provides suggestions in real time based on the analysis results.

[0199] [Postscript 5]

[0200] The system as described in Appendix 1 is characterized in that the providing unit provides specific suggestions to the user.

[0201] [Postscript 6]

[0202] The system as described in Appendix 1 is characterized in that the camera unit infers the user's emotions and adjusts the camera's capture angle based on the inferred user emotions.

[0203] [Postscript 7]

[0204] The system as described in Appendix 1 is characterized in that the camera unit analyzes the user's past action history and selects an appropriate capture time.

[0205] [Postscript 8]

[0206] The system as described in Appendix 1 is characterized in that, when capturing user actions, the camera unit is able to emphasize specific actions or postures for capture.

[0207] [Postscript 9]

[0208] The system as described in Appendix 1 is characterized in that the camera unit infers the user's emotions and determines the priority of capturing actions based on the inferred user emotions.

[0209] [Postscript 10]

[0210] The system as described in Appendix 1 is characterized in that the camera unit takes into account the user's geographical location information and prioritizes capturing highly relevant actions.

[0211] [Postscript 11]

[0212] The system as described in Appendix 1 is characterized in that the camera unit analyzes the user's social media activities and captures relevant actions.

[0213] [Postscript 12]

[0214] The system as described in Appendix 1 is characterized in that the parsing unit infers the user's emotions and adjusts the parsing algorithm based on the inferred user emotions.

[0215] [Postscript 13]

[0216] The system as described in Appendix 1 is characterized in that,

[0217] When analyzing the captured data, the analysis unit adjusts the level of detail based on the importance of the action.

[0218] [Postscript 14]

[0219] The system as described in Appendix 1 is characterized in that,

[0220] When parsing the captured data, the parsing unit applies different parsing algorithms according to the type of action.

[0221] [Postscript 15]

[0222] The system as described in Appendix 1 is characterized in that,

[0223] The analysis unit infers the user's emotions and adjusts the display method of the analysis results according to the inferred user emotions.

[0224] [Postscript 16]

[0225] The system as described in Appendix 1 is characterized in that,

[0226] When parsing the captured data, the parsing unit determines the parsing priority based on the timing of the action submission.

[0227] [Postscript 17]

[0228] The system as described in Appendix 1 is characterized in that,

[0229] When parsing the captured data, the parsing unit adjusts the parsing order according to the relevance of the actions.

[0230] [Postscript 18]

[0231] The system as described in Appendix 1 is characterized in that,

[0232] The provider infers the user's emotions and adjusts the way suggestions are expressed based on the inferred user emotions.

[0233] [Postscript 19]

[0234] The system as described in Appendix 1 is characterized in that,

[0235] When providing suggestions based on the analysis results, the providing department adjusts the level of detail according to the importance of the suggestions.

[0236] [Postscript 20]

[0237] The system as described in Appendix 1 is characterized in that,

[0238] When providing suggestions based on the analysis results, the providing unit applies different suggestion algorithms according to the category of the action.

[0239] [Postscript 21]

[0240] The system as described in Appendix 1 is characterized in that,

[0241] The provider infers the user's emotions and adjusts the length of the suggestion based on the inferred user emotions.

[0242] [Postscript 22]

[0243] The system as described in Appendix 1 is characterized in that,

[0244] When providing suggestions based on the analysis results, the providing unit determines the priority of the suggestions according to the timing of the action submission.

[0245] [Postscript 23]

[0246] The system as described in Appendix 1 is characterized in that,

[0247] When providing suggestions based on the analysis results, the providing unit adjusts the order of the suggestions according to the relevance of the actions.

Claims

1. A system, characterized by The system comprises: a camera unit configured to capture the user's actions; an analysis unit configured to analyze the data captured by the camera unit; a providing unit configured to provide real-time suggestions based on the analysis results obtained by the analysis unit.

2. The system according to claim 1, wherein the camera unit is capable of capturing the user's actions or gestures in real time.

3. The system according to claim 1, wherein the analysis unit analyzes the captured data and determines how the user's actions or gestures differ from those of a master.

4. The system according to claim 1, wherein the providing unit provides real-time suggestions based on the analysis results.

5. The system according to claim 1, wherein the providing unit provides specific suggestion content to the user.

6. The system according to claim 1, wherein the camera unit infers the user's emotions and adjusts the capture angle of the camera according to the inferred emotions of the user.

7. The system according to claim 1, wherein the camera unit analyzes the user's past action history and selects an appropriate capture opportunity.

8. The system according to claim 1, wherein the camera unit is capable of emphasizing specific actions or gestures for capture when capturing the user's actions.

Images