system

Abstract

We provide the system. [Solution] A sensor that records the user's movements, A computer that receives recorded motion data and analyzes areas for improvement in motion by comparing it with ideal motion data, A support device that presents the analyzed areas for improvement to the user and modifies the operation, A system that includes this.

Description

Technical Field

[0001] The technology of the present disclosure relates to a system.

Background Art

[0002] Patent Document 1 discloses a method for controlling a persona chatbot, which is performed by at least one processor and includes steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to an explanation of a chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In many sports, improving one's own form and movements is essential for improving scores and performance. However, it is an extremely difficult task to recognize specifically which parts should be corrected and execute it alone. There is a need for a technology that enables individual sports players to accurately analyze their own movements and effectively improve them even in an environment without expert guidance.

Means for Solving the Problems

[0005] To solve this problem, we provide a system that includes a sensor to record user movements, a computer that receives the recorded movement data and analyzes areas for improvement by comparing it with ideal movement data, and a support device that presents the analyzed areas for improvement to the user and helps them correct their movements. This system can convert the recorded video data and movement data into a 3D model, making it possible to visually represent the user's movements. Furthermore, by highlighting specific parts of the movements with visual display means, it becomes easier for the user to understand specific points for improvement. With this configuration, the user can virtually enjoy the perspective of an expert and autonomously proceed with individual movement improvements.

[0006] A "sensor" is a device that physically or electronically detects a user's actions and outputs that information as a signal.

[0007] A "computer" is an electronic device that analyzes input data and processes it according to a specific purpose; in this context, it is responsible for analyzing operational data.

[0008] "Motion data" refers to information about the user's body movements recorded by sensors, and includes information such as position, angle, and speed.

[0009] "Ideal motion data" refers to data that represents the optimal sports movements to aim for and serves as a benchmark for comparison with the user's motion data.

[0010] A "support device" is a device that presents the results of computer analysis to the user and assists in improving the user's actions, providing visual and auditory feedback.

[0011] A "3D model" is a three-dimensional visual representation generated based on user action data, which reproduces each part of the form in three dimensions.

[0012] "Visual display means" refers to a device or method that uses images or videos to present information to the user, and in this context, it serves to clearly indicate points for improvement in operation. [Brief explanation of the drawing]

[0013] [Figure 1] This is a conceptual diagram showing an example of the configuration of a data processing system according to the first embodiment. [Figure 2] This is a conceptual diagram showing an example of the essential functions of a data processing device and a smart device according to the first embodiment. [Figure 3] This is a conceptual diagram showing an example of the configuration of a data processing system according to the second embodiment. [Figure 4] This is a conceptual diagram showing an example of the main functions of a data processing device and smart glasses according to the second embodiment. [Figure 5] This is a conceptual diagram showing an example of the configuration of a data processing system according to the third embodiment. [Figure 6] This is a conceptual diagram showing an example of the main functions of a data processing device and a headset-type terminal according to the third embodiment. [Figure 7] This is a conceptual diagram showing an example of the configuration of a data processing system according to the fourth embodiment. [Figure 8] This is a conceptual diagram showing an example of the main functions of a data processing device and a robot according to the fourth embodiment. [Figure 9] This shows an emotion map where multiple emotions are mapped. [Figure 10] This shows an emotion map where multiple emotions are mapped. [Figure 11] This is a sequence diagram showing the processing flow of the data processing system in Example 1. [Figure 12] This is a sequence diagram showing the processing flow of the data processing system in Application Example 1. [Figure 13] This is a sequence diagram showing the processing flow of the data processing system in Example 2, which incorporates an emotion engine. [Figure 14]It is a sequence diagram showing the processing flow of a data processing system in Application Example 2 when a sentiment engine is combined.

Embodiments for Carrying Out the Invention

[0014] Hereinafter, an example of an embodiment of a system according to the technology of the present disclosure will be described with reference to the accompanying drawings.

[0015] First, the terms used in the following description will be explained.

[0016] In the following embodiments, a numbered processor (hereinafter simply referred to as "processor") may be a single arithmetic unit or a combination of multiple arithmetic units. Also, the processor may be a single type of arithmetic unit or a combination of multiple types of arithmetic units. Examples of arithmetic units include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), an APU (Accelerated Processing Unit), and the like.

[0017] In the following embodiments, a numbered RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a work memory by the processor.

[0018] In the following embodiments, a numbered storage is one or more non-volatile storage devices that store various programs and various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), or magnetic tapes, and the like.

[0019] In the following embodiments, the signed communication interface (I / F) is an interface that includes a communication processor and an antenna, etc. The communication interface manages communication between multiple computers. Examples of communication standards applicable to the communication interface include wireless communication standards such as 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), or Bluetooth (registered trademark).

[0020] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." That is, "A and / or B" means that it may be A alone, or B alone, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" applies when expressing three or more things linked by "and / or."

[0021] [First Embodiment]

[0022] Figure 1 shows an example of the configuration of the data processing system 10 according to the first embodiment.

[0023] As shown in Figure 1, the data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0024] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0025] The smart device 14 comprises a computer 36, a reception device 38, an output device 40, a camera 42, and a communication interface 44. The computer 36 comprises a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The reception device 38, output device 40, and camera 42 are also connected to the bus 52.

[0026] The reception device 38 is equipped with a touch panel 38A and a microphone 38B, etc., and receives user input. The touch panel 38A receives user input by detecting contact with an object (e.g., a pen or finger). The microphone 38B receives user input by detecting the user's voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0027] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form perceptible to the user 20 (e.g., audio and / or text). The display 40A displays visible information such as text and images according to instructions from the processor 46. The speaker 40B outputs audio according to instructions from the processor 46. The camera 42 is a small digital camera equipped with an optical system such as a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0028] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various types of information between processor 46 and processor 28 via network 54.

[0029] Figure 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0030] As shown in Figure 2, in the data processing device 12, a specific processing is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" related to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0031] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0032] In the smart device 14, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The reception output program 60 is used in conjunction with a specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0033] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0034] The present invention is a system for users to improve their sports form in real time, and a specific embodiment thereof is shown below.

[0035] This system uses sensors mounted on devices such as smart glasses to record the user's movements. The smart glasses' camera records the user's movements from multiple angles, while the G-sensor acquires detailed data such as subtle body movements and angles. This collects the basic data necessary to accurately generate a 3D model of the entire movement.

[0036] The collected data is transmitted wirelessly to a cloud server. The server applies an AI algorithm based on the received data to analyze the user's current form. This analysis can identify how the user's movements deviate from ideal motion data.

[0037] Based on the analysis results, the server generates specific suggestions for improving your form. For example, if the body rotation is insufficient in a golf swing, the server will create improvement instructions for the user such as "increase your shoulder rotation a little." The server then quickly transfers the generated feedback to the user's device.

[0038] The device, based on improvement suggestions sent from the server, presents instructions for correcting movements by overlaying them onto the real world within the user's field of view. The smart glasses' display uses, for example, arrows and highlights to clearly indicate the body parts that need correction. It also shows the direction and angle of movement in real time according to the target ideal form, helping the user adjust their movements on the spot.

[0039] For example, if a tennis player uses this system, they can receive feedback while practicing their serve form if their body weight distribution is incorrect. The user can see at a glance which parts of their body they should pay attention to through the smart glasses, allowing them to efficiently correct their movements.

[0040] In this way, this system supports users in the process of acquiring and improving specific sports skills and achieves performance improvement through its unique form analysis.

[0041] The following describes the processing flow.

[0042] Step 1:

[0043] The user puts on the device and begins their sports activity. The device records the user's movements through the camera and acquires detailed movement data using the G-sensor.

[0044] Step 2:

[0045] The terminal transmits real-time collected video data and sensor information to the server via wireless communication. If edge computing is recommended, the terminal can also perform preliminary data processing.

[0046] Step 3:

[0047] The server analyzes the received raw data. First, it uses an AI algorithm to convert the user's actions into a digitized 3D model. This allows the actual actions to be reproduced in three dimensions.

[0048] Step 4:

[0049] The server compares the generated 3D model with ideal motion data and evaluates each movement. Based on its own criteria, the server identifies deviations in the user's form and determines which parts need improvement and to what extent.

[0050] Step 5:

[0051] The server generates improvement plans based on the analysis results. In this process, it generates specific guidance that directly contributes to improving user performance, including, for example, specific methods for adjusting its operation.

[0052] Step 6:

[0053] The server sends the generated improvement suggestions to the user's device. This data includes information necessary for visual feedback.

[0054] Step 7:

[0055] Based on the received data, the device displays improvement suggestions within the user's field of view. Arrows and text appearing within the field of view indicate specific points for correction, allowing the user to adjust their form in real time.

[0056] (Example 1)

[0057] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0058] This invention relates to a system for assisting users in effectively improving specific sports movements. Conventional systems have struggled to analyze user movements in real time and provide immediate, specific feedback based on that analysis. Furthermore, there have been challenges in providing clear visual representations that allow users to intuitively understand where their movements need correction. In addition, there has been a need for technical improvements to enable efficient and secure data communication and analysis.

[0059] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0060] In this invention, the server includes a communication device that wirelessly transmits acquired motion data, an information processing device that analyzes the motion using a generated AI model based on the received data, and a computing device that compares the analyzed motion data with ideal motion data and identifies areas for improvement in the motion. This makes it possible for users to analyze their sports movements in real time and be presented with specific and visual suggestions for necessary corrections.

[0061] An "information acquisition device" is a device that records a user's actions and collects those movements as data.

[0062] A "communication device" is a device that transmits acquired data to an external source using communication means such as wireless communication.

[0063] An "information processing device" is a device used to perform calculations and analyses based on received data.

[0064] A "generative AI model" is a model that uses machine learning to analyze behavioral data and understand its characteristics.

[0065] A "calculation unit" is a device that compares analyzed data with ideal data and derives improvements to the operation as numerical values ​​or evaluations.

[0066] A "generation device" is a device that generates specific correction instructions for the user based on the results obtained from the computing device.

[0067] A "display device" is a device that presents visual information to the user and delivers instructions for correcting actions.

[0068] This invention provides a system for users to improve their sports form in real time. This system includes integrated technology for accurately recording, analyzing, and providing feedback on the user's physical movements.

[0069] This system utilizes smart glasses and sensor devices as information acquisition devices. These devices are equipped with cameras and accelerometers to accurately record the user's movements. This allows for the collection of detailed data about the user's actions.

[0070] The collected data is transmitted to a cloud server via a communication device. This communication utilizes wireless technologies such as Wi-Fi and Bluetooth to ensure stable and high-speed data transfer.

[0071] The server uses an information processing device to analyze the received data. This analysis employs a method that compares motion data with ideal motion patterns using a generative AI model. The AI ​​model learns the details of the user's movements and automatically detects inappropriate actions.

[0072] Based on the analysis results, the computing unit quantifies areas for improvement in the user's operation and identifies specific correction points. Correction instructions are generated by the generation unit and produced in a format that is easy for the user to understand.

[0073] Finally, the generated instructions are communicated to the user through a display device. The smart glasses' display provides real-time feedback for correcting actions, visually indicating points that need correction using arrows and highlights.

[0074] As a concrete example, when practicing a tennis serve, this system allows the user to instantly correct insufficient weight transfer or shoulder rotation. The input to the generating AI model is a prompt such as, "Generate feedback on weight transfer and shoulder rotation to improve my tennis serve form."

[0075] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0076] Step 1:

[0077] The device records the user's movements. Specifically, the smart glasses' camera captures the user's movements as video, and the accelerometer collects data on body movements. The input is the user's actual movements, and the output is detailed movement data and video data. In this way, information about the entire movement is captured by the device.

[0078] Step 2:

[0079] The terminal sends the collected motion data to the cloud server. The input is the motion data and video data obtained in step 1, and the output is the data sent to the server. Wireless communication technology is used in this process, and the data is appropriately compressed and encrypted before transmission.

[0080] Step 3:

[0081] The server analyzes the received data. The information processing device uses a generative AI model to compare the motion data with ideal motion. The input is the motion data sent in step 2, and the output is the analysis result. Through the analysis, it is possible to obtain numerical values ​​and evaluations of how close the user's motion is to the ideal and any specific differences.

[0082] Step 4:

[0083] The server generates suggestions for improving the operation based on the analysis results. The input is the analysis results from step 3, and the output is specific improvement instructions for the user. The generator uses the prompt sentences obtained from the generated AI model to create instructions such as "You should pull your shoulders back more."

[0084] Step 5:

[0085] The device visually displays the improvement instructions received from the server. The input is the improvement instructions from step 4, and the output is the correction points displayed on the smart glasses' screen. The smart glasses use arrows and highlights to show the user where the behavior needs to be corrected, supporting real-time improvements.

[0086] (Application Example 1)

[0087] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0088] Automated machinery in manufacturing, particularly factory robots, sometimes exhibits inefficient or inaccurate movements. This can lead to decreased production efficiency and variations in product quality. Conventional systems often lack sufficient real-time analysis and improvement instructions, and visual feedback is limited, making rapid correction difficult. Therefore, there is a need for a system that can record, analyze, and efficiently correct factory robot movements in real time.

[0089] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0090] In this invention, the server includes means for using a detection device to record user movements or machine movements, means for using a computer to analyze the recorded movement data by comparing it with ideal movement information, and means for using a display device to present correction instructions to the user or machine based on the analysis results. This enables real-time analysis of factory robot movements, allowing for rapid improvement of operational efficiency and accuracy.

[0091] A "detection device" is a device used to sense and record the actions of a user or machine.

[0092] "Motion data" refers to information about the movements of the user or machine acquired by a detection device.

[0093] "Ideal motion information" refers to data about motion that serves as a standard for improving the efficiency and precision of motion.

[0094] A "computer" is a computer system used to analyze recorded motion data by comparing it with ideal motion information.

[0095] An "auxiliary device" is a device that instructs the user or machine to modify its operation based on the analyzed areas for improvement.

[0096] A "control device" is a device used to record and improve the operation of a machine in real time.

[0097] A "display device" is a device that visually displays instructions for correcting operations to users or operators.

[0098] "Three-dimensional representation" is a format that expresses recorded motion information in a three-dimensional form.

[0099] The system realizing this invention uses a device equipped with multiple sensors and cameras as a detection device to record the movements of the user and the machine. The movement data is transmitted to a server in real time. On the server, based on the received movement data, a generating AI model is used to perform comparative analysis with ideal movement information. Based on the analysis results, the computer identifies areas for improvement and generates specific correction instructions.

[0100] These correction instructions are presented visually through a terminal displayed in the user's field of view or through a display device for the operator. This could include, for example, displaying arrows or highlights on the user's smart glasses. Furthermore, when correcting the operation of a factory robot, this is done via a control device.

[0101] For example, if a robotic arm in a factory makes an angle error while precisely tightening a screw, the server generates an instruction such as "Rotate the arm 5 degrees to the right" and displays it on the display screen. This allows the operator to correct the error immediately.

[0102] An example of a prompt to be input to the generating AI model is, "Generate correction instructions to optimize the robot arm angle and improve the accuracy of screw tightening." Through this form, the present invention can efficiently and effectively improve the operation of the user and the machine, thereby increasing productivity.

[0103] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0104] Step 1:

[0105] The terminal uses detection devices to record user or machine movements in real time. During this process, cameras and multiple sensors are used to acquire video data and angle information of the movements. The input is the user or machine movement itself, and the output is video data and movement data.

[0106] Step 2:

[0107] The terminal wirelessly transmits recorded motion data to the server. This data contains detailed numerical information about the motion, which the server receives. The input consists of previously acquired video data and motion data, forming a dataset ready to be transferred to the server as output.

[0108] Step 3:

[0109] The server inputs the received behavioral data into the generating AI model and analyzes it while comparing it with ideal behavioral information. This process uses prompts to request optimization of the behavior. The input is the behavioral data received by the server, and through processing by the generating AI model, areas for improvement in the behavior are identified as output.

[0110] Step 4:

[0111] The server generates correction instructions for the user or operator based on the areas for improvement identified through analysis. Specifically, it creates instructions that clearly specify which parts need to be corrected and how. The input is the server's analysis results, and the output is the specific correction instructions.

[0112] Step 5:

[0113] Correction instructions generated by the server are visually presented to the user or operator via the terminal's display device. In this case, arrows and highlights are used on the smart glasses' display. The input is correction instructions, and the output is visual information displayed in the user's field of view or on the operator's screen.

[0114] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0115] This invention is a sports form improvement system that combines an emotion engine that recognizes the user's emotions. This configuration is shown below.

[0116] This system incorporates an emotion engine that recognizes the user's emotional state in real time, in addition to existing sports form analysis devices that record and analyze user movements. Devices such as smart glasses collect physiological data such as the user's facial expressions, voice, and heart rate through cameras and sensors.

[0117] The device first sends video and data related to sports movements to a cloud server, where AI is used to analyze the movements. Next, the device separately sends collected emotional data, and the emotion engine evaluates the user's current emotional state. This evaluation is performed using facial expression analysis, voice tone analysis, or physiological data. The emotion engine on the server compares the current emotional data with past emotional history to track changes in the user's emotions.

[0118] The server adjusts the analyzed behavior improvement suggestions based on emotional information obtained from the emotion engine. For example, it optimizes the content and timing of the behavior improvement suggestions so that users receive them more positively. This process is designed to ensure that the feedback provided does not cause unnecessary stress to the user and that motivation is strengthened.

[0119] The device displays emotionally-adjusted feedback within the user's field of vision. This feedback changes to simpler instructions when the user is excited, while providing detailed suggestions for correcting their actions when they are relaxed, displaying information tailored to their individual emotional state.

[0120] For example, if a user is feeling stressed during tennis practice, the device's emotion engine will recognize this state. The server will then prioritize presenting only small, easily actionable improvement suggestions to address that emotion. This allows the user to effectively work on improving their form without being overwhelmed by excessive information. In this way, sports form analysis that takes user emotions into account contributes to improved performance.

[0121] The following describes the processing flow.

[0122] Step 1:

[0123] The user puts on the device and begins their sports activity. The device uses a camera and sensors to simultaneously record the user's movements, facial expressions, and voice.

[0124] Step 2:

[0125] The device transmits recorded motion and emotional data to the server. This data is necessary to simultaneously analyze the user's sports form and emotional state.

[0126] Step 3:

[0127] The server analyzes the received motion data using an AI algorithm and converts the user's current form into a 3D model. This allows for a three-dimensional reproduction of the motion details.

[0128] Step 4:

[0129] The server analyzes the user's emotional data using an emotion engine. Through facial expression analysis, voice tone analysis, and evaluation of physiological data, it identifies the user's current emotional state.

[0130] Step 5:

[0131] The server integrates the results of the motion analysis with the emotional state generated by the emotion engine to generate motion improvement suggestions optimized for the user's state. This includes the timing and content of appropriate feedback.

[0132] Step 6:

[0133] The server sends the generated improvement suggestions to the user's device. The feedback is adjusted to reflect the user's current emotional state.

[0134] Step 7:

[0135] The device displays received feedback in real time within the user's field of view. The explanation of improvement suggestions changes in conciseness or detail depending on the user's emotions. The user then adjusts their actions and improves the form based on this feedback.

[0136] (Example 2)

[0137] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0138] In improving athletic performance in sports and other activities, conventional systems have only analyzed the user's exercise data and suggested areas for improvement. However, when a user's emotional state influences their athletic performance and acceptance of feedback, mere data analysis alone is insufficient to achieve adequate improvement. Therefore, it is necessary to integrate and analyze both exercise and emotional data to enable users to work on improvement more effectively.

[0139] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0140] In this invention, the server includes means for using a detector to record the user's movements, means for using an information processing device to receive the recorded movement data and compare it with ideal movement information, means for using an emotion analysis device to evaluate the user's emotional information, and means for using an auxiliary device to adjust and present the analyzed areas for improvement based on the emotional information. This makes it possible to provide optimal feedback that takes into account the user's movements and emotional state.

[0141] A "user" is the individual who performs the movement, and is the subject that is recorded and analyzed by the system.

[0142] "Exercise" refers to physical movements performed by the user and the series of activities associated with them.

[0143] A "detector" is a device used to record a user's movements, and includes cameras, sensors, and other similar components.

[0144] "Motion data" refers to information about the user's physical movements recorded by the detector.

[0145] An "information processing device" is a device that receives motion data and has a computational function to analyze areas for improvement by comparing it with ideal motion information.

[0146] "Ideal exercise information" refers to a standard that indicates desirable exercise performance and is used for comparison with exercise data.

[0147] "Emotional information" refers to data that indicates the user's psychological or physiological state and is evaluated by an emotion analysis device.

[0148] An "emotion analysis device" is a device used to evaluate a user's emotional information, performing tasks such as facial expression analysis and voice tone analysis.

[0149] An "auxiliary device" is a device that adjusts the analyzed areas for improvement based on emotional information and presents them to the user.

[0150] This invention is a system that supports users in improving their exercise performance by analyzing exercise data and emotional information to provide feedback. The aim of this system is to simultaneously grasp the exercise data and emotional state of the user when they engage in sports or other exercise, and to suggest effective improvement plans.

[0151] First, the device utilizes detectors such as cameras and sensors to collect data from users exercising. This device not only records the user's movements as video but also acquires physiological data such as audio and heart rate. This information records the user's real-time exercise status and is used as foundational data for subsequent analysis.

[0152] Next, the data obtained from this device is sent to a server. The server analyzes this data using AI technology. Specifically, it uses a generative AI model to compare the recorded exercise data with ideal movement information and extract areas for improvement in the exercise. In addition, the server evaluates emotional information from the user's facial expressions, voice tone, and physiological data through an emotion analysis device. This allows the server to understand the emotions the user is experiencing during exercise, such as stress and relaxation.

[0153] The server integrates the analysis results of this movement with emotional information to generate optimal feedback for the user. The assistive device adjusts this feedback in a way that is easily accepted by the user and presents it as a final improvement plan. This improvement plan is adjusted in various ways depending on the user's emotional state, ranging from complex technical explanations to concise advice.

[0154] For example, if a user is feeling stressed during tennis practice, the device might offer a short instruction such as, "Relax your shoulders and take a deep breath," guiding the user towards effective improvement without overwhelming them with excessive information.

[0155] An example of a prompt would be, "Provide the optimal exercise improvement plan for a user experiencing stress during tennis practice." This prompt is used as input for analysis by a generative AI model.

[0156] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0157] Step 1:

[0158] The device records the user's exercise. Specifically, it records video with a camera, collects audio with a microphone, and collects physiological data with a heart rate sensor. The data collected in this process reflects the user's real-time exercise status and emotional state. The input is the user's exercise and physiological responses, and the output is the recorded data of these.

[0159] Step 2:

[0160] The device transmits the collected movement and emotion data to the server. This communication takes place via a high-speed internet connection. The input is the recorded data generated in step 1, and the output is the completion of data transmission to the server.

[0161] Step 3:

[0162] The server analyzes the received motion data. Using a generative AI model, it identifies areas for improvement by comparing the data with ideal motion information. In this analysis, swing speed, angle, trajectory, etc., are compared to a baseline. The input is the motion data transmitted from the terminal, and the output is the analysis results, including the areas for improvement.

[0163] Step 4:

[0164] The server evaluates emotional information using an emotion analysis device. Based on facial expression analysis, voice tone analysis, and physiological data, it classifies the user's emotions with labels such as "stress" or "relaxed." The input is emotion-related data transmitted from the terminal, and the output is the evaluated emotional information.

[0165] Step 5:

[0166] The server integrates movement analysis results and emotional information. It optimizes the analyzed improvement suggestions based on the emotional information and generates feedback that is easily accepted by the user. Depending on the user's state, it adjusts the content to provide detailed explanations or concise advice. The input is the movement analysis results and emotional information, and the output is the adjusted feedback.

[0167] Step 6:

[0168] The terminal presents feedback sent from the server to the user. For example, it may display simple instructions on the screen or detailed explanations in text. The input is feedback information sent from the server, and the output is a visual presentation to the user.

[0169] (Application Example 2)

[0170] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as a "server" and the smart device 14 as a "terminal".

[0171] The objective of this invention is to improve the inefficiency caused by failing to consider the user's emotional state when improving user movements. Specifically, it aims to improve efficient exercise form while strengthening motivation by adjusting suggested movements according to whether the user is stressed or relaxed, and by providing optimal feedback to the user.

[0172] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0173] In this invention, the server includes means for collecting data using sensors that record the user's movements; means for using a computer that compares the recorded movement data with ideal movement data and analyzes areas for improvement in the movements; means for using an emotion engine that adjusts the analyzed areas for improvement based on the user's emotional state and generates feedback; and means for using a support device that presents the user with feedback corresponding to the analyzed areas for improvement and emotional state to correct the movements. This makes it possible to provide the user with movement correction feedback that takes emotions into account.

[0174] A "user" refers to an individual who uses the system to analyze and improve its operation.

[0175] "Motion data" refers to digital data that records information about a user's movements, posture, and actions.

[0176] A "sensor" refers to a device that acquires information from the physical environment and converts it into digital data.

[0177] A "computer" refers to an electronic device used to process and analyze digital data.

[0178] "Ideal behavioral data" refers to model data that serves as a standard for evaluating user behavior.

[0179] "Areas for improvement" refers to parts of the operation that need improvement, identified by comparing the current operation data with ideal operation data.

[0180] An "emotion engine" refers to software that recognizes and analyzes a user's emotional state in real time.

[0181] "Feedback" refers to information and instructions provided to the user based on behavioral analysis and emotional state.

[0182] "Support device" refers to equipment used to provide users with information to improve their performance.

[0183] "Emotional state" refers to the user's current psychological or emotional state.

[0184] This invention is a system for integrating and processing user behavior and emotional state. The main components of this system are sensors for recording movement, a computer for processing the data, an emotion engine for analyzing emotions, and a support device for providing feedback.

[0185] The sensors use cameras and biometric devices to collect user movement data and physiological information such as facial expressions, voice, and heart rate. This allows for real-time acquisition of data on the user's movements and emotional state. The device sends this data to the cloud, where a computer compares the movement data to ideal movement data and analyzes areas for improvement.

[0186] The server analyzes user behavior on cloud computing resources and evaluates user emotions. The emotion engine integrates facial expression analysis, voice tone, and physiological information to understand emotional states and tracks changes from past history. Common platforms such as Amazon Rekognition and TENSORFLOW® are used as emotion engines.

[0187] The support device provides the user with feedback tailored to their emotional state. The feedback is simplified and easy to follow when the user is stressed, and provides more detailed instructions when they are relaxed. For example, if the emotional engine determines that a user working in a factory is stressed, a simple instruction such as "We recommend taking a 15-minute break" is provided.

[0188] This system provides feedback to improve user behavior based on their emotional state, enabling a more efficient and comfortable work environment.

[0189] An example of a prompt would be: "To improve work efficiency in the factory, propose emotion-recognition-based feedback methods. In particular, describe specific approaches to stress reduction."

[0190] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0191] Step 1:

[0192] The device collects user motion and physiological data using sensors. It acquires video and heart rate data using cameras and heart rate monitors, recording them as digital data. Inputs include user motion, facial expressions, voice, and physiological data, which are captured by the sensors. Outputs are generated in real time and transmitted to the next processing step.

[0193] Step 2:

[0194] The device sends the acquired data to a cloud server. Using data transfer methods such as Bluetooth or Wi-Fi, the collected behavioral and physiological data are uploaded to the cloud service. The input is the data transmitted from the device, and the output is the data transferred to the cloud server. This data is used for analysis on the server.

[0195] Step 3:

[0196] The server analyzes the behavior using the received data. The computer compares the behavior data to an ideal behavior data model to identify areas for improvement. The input is behavior data uploaded to the cloud. The output is a list of identified behavioral improvements. An AI analysis engine can be utilized in this process.

[0197] Step 4:

[0198] The server uses an emotion engine to evaluate the user's emotions. It integrates received facial expressions, voice, and physiological data to analyze the user's emotional state. The input is physiological data transferred to the cloud. The output is the evaluation result of the user's emotional state. Common AI tools can be used for the emotion engine.

[0199] Step 5:

[0200] The server generates feedback that takes into account emotional states in relation to improvements made to the system's operation. Based on the analysis results, it creates optimal feedback that matches the user's emotions. The input consists of improvements made to the system and the emotional evaluation results. The output is emotionally adjusted feedback for the user. This information is formatted for delivery.

[0201] Step 6:

[0202] The terminal provides feedback to the user. Through the assistive device, feedback tailored to the user's current situation is displayed from the terminal. For example, smart glasses might display "Take a break and prepare to proceed further." The input is the feedback content generated by the server, and the output is the feedback displayed in the user's field of vision.

[0203] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0204] Data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of data generation model 58 is ChatGPT (registered trademark) (Internet search).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0205] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart device 14.

[0206] [Second Embodiment]

[0207] Figure 3 shows an example of the configuration of the data processing system 210 according to the second embodiment.

[0208] As shown in Figure 3, the data processing system 210 includes a data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0209] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0210] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication interface 44. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, and camera 42 are also connected to the bus 52.

[0211] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0212] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0213] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0214] Figure 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Figure 4, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0215] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0216] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0217] In the smart glasses 214, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0218] Next, the identification processing performed by the identification processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0219] The present invention is a system for users to improve their sports form in real time, and a specific embodiment thereof is shown below.

[0220] This system uses sensors mounted on devices such as smart glasses to record the user's movements. The smart glasses' camera records the user's movements from multiple angles, while the G-sensor acquires detailed data such as subtle body movements and angles. This collects the basic data necessary to accurately generate a 3D model of the entire movement.

[0221] The collected data is transmitted wirelessly to a cloud server. The server applies an AI algorithm based on the received data to analyze the user's current form. This analysis can identify how the user's movements deviate from ideal motion data.

[0222] Based on the analysis results, the server generates specific suggestions for improving your form. For example, if the body rotation is insufficient in a golf swing, the server will create improvement instructions for the user such as "increase your shoulder rotation a little." The server then quickly transfers the generated feedback to the user's device.

[0223] The device, based on improvement suggestions sent from the server, presents instructions for correcting movements by overlaying them onto the real world within the user's field of view. The smart glasses' display uses, for example, arrows and highlights to clearly indicate the body parts that need correction. It also shows the direction and angle of movement in real time according to the target ideal form, helping the user adjust their movements on the spot.

[0224] For example, if a tennis player uses this system, they can receive feedback while practicing their serve form if their body weight distribution is incorrect. The user can see at a glance which parts of their body they should pay attention to through the smart glasses, allowing them to efficiently correct their movements.

[0225] In this way, this system supports users in the process of acquiring and improving specific sports skills and achieves performance improvement through its unique form analysis.

[0226] The following describes the processing flow.

[0227] Step 1:

[0228] The user puts on the device and begins their sports activity. The device records the user's movements through the camera and acquires detailed movement data using the G-sensor.

[0229] Step 2:

[0230] The terminal transmits real-time collected video data and sensor information to the server via wireless communication. If edge computing is recommended, the terminal can also perform preliminary data processing.

[0231] Step 3:

[0232] The server analyzes the received raw data. First, it uses an AI algorithm to convert the user's actions into a digitized 3D model. This allows the actual actions to be reproduced in three dimensions.

[0233] Step 4:

[0234] The server compares the generated 3D model with ideal motion data and evaluates each movement. Based on its own criteria, the server identifies deviations in the user's form and determines which parts need improvement and to what extent.

[0235] Step 5:

[0236] The server generates improvement plans based on the analysis results. In this process, it generates specific guidance that directly contributes to improving user performance, including, for example, specific methods for adjusting its operation.

[0237] Step 6:

[0238] The server sends the generated improvement suggestions to the user's device. This data includes information necessary for visual feedback.

[0239] Step 7:

[0240] Based on the received data, the device displays improvement suggestions within the user's field of view. Arrows and text appearing within the field of view indicate specific points for correction, allowing the user to adjust their form in real time.

[0241] (Example 1)

[0242] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0243] This invention relates to a system for assisting users in effectively improving specific sports movements. Conventional systems have struggled to analyze user movements in real time and provide immediate, specific feedback based on that analysis. Furthermore, there have been challenges in providing clear visual representations that allow users to intuitively understand where their movements need correction. In addition, there has been a need for technical improvements to enable efficient and secure data communication and analysis.

[0244] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0245] In this invention, the server includes a communication device that wirelessly transmits acquired motion data, an information processing device that analyzes the motion using a generated AI model based on the received data, and a computing device that compares the analyzed motion data with ideal motion data and identifies areas for improvement in the motion. This makes it possible for users to analyze their sports movements in real time and be presented with specific and visual suggestions for necessary corrections.

[0246] An "information acquisition device" is a device that records a user's actions and collects those movements as data.

[0247] A "communication device" is a device that transmits acquired data to an external source using communication means such as wireless communication.

[0248] An "information processing device" is a device used to perform calculations and analyses based on received data.

[0249] A "generative AI model" is a model that uses machine learning to analyze behavioral data and understand its characteristics.

[0250] A "calculation unit" is a device that compares analyzed data with ideal data and derives improvements to the operation as numerical values ​​or evaluations.

[0251] A "generation device" is a device that generates specific correction instructions for the user based on the results obtained from the computing device.

[0252] A "display device" is a device that presents visual information to the user and delivers instructions for correcting actions.

[0253] This invention provides a system for users to improve their sports form in real time. This system includes integrated technology for accurately recording, analyzing, and providing feedback on the user's physical movements.

[0254] This system utilizes smart glasses and sensor devices as information acquisition devices. These devices are equipped with cameras and accelerometers to accurately record the user's movements. This allows for the collection of detailed data about the user's actions.

[0255] The collected data is transmitted to a cloud server via a communication device. This communication utilizes wireless technologies such as Wi-Fi and Bluetooth to ensure stable and high-speed data transfer.

[0256] The server uses an information processing device to analyze the received data. This analysis employs a method that compares motion data with ideal motion patterns using a generative AI model. The AI ​​model learns the details of the user's movements and automatically detects inappropriate actions.

[0257] Based on the analysis results, the computing unit quantifies areas for improvement in the user's operation and identifies specific correction points. Correction instructions are generated by the generation unit and produced in a format that is easy for the user to understand.

[0258] Finally, the generated instructions are communicated to the user through a display device. The smart glasses' display provides real-time feedback for correcting actions, visually indicating points that need correction using arrows and highlights.

[0259] As a concrete example, when practicing a tennis serve, this system allows the user to instantly correct insufficient weight transfer or shoulder rotation. The input to the generating AI model is a prompt such as, "Generate feedback on weight transfer and shoulder rotation to improve my tennis serve form."

[0260] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0261] Step 1:

[0262] The device records the user's movements. Specifically, the smart glasses' camera captures the user's movements as video, and the accelerometer collects data on body movements. The input is the user's actual movements, and the output is detailed movement data and video data. In this way, information about the entire movement is captured by the device.

[0263] Step 2:

[0264] The terminal sends the collected motion data to the cloud server. The input is the motion data and video data obtained in step 1, and the output is the data sent to the server. Wireless communication technology is used in this process, and the data is appropriately compressed and encrypted before transmission.

[0265] Step 3:

[0266] The server analyzes the received data. The information processing device uses a generative AI model to compare the motion data with ideal motion. The input is the motion data sent in step 2, and the output is the analysis result. Through the analysis, it is possible to obtain numerical values ​​and evaluations of how close the user's motion is to the ideal and any specific differences.

[0267] Step 4:

[0268] The server generates suggestions for improving the operation based on the analysis results. The input is the analysis results from step 3, and the output is specific improvement instructions for the user. The generator uses the prompt sentences obtained from the generated AI model to create instructions such as "You should pull your shoulders back more."

[0269] Step 5:

[0270] The device visually displays the improvement instructions received from the server. The input is the improvement instructions from step 4, and the output is the correction points displayed on the smart glasses' screen. The smart glasses use arrows and highlights to show the user where the behavior needs to be corrected, supporting real-time improvements.

[0271] (Application Example 1)

[0272] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0273] Automated machinery in manufacturing, particularly factory robots, sometimes exhibits inefficient or inaccurate movements. This can lead to decreased production efficiency and variations in product quality. Conventional systems often lack sufficient real-time analysis and improvement instructions, and visual feedback is limited, making rapid correction difficult. Therefore, there is a need for a system that can record, analyze, and efficiently correct factory robot movements in real time.

[0274] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0275] In this invention, the server includes means for using a detection device to record user movements or machine movements, means for using a computer to analyze the recorded movement data by comparing it with ideal movement information, and means for using a display device to present correction instructions to the user or machine based on the analysis results. This enables real-time analysis of factory robot movements, allowing for rapid improvement of operational efficiency and accuracy.

[0276] A "detection device" is a device used to sense and record the actions of a user or machine.

[0277] "Motion data" refers to information about the movements of the user or machine acquired by a detection device.

[0278] "Ideal motion information" refers to data about motion that serves as a standard for improving the efficiency and precision of motion.

[0279] A "computer" is a computer system used to analyze recorded motion data by comparing it with ideal motion information.

[0280] An "auxiliary device" is a device that instructs the user or machine to modify its operation based on the analyzed areas for improvement.

[0281] A "control device" is a device used to record and improve the operation of a machine in real time.

[0282] A "display device" is a device that visually displays instructions for correcting operations to users or operators.

[0283] "Three-dimensional representation" is a format that expresses recorded motion information in a three-dimensional form.

[0284] In the system that realizes this invention, a device equipped with a plurality of sensors and a camera is used as a detection device to record the actions of users and machines. The operation data is sent to the server in real time. Based on the received operation data, the server performs comparative analysis with ideal operation information using a generated AI model. Based on the analysis results, the computer identifies areas for improvement and generates specific correction instructions.

[0285] This correction instruction is visually presented through a terminal displayed within the user's field of vision or a display device for the operator. This can be done, for example, by displaying arrows or highlights on the user's smart glasses. Also, when correcting the operation of a factory robot, it is implemented via a control device.

[0286] As a specific example, if an angular error occurs when a robot arm accurately tightens a screw in a factory, the server generates an instruction such as "Please rotate the arm 5 degrees to the right" and presents it on the display device. This enables the operator to make corrections immediately.

[0287] An example of a prompt sentence input to the generated AI model is "Please generate correction instructions to optimize the angle of the robot arm and improve the accuracy of screw tightening." Through this form, the present invention can efficiently and effectively improve the operations of users and machines, and achieve improved productivity.

[0288] The flow of specific processing in Application Example 1 will be described using FIG. 12.

[0289] Step 1:

[0290] The terminal uses a detection device to record the actions of the user or machine in real time. At this time, a camera and a plurality of sensors are used to obtain video data and angle information of the action. What is input is the action of the user or machine itself, and what is obtained as output is video data and operation data.

[0291] Step 2:

[0292] The terminal wirelessly transmits recorded motion data to the server. This data contains detailed numerical information about the motion, which the server receives. The input consists of previously acquired video data and motion data, forming a dataset ready to be transferred to the server as output.

[0293] Step 3:

[0294] The server inputs the received behavioral data into the generating AI model and analyzes it while comparing it with ideal behavioral information. This process uses prompts to request optimization of the behavior. The input is the behavioral data received by the server, and through processing by the generating AI model, areas for improvement in the behavior are identified as output.

[0295] Step 4:

[0296] The server generates correction instructions for the user or operator based on the areas for improvement identified through analysis. Specifically, it creates instructions that clearly specify which parts need to be corrected and how. The input is the server's analysis results, and the output is the specific correction instructions.

[0297] Step 5:

[0298] Correction instructions generated by the server are visually presented to the user or operator via the terminal's display device. In this case, arrows and highlights are used on the smart glasses' display. The input is correction instructions, and the output is visual information displayed in the user's field of view or on the operator's screen.

[0299] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0300] This invention is a sports form improvement system that combines an emotion engine that recognizes the user's emotions. This configuration is shown below.

[0301] This system incorporates an emotion engine that recognizes the user's emotional state in real time, in addition to existing sports form analysis devices that record and analyze user movements. Devices such as smart glasses collect physiological data such as the user's facial expressions, voice, and heart rate through cameras and sensors.

[0302] The device first sends video and data related to sports movements to a cloud server, where AI is used to analyze the movements. Next, the device separately sends collected emotional data, and the emotion engine evaluates the user's current emotional state. This evaluation is performed using facial expression analysis, voice tone analysis, or physiological data. The emotion engine on the server compares the current emotional data with past emotional history to track changes in the user's emotions.

[0303] The server adjusts the analyzed behavior improvement suggestions based on emotional information obtained from the emotion engine. For example, it optimizes the content and timing of the behavior improvement suggestions so that users receive them more positively. This process is designed to ensure that the feedback provided does not cause unnecessary stress to the user and that motivation is strengthened.

[0304] The device displays emotionally-adjusted feedback within the user's field of vision. This feedback changes to simpler instructions when the user is excited, while providing detailed suggestions for correcting their actions when they are relaxed, displaying information tailored to their individual emotional state.

[0305] For example, when a user feels stressed during tennis practice, the state is recognized by the emotion engine of the terminal. In order to respond to that emotion, the server preferentially presents only small improvement actions that the user can easily execute. As a result, the user is not overwhelmed by excessive information and can effectively work on form improvement. By realizing sports form analysis that takes into account the user's emotions in this way, it contributes to performance improvement.

[0306] The processing flow will be described below.

[0307] Step 1:

[0308] The user wears the terminal and starts a sports activity. The terminal uses a camera and sensors to simultaneously record the user's movements, expressions, and voice.

[0309] Step 2:

[0310] The terminal sends the recorded motion data and emotion data to the server. This data is necessary to simultaneously analyze the user's sports form and emotional state.

[0311] Step 3:

[0312] The server analyzes the received motion data using an AI algorithm and converts the user's current form into a 3D model. As a result, the details of the movement are reproduced three-dimensionally.

[0313] Step 4:

[0314] The server analyzes the user's emotion data using an emotion engine. By analyzing facial expressions, voice tones, and physiological data, the server identifies the user's current emotional state.

[0315] Step 5:

[0316] The server integrates the results of the motion analysis with the emotional state generated by the emotion engine to generate motion improvement suggestions optimized for the user's state. This includes the timing and content of appropriate feedback.

[0317] Step 6:

[0318] The server sends the generated improvement suggestions to the user's device. The feedback is adjusted to reflect the user's current emotional state.

[0319] Step 7:

[0320] The device displays received feedback in real time within the user's field of view. The explanation of improvement suggestions changes in conciseness or detail depending on the user's emotions. The user then adjusts their actions and improves the form based on this feedback.

[0321] (Example 2)

[0322] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0323] In improving athletic performance in sports and other activities, conventional systems have only analyzed the user's exercise data and suggested areas for improvement. However, when a user's emotional state influences their athletic performance and acceptance of feedback, mere data analysis alone is insufficient to achieve adequate improvement. Therefore, it is necessary to integrate and analyze both exercise and emotional data to enable users to work on improvement more effectively.

[0324] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0325] In this invention, the server includes means for using a detector to record the user's movements, means for using an information processing device to receive the recorded movement data and compare it with ideal movement information, means for using an emotion analysis device to evaluate the user's emotional information, and means for using an auxiliary device to adjust and present the analyzed areas for improvement based on the emotional information. This makes it possible to provide optimal feedback that takes into account the user's movements and emotional state.

[0326] A "user" is the individual who performs the movement, and is the subject that is recorded and analyzed by the system.

[0327] "Exercise" refers to physical movements performed by the user and the series of activities associated with them.

[0328] A "detector" is a device used to record a user's movements, and includes cameras, sensors, and other similar components.

[0329] "Motion data" refers to information about the user's physical movements recorded by the detector.

[0330] An "information processing device" is a device that receives motion data and has a computational function to analyze areas for improvement by comparing it with ideal motion information.

[0331] "Ideal exercise information" refers to a standard that indicates desirable exercise performance and is used for comparison with exercise data.

[0332] "Emotional information" refers to data that indicates the user's psychological or physiological state and is evaluated by an emotion analysis device.

[0333] An "emotion analysis device" is a device used to evaluate a user's emotional information, performing tasks such as facial expression analysis and voice tone analysis.

[0334] An "auxiliary device" is a device that adjusts the analyzed areas for improvement based on emotional information and presents them to the user.

[0335] This invention is a system that supports users in improving their exercise performance by analyzing exercise data and emotional information to provide feedback. The aim of this system is to simultaneously grasp the exercise data and emotional state of the user when they engage in sports or other exercise, and to suggest effective improvement plans.

[0336] First, the device utilizes detectors such as cameras and sensors to collect data from users exercising. This device not only records the user's movements as video but also acquires physiological data such as audio and heart rate. This information records the user's real-time exercise status and is used as foundational data for subsequent analysis.

[0337] Next, the data obtained from this device is sent to a server. The server analyzes this data using AI technology. Specifically, it uses a generative AI model to compare the recorded exercise data with ideal movement information and extract areas for improvement in the exercise. In addition, the server evaluates emotional information from the user's facial expressions, voice tone, and physiological data through an emotion analysis device. This allows the server to understand the emotions the user is experiencing during exercise, such as stress and relaxation.

[0338] The server integrates the analysis results of this movement with emotional information to generate optimal feedback for the user. The assistive device adjusts this feedback in a way that is easily accepted by the user and presents it as a final improvement plan. This improvement plan is adjusted in various ways depending on the user's emotional state, ranging from complex technical explanations to concise advice.

[0339] For example, if a user is feeling stressed during tennis practice, the device might offer a short instruction such as, "Relax your shoulders and take a deep breath," guiding the user towards effective improvement without overwhelming them with excessive information.

[0340] An example of a prompt would be, "Provide the optimal exercise improvement plan for a user experiencing stress during tennis practice." This prompt is used as input for analysis by a generative AI model.

[0341] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0342] Step 1:

[0343] The device records the user's exercise. Specifically, it records video with a camera, collects audio with a microphone, and collects physiological data with a heart rate sensor. The data collected in this process reflects the user's real-time exercise status and emotional state. The input is the user's exercise and physiological responses, and the output is the recorded data of these.

[0344] Step 2:

[0345] The device transmits the collected movement and emotion data to the server. This communication takes place via a high-speed internet connection. The input is the recorded data generated in step 1, and the output is the completion of data transmission to the server.

[0346] Step 3:

[0347] The server analyzes the received motion data. Using a generative AI model, it identifies areas for improvement by comparing the data with ideal motion information. In this analysis, swing speed, angle, trajectory, etc., are compared to a baseline. The input is the motion data transmitted from the terminal, and the output is the analysis results, including the areas for improvement.

[0348] Step 4:

[0349] The server evaluates emotional information using an emotion analysis device. Based on facial expression analysis, voice tone analysis, and physiological data, it classifies the user's emotions with labels such as "stress" or "relaxed." The input is emotion-related data transmitted from the terminal, and the output is the evaluated emotional information.

[0350] Step 5:

[0351] The server integrates movement analysis results and emotional information. It optimizes the analyzed improvement suggestions based on the emotional information and generates feedback that is easily accepted by the user. Depending on the user's state, it adjusts the content to provide detailed explanations or concise advice. The input is the movement analysis results and emotional information, and the output is the adjusted feedback.

[0352] Step 6:

[0353] The terminal presents feedback sent from the server to the user. For example, it may display simple instructions on the screen or detailed explanations in text. The input is feedback information sent from the server, and the output is a visual presentation to the user.

[0354] (Application Example 2)

[0355] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0356] The objective of this invention is to improve the inefficiency caused by failing to consider the user's emotional state when improving user movements. Specifically, it aims to improve efficient exercise form while strengthening motivation by adjusting suggested movements according to whether the user is stressed or relaxed, and by providing optimal feedback to the user.

[0357] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0358] In this invention, the server includes means for collecting data using sensors that record the user's movements; means for using a computer that compares the recorded movement data with ideal movement data and analyzes areas for improvement in the movements; means for using an emotion engine that adjusts the analyzed areas for improvement based on the user's emotional state and generates feedback; and means for using a support device that presents the user with feedback corresponding to the analyzed areas for improvement and emotional state to correct the movements. This makes it possible to provide the user with movement correction feedback that takes emotions into account.

[0359] A "user" refers to an individual who uses the system to analyze and improve its operation.

[0360] "Motion data" refers to digital data that records information about a user's movements, posture, and actions.

[0361] A "sensor" refers to a device that acquires information from the physical environment and converts it into digital data.

[0362] A "computer" refers to an electronic device used to process and analyze digital data.

[0363] "Ideal behavioral data" refers to model data that serves as a standard for evaluating user behavior.

[0364] "Areas for improvement" refers to parts of the operation that need improvement, identified by comparing the current operation data with ideal operation data.

[0365] An "emotion engine" refers to software that recognizes and analyzes a user's emotional state in real time.

[0366] "Feedback" refers to information and instructions provided to the user based on behavioral analysis and emotional state.

[0367] "Support device" refers to equipment used to provide users with information to improve their performance.

[0368] "Emotional state" refers to the user's current psychological or emotional state.

[0369] This invention is a system for integrating and processing user behavior and emotional state. The main components of this system are sensors for recording movement, a computer for processing the data, an emotion engine for analyzing emotions, and a support device for providing feedback.

[0370] The sensors use cameras and biometric devices to collect user movement data and physiological information such as facial expressions, voice, and heart rate. This allows for real-time acquisition of data on the user's movements and emotional state. The device sends this data to the cloud, where a computer compares the movement data to ideal movement data and analyzes areas for improvement.

[0371] The server analyzes user behavior on cloud computing resources and evaluates user emotions. The emotion engine integrates facial expression analysis, voice tone, and physiological information to understand emotional states and tracks changes from past history. Common platforms such as Amazon Rekognition and TensorFlow are used as emotion engines.

[0372] The support device provides the user with feedback tailored to their emotional state. The feedback is simplified and easy to follow when the user is stressed, and provides more detailed instructions when they are relaxed. For example, if the emotional engine determines that a user working in a factory is stressed, a simple instruction such as "We recommend taking a 15-minute break" is provided.

[0373] This system provides feedback to improve user behavior based on their emotional state, enabling a more efficient and comfortable work environment.

[0374] An example of a prompt would be: "To improve work efficiency in the factory, propose emotion-recognition-based feedback methods. In particular, describe specific approaches to stress reduction."

[0375] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0376] Step 1:

[0377] The device collects user motion and physiological data using sensors. It acquires video and heart rate data using cameras and heart rate monitors, recording them as digital data. Inputs include user motion, facial expressions, voice, and physiological data, which are captured by the sensors. Outputs are generated in real time and transmitted to the next processing step.

[0378] Step 2:

[0379] The device sends the acquired data to a cloud server. Using data transfer methods such as Bluetooth or Wi-Fi, the collected behavioral and physiological data are uploaded to the cloud service. The input is the data transmitted from the device, and the output is the data transferred to the cloud server. This data is used for analysis on the server.

[0380] Step 3:

[0381] The server analyzes the behavior using the received data. The computer compares the behavior data to an ideal behavior data model to identify areas for improvement. The input is behavior data uploaded to the cloud. The output is a list of identified behavioral improvements. An AI analysis engine can be utilized in this process.

[0382] Step 4:

[0383] The server uses an emotion engine to evaluate the user's emotions. It integrates received facial expressions, voice, and physiological data to analyze the user's emotional state. The input is physiological data transferred to the cloud. The output is the evaluation result of the user's emotional state. Common AI tools can be used for the emotion engine.

[0384] Step 5:

[0385] The server generates feedback that takes into account emotional states in relation to improvements made to the system's operation. Based on the analysis results, it creates optimal feedback that matches the user's emotions. The input consists of improvements made to the system and the emotional evaluation results. The output is emotionally adjusted feedback for the user. This information is formatted for delivery.

[0386] Step 6:

[0387] The terminal provides feedback to the user. Through the assistive device, feedback tailored to the user's current situation is displayed from the terminal. For example, smart glasses might display "Take a break and prepare to proceed further." The input is the feedback content generated by the server, and the output is the feedback displayed in the user's field of vision.

[0388] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0389] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0390] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart glasses 214.

[0391] [Third Embodiment]

[0392] Figure 5 shows an example of the configuration of the data processing system 310 according to the third embodiment.

[0393] As shown in Figure 5, the data processing system 310 includes a data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0394] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0395] The headset terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a display 343. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and display 343 are also connected to the bus 52.

[0396] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0397] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0398] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0399] Figure 6 shows an example of the main functions of the data processing device 12 and the headset terminal 314. As shown in Figure 6, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0400] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0401] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0402] In the headset terminal 314, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0403] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the headset terminal 314 will be referred to as the "terminal".

[0404] The present invention is a system for users to improve their sports form in real time, and a specific embodiment thereof is shown below.

[0405] This system uses sensors mounted on devices such as smart glasses to record the user's movements. The smart glasses' camera records the user's movements from multiple angles, while the G-sensor acquires detailed data such as subtle body movements and angles. This collects the basic data necessary to accurately generate a 3D model of the entire movement.

[0406] The collected data is transmitted wirelessly to a cloud server. The server applies an AI algorithm based on the received data to analyze the user's current form. This analysis can identify how the user's movements deviate from ideal motion data.

[0407] Based on the analysis results, the server generates specific suggestions for improving your form. For example, if the body rotation is insufficient in a golf swing, the server will create improvement instructions for the user such as "increase your shoulder rotation a little." The server then quickly transfers the generated feedback to the user's device.

[0408] The device, based on improvement suggestions sent from the server, presents instructions for correcting movements by overlaying them onto the real world within the user's field of view. The smart glasses' display uses, for example, arrows and highlights to clearly indicate the body parts that need correction. It also shows the direction and angle of movement in real time according to the target ideal form, helping the user adjust their movements on the spot.

[0409] For example, if a tennis player uses this system, they can receive feedback while practicing their serve form if their body weight distribution is incorrect. The user can see at a glance which parts of their body they should pay attention to through the smart glasses, allowing them to efficiently correct their movements.

[0410] In this way, this system supports users in the process of acquiring and improving specific sports skills and achieves performance improvement through its unique form analysis.

[0411] The following describes the processing flow.

[0412] Step 1:

[0413] The user puts on the device and begins their sports activity. The device records the user's movements through the camera and acquires detailed movement data using the G-sensor.

[0414] Step 2:

[0415] The terminal transmits real-time collected video data and sensor information to the server via wireless communication. If edge computing is recommended, the terminal can also perform preliminary data processing.

[0416] Step 3:

[0417] The server analyzes the received raw data. First, it uses an AI algorithm to convert the user's actions into a digitized 3D model. This allows the actual actions to be reproduced in three dimensions.

[0418] Step 4:

[0419] The server compares the generated 3D model with ideal motion data and evaluates each movement. Based on its own criteria, the server identifies deviations in the user's form and determines which parts need improvement and to what extent.

[0420] Step 5:

[0421] The server generates improvement plans based on the analysis results. In this process, it generates specific guidance that directly contributes to improving user performance, including, for example, specific methods for adjusting its operation.

[0422] Step 6:

[0423] The server sends the generated improvement suggestions to the user's device. This data includes information necessary for visual feedback.

[0424] Step 7:

[0425] Based on the received data, the device displays improvement suggestions within the user's field of view. Arrows and text appearing within the field of view indicate specific points for correction, allowing the user to adjust their form in real time.

[0426] (Example 1)

[0427] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0428] This invention relates to a system for assisting users in effectively improving specific sports movements. Conventional systems have struggled to analyze user movements in real time and provide immediate, specific feedback based on that analysis. Furthermore, there have been challenges in providing clear visual representations that allow users to intuitively understand where their movements need correction. In addition, there has been a need for technical improvements to enable efficient and secure data communication and analysis.

[0429] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0430] In this invention, the server includes a communication device that wirelessly transmits acquired motion data, an information processing device that analyzes the motion using a generated AI model based on the received data, and a computing device that compares the analyzed motion data with ideal motion data and identifies areas for improvement in the motion. This makes it possible for users to analyze their sports movements in real time and be presented with specific and visual suggestions for necessary corrections.

[0431] An "information acquisition device" is a device that records a user's actions and collects those movements as data.

[0432] A "communication device" is a device that transmits acquired data to an external source using communication means such as wireless communication.

[0433] An "information processing device" is a device used to perform calculations and analyses based on received data.

[0434] A "generative AI model" is a model that uses machine learning to analyze behavioral data and understand its characteristics.

[0435] A "calculation unit" is a device that compares analyzed data with ideal data and derives improvements to the operation as numerical values ​​or evaluations.

[0436] A "generation device" is a device that generates specific correction instructions for the user based on the results obtained from the computing device.

[0437] A "display device" is a device that presents visual information to the user and delivers instructions for correcting actions.

[0438] This invention provides a system for users to improve their sports form in real time. This system includes integrated technology for accurately recording, analyzing, and providing feedback on the user's physical movements.

[0439] This system utilizes smart glasses and sensor devices as information acquisition devices. These devices are equipped with cameras and accelerometers to accurately record the user's movements. This allows for the collection of detailed data about the user's actions.

[0440] The collected data is transmitted to a cloud server via a communication device. This communication utilizes wireless technologies such as Wi-Fi and Bluetooth to ensure stable and high-speed data transfer.

[0441] The server uses an information processing device to analyze the received data. This analysis employs a method that compares motion data with ideal motion patterns using a generative AI model. The AI ​​model learns the details of the user's movements and automatically detects inappropriate actions.

[0442] Based on the analysis results, the computing unit quantifies areas for improvement in the user's operation and identifies specific correction points. Correction instructions are generated by the generation unit and produced in a format that is easy for the user to understand.

[0443] Finally, the generated instructions are communicated to the user through a display device. The smart glasses' display provides real-time feedback for correcting actions, visually indicating points that need correction using arrows and highlights.

[0444] As a concrete example, when practicing a tennis serve, this system allows the user to instantly correct insufficient weight transfer or shoulder rotation. The input to the generating AI model is a prompt such as, "Generate feedback on weight transfer and shoulder rotation to improve my tennis serve form."

[0445] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0446] Step 1:

[0447] The device records the user's movements. Specifically, the smart glasses' camera captures the user's movements as video, and the accelerometer collects data on body movements. The input is the user's actual movements, and the output is detailed movement data and video data. In this way, information about the entire movement is captured by the device.

[0448] Step 2:

[0449] The terminal sends the collected motion data to the cloud server. The input is the motion data and video data obtained in step 1, and the output is the data sent to the server. Wireless communication technology is used in this process, and the data is appropriately compressed and encrypted before transmission.

[0450] Step 3:

[0451] The server analyzes the received data. The information processing device uses a generative AI model to compare the motion data with ideal motion. The input is the motion data sent in step 2, and the output is the analysis result. Through the analysis, it is possible to obtain numerical values ​​and evaluations of how close the user's motion is to the ideal and any specific differences.

[0452] Step 4:

[0453] The server generates suggestions for improving the operation based on the analysis results. The input is the analysis results from step 3, and the output is specific improvement instructions for the user. The generator uses the prompt sentences obtained from the generated AI model to create instructions such as "You should pull your shoulders back more."

[0454] Step 5:

[0455] The device visually displays the improvement instructions received from the server. The input is the improvement instructions from step 4, and the output is the correction points displayed on the smart glasses' screen. The smart glasses use arrows and highlights to show the user where the behavior needs to be corrected, supporting real-time improvements.

[0456] (Application Example 1)

[0457] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0458] Automated machinery in manufacturing, particularly factory robots, sometimes exhibits inefficient or inaccurate movements. This can lead to decreased production efficiency and variations in product quality. Conventional systems often lack sufficient real-time analysis and improvement instructions, and visual feedback is limited, making rapid correction difficult. Therefore, there is a need for a system that can record, analyze, and efficiently correct factory robot movements in real time.

[0459] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0460] In this invention, the server includes means for using a detection device to record user movements or machine movements, means for using a computer to analyze the recorded movement data by comparing it with ideal movement information, and means for using a display device to present correction instructions to the user or machine based on the analysis results. This enables real-time analysis of factory robot movements, allowing for rapid improvement of operational efficiency and accuracy.

[0461] A "detection device" is a device used to sense and record the actions of a user or machine.

[0462] "Motion data" refers to information about the movements of the user or machine acquired by a detection device.

[0463] "Ideal motion information" refers to data about motion that serves as a standard for improving the efficiency and precision of motion.

[0464] A "computer" is a computer system used to analyze recorded motion data by comparing it with ideal motion information.

[0465] An "auxiliary device" is a device that instructs the user or machine to modify its operation based on the analyzed areas for improvement.

[0466] A "control device" is a device used to record and improve the operation of a machine in real time.

[0467] A "display device" is a device that visually displays instructions for correcting operations to users or operators.

[0468] "Three-dimensional representation" is a format that expresses recorded motion information in a three-dimensional form.

[0469] The system realizing this invention uses a device equipped with multiple sensors and cameras as a detection device to record the movements of the user and the machine. The movement data is transmitted to a server in real time. On the server, based on the received movement data, a generating AI model is used to perform comparative analysis with ideal movement information. Based on the analysis results, the computer identifies areas for improvement and generates specific correction instructions.

[0470] These correction instructions are presented visually through a terminal displayed in the user's field of view or through a display device for the operator. This could include, for example, displaying arrows or highlights on the user's smart glasses. Furthermore, when correcting the operation of a factory robot, this is done via a control device.

[0471] For example, if a robotic arm in a factory makes an angle error while precisely tightening a screw, the server generates an instruction such as "Rotate the arm 5 degrees to the right" and displays it on the display screen. This allows the operator to correct the error immediately.

[0472] An example of a prompt to be input to the generating AI model is, "Generate correction instructions to optimize the robot arm angle and improve the accuracy of screw tightening." Through this form, the present invention can efficiently and effectively improve the operation of the user and the machine, thereby increasing productivity.

[0473] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0474] Step 1:

[0475] The terminal uses detection devices to record user or machine movements in real time. During this process, cameras and multiple sensors are used to acquire video data and angle information of the movements. The input is the user or machine movement itself, and the output is video data and movement data.

[0476] Step 2:

[0477] The terminal wirelessly transmits recorded motion data to the server. This data contains detailed numerical information about the motion, which the server receives. The input consists of previously acquired video data and motion data, forming a dataset ready to be transferred to the server as output.

[0478] Step 3:

[0479] The server inputs the received behavioral data into the generating AI model and analyzes it while comparing it with ideal behavioral information. This process uses prompts to request optimization of the behavior. The input is the behavioral data received by the server, and through processing by the generating AI model, areas for improvement in the behavior are identified as output.

[0480] Step 4:

[0481] The server generates correction instructions for the user or operator based on the areas for improvement identified through analysis. Specifically, it creates instructions that clearly specify which parts need to be corrected and how. The input is the server's analysis results, and the output is the specific correction instructions.

[0482] Step 5:

[0483] Correction instructions generated by the server are visually presented to the user or operator via the terminal's display device. In this case, arrows and highlights are used on the smart glasses' display. The input is correction instructions, and the output is visual information displayed in the user's field of view or on the operator's screen.

[0484] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0485] This invention is a sports form improvement system that combines an emotion engine that recognizes the user's emotions. This configuration is shown below.

[0486] This system incorporates an emotion engine that recognizes the user's emotional state in real time, in addition to existing sports form analysis devices that record and analyze user movements. Devices such as smart glasses collect physiological data such as the user's facial expressions, voice, and heart rate through cameras and sensors.

[0487] The device first sends video and data related to sports movements to a cloud server, where AI is used to analyze the movements. Next, the device separately sends collected emotional data, and the emotion engine evaluates the user's current emotional state. This evaluation is performed using facial expression analysis, voice tone analysis, or physiological data. The emotion engine on the server compares the current emotional data with past emotional history to track changes in the user's emotions.

[0488] The server adjusts the analyzed behavior improvement suggestions based on emotional information obtained from the emotion engine. For example, it optimizes the content and timing of the behavior improvement suggestions so that users receive them more positively. This process is designed to ensure that the feedback provided does not cause unnecessary stress to the user and that motivation is strengthened.

[0489] The device displays emotionally-adjusted feedback within the user's field of vision. This feedback changes to simpler instructions when the user is excited, while providing detailed suggestions for correcting their actions when they are relaxed, displaying information tailored to their individual emotional state.

[0490] For example, if a user is feeling stressed during tennis practice, the device's emotion engine will recognize this state. The server will then prioritize presenting only small, easily actionable improvement suggestions to address that emotion. This allows the user to effectively work on improving their form without being overwhelmed by excessive information. In this way, sports form analysis that takes user emotions into account contributes to improved performance.

[0491] The following describes the processing flow.

[0492] Step 1:

[0493] The user puts on the device and begins their sports activity. The device uses a camera and sensors to simultaneously record the user's movements, facial expressions, and voice.

[0494] Step 2:

[0495] The device transmits recorded motion and emotional data to the server. This data is necessary to simultaneously analyze the user's sports form and emotional state.

[0496] Step 3:

[0497] The server analyzes the received motion data using an AI algorithm and converts the user's current form into a 3D model. This allows for a three-dimensional reproduction of the motion details.

[0498] Step 4:

[0499] The server analyzes the user's emotional data using an emotion engine. Through facial expression analysis, voice tone analysis, and evaluation of physiological data, it identifies the user's current emotional state.

[0500] Step 5:

[0501] The server integrates the results of the motion analysis with the emotional state generated by the emotion engine to generate motion improvement suggestions optimized for the user's state. This includes the timing and content of appropriate feedback.

[0502] Step 6:

[0503] The server sends the generated improvement suggestions to the user's device. The feedback is adjusted to reflect the user's current emotional state.

[0504] Step 7:

[0505] The device displays received feedback in real time within the user's field of view. The explanation of improvement suggestions changes in conciseness or detail depending on the user's emotions. The user then adjusts their actions and improves the form based on this feedback.

[0506] (Example 2)

[0507] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0508] In improving athletic performance in sports and other activities, conventional systems have only analyzed the user's exercise data and suggested areas for improvement. However, when a user's emotional state influences their athletic performance and acceptance of feedback, mere data analysis alone is insufficient to achieve adequate improvement. Therefore, it is necessary to integrate and analyze both exercise and emotional data to enable users to work on improvement more effectively.

[0509] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0510] In this invention, the server includes means for using a detector to record the user's movements, means for using an information processing device to receive the recorded movement data and compare it with ideal movement information, means for using an emotion analysis device to evaluate the user's emotional information, and means for using an auxiliary device to adjust and present the analyzed areas for improvement based on the emotional information. This makes it possible to provide optimal feedback that takes into account the user's movements and emotional state.

[0511] A "user" is the individual who performs the movement, and is the subject that is recorded and analyzed by the system.

[0512] "Exercise" refers to physical movements performed by the user and the series of activities associated with them.

[0513] A "detector" is a device used to record a user's movements, and includes cameras, sensors, and other similar components.

[0514] "Motion data" refers to information about the user's physical movements recorded by the detector.

[0515] An "information processing device" is a device that receives motion data and has a computational function to analyze areas for improvement by comparing it with ideal motion information.

[0516] "Ideal exercise information" refers to a standard that indicates desirable exercise performance and is used for comparison with exercise data.

[0517] "Emotional information" refers to data that indicates the user's psychological or physiological state and is evaluated by an emotion analysis device.

[0518] An "emotion analysis device" is a device used to evaluate a user's emotional information, performing tasks such as facial expression analysis and voice tone analysis.

[0519] An "auxiliary device" is a device that adjusts the analyzed areas for improvement based on emotional information and presents them to the user.

[0520] This invention is a system that supports users in improving their exercise performance by analyzing exercise data and emotional information to provide feedback. The aim of this system is to simultaneously grasp the exercise data and emotional state of the user when they engage in sports or other exercise, and to suggest effective improvement plans.

[0521] First, the device utilizes detectors such as cameras and sensors to collect data from users exercising. This device not only records the user's movements as video but also acquires physiological data such as audio and heart rate. This information records the user's real-time exercise status and is used as foundational data for subsequent analysis.

[0522] Next, the data obtained from this device is sent to a server. The server analyzes this data using AI technology. Specifically, it uses a generative AI model to compare the recorded exercise data with ideal movement information and extract areas for improvement in the exercise. In addition, the server evaluates emotional information from the user's facial expressions, voice tone, and physiological data through an emotion analysis device. This allows the server to understand the emotions the user is experiencing during exercise, such as stress and relaxation.

[0523] The server integrates the analysis results of this movement with emotional information to generate optimal feedback for the user. The assistive device adjusts this feedback in a way that is easily accepted by the user and presents it as a final improvement plan. This improvement plan is adjusted in various ways depending on the user's emotional state, ranging from complex technical explanations to concise advice.

[0524] For example, if a user is feeling stressed during tennis practice, the device might offer a short instruction such as, "Relax your shoulders and take a deep breath," guiding the user towards effective improvement without overwhelming them with excessive information.

[0525] An example of a prompt would be, "Provide the optimal exercise improvement plan for a user experiencing stress during tennis practice." This prompt is used as input for analysis by a generative AI model.

[0526] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0527] Step 1:

[0528] The device records the user's exercise. Specifically, it records video with a camera, collects audio with a microphone, and collects physiological data with a heart rate sensor. The data collected in this process reflects the user's real-time exercise status and emotional state. The input is the user's exercise and physiological responses, and the output is the recorded data of these.

[0529] Step 2:

[0530] The device transmits the collected movement and emotion data to the server. This communication takes place via a high-speed internet connection. The input is the recorded data generated in step 1, and the output is the completion of data transmission to the server.

[0531] Step 3:

[0532] The server analyzes the received motion data. Using a generative AI model, it identifies areas for improvement by comparing the data with ideal motion information. In this analysis, swing speed, angle, trajectory, etc., are compared to a baseline. The input is the motion data transmitted from the terminal, and the output is the analysis results, including the areas for improvement.

[0533] Step 4:

[0534] The server evaluates emotional information using an emotion analysis device. Based on facial expression analysis, voice tone analysis, and physiological data, it classifies the user's emotions with labels such as "stress" or "relaxed." The input is emotion-related data transmitted from the terminal, and the output is the evaluated emotional information.

[0535] Step 5:

[0536] The server integrates movement analysis results and emotional information. It optimizes the analyzed improvement suggestions based on the emotional information and generates feedback that is easily accepted by the user. Depending on the user's state, it adjusts the content to provide detailed explanations or concise advice. The input is the movement analysis results and emotional information, and the output is the adjusted feedback.

[0537] Step 6:

[0538] The terminal presents feedback sent from the server to the user. For example, it may display simple instructions on the screen or detailed explanations in text. The input is feedback information sent from the server, and the output is a visual presentation to the user.

[0539] (Application Example 2)

[0540] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0541] The objective of this invention is to improve the inefficiency caused by failing to consider the user's emotional state when improving user movements. Specifically, it aims to improve efficient exercise form while strengthening motivation by adjusting suggested movements according to whether the user is stressed or relaxed, and by providing optimal feedback to the user.

[0542] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0543] In this invention, the server includes means for collecting data using sensors that record the user's movements; means for using a computer that compares the recorded movement data with ideal movement data and analyzes areas for improvement in the movements; means for using an emotion engine that adjusts the analyzed areas for improvement based on the user's emotional state and generates feedback; and means for using a support device that presents the user with feedback corresponding to the analyzed areas for improvement and emotional state to correct the movements. This makes it possible to provide the user with movement correction feedback that takes emotions into account.

[0544] A "user" refers to an individual who uses the system to analyze and improve its operation.

[0545] "Motion data" refers to digital data that records information about a user's movements, posture, and actions.

[0546] A "sensor" refers to a device that acquires information from the physical environment and converts it into digital data.

[0547] A "computer" refers to an electronic device used to process and analyze digital data.

[0548] "Ideal behavioral data" refers to model data that serves as a standard for evaluating user behavior.

[0549] "Areas for improvement" refers to parts of the operation that need improvement, identified by comparing the current operation data with ideal operation data.

[0550] An "emotion engine" refers to software that recognizes and analyzes a user's emotional state in real time.

[0551] "Feedback" refers to information and instructions provided to the user based on behavioral analysis and emotional state.

[0552] "Support device" refers to equipment used to provide users with information to improve their performance.

[0553] "Emotional state" refers to the user's current psychological or emotional state.

[0554] This invention is a system for integrating and processing user behavior and emotional state. The main components of this system are sensors for recording movement, a computer for processing the data, an emotion engine for analyzing emotions, and a support device for providing feedback.

[0555] The sensors use cameras and biometric devices to collect user movement data and physiological information such as facial expressions, voice, and heart rate. This allows for real-time acquisition of data on the user's movements and emotional state. The device sends this data to the cloud, where a computer compares the movement data to ideal movement data and analyzes areas for improvement.

[0556] The server analyzes user behavior on cloud computing resources and evaluates user emotions. The emotion engine integrates facial expression analysis, voice tone, and physiological information to understand emotional states and tracks changes from past history. Common platforms such as Amazon Rekognition and TensorFlow are used as emotion engines.

[0557] The support device provides the user with feedback tailored to their emotional state. The feedback is simplified and easy to follow when the user is stressed, and provides more detailed instructions when they are relaxed. For example, if the emotional engine determines that a user working in a factory is stressed, a simple instruction such as "We recommend taking a 15-minute break" is provided.

[0558] This system provides feedback to improve user behavior based on their emotional state, enabling a more efficient and comfortable work environment.

[0559] An example of a prompt would be: "To improve work efficiency in the factory, propose emotion-recognition-based feedback methods. In particular, describe specific approaches to stress reduction."

[0560] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0561] Step 1:

[0562] The device collects user motion and physiological data using sensors. It acquires video and heart rate data using cameras and heart rate monitors, recording them as digital data. Inputs include user motion, facial expressions, voice, and physiological data, which are captured by the sensors. Outputs are generated in real time and transmitted to the next processing step.

[0563] Step 2:

[0564] The device sends the acquired data to a cloud server. Using data transfer methods such as Bluetooth or Wi-Fi, the collected behavioral and physiological data are uploaded to the cloud service. The input is the data transmitted from the device, and the output is the data transferred to the cloud server. This data is used for analysis on the server.

[0565] Step 3:

[0566] The server analyzes the behavior using the received data. The computer compares the behavior data to an ideal behavior data model to identify areas for improvement. The input is behavior data uploaded to the cloud. The output is a list of identified behavioral improvements. An AI analysis engine can be utilized in this process.

[0567] Step 4:

[0568] The server uses an emotion engine to evaluate the user's emotions. It integrates received facial expressions, voice, and physiological data to analyze the user's emotional state. The input is physiological data transferred to the cloud. The output is the evaluation result of the user's emotional state. Common AI tools can be used for the emotion engine.

[0569] Step 5:

[0570] The server generates feedback that takes into account emotional states in relation to improvements made to the system's operation. Based on the analysis results, it creates optimal feedback that matches the user's emotions. The input consists of improvements made to the system and the emotional evaluation results. The output is emotionally adjusted feedback for the user. This information is formatted for delivery.

[0571] Step 6:

[0572] The terminal provides feedback to the user. Through the assistive device, feedback tailored to the user's current situation is displayed from the terminal. For example, smart glasses might display "Take a break and prepare to proceed further." The input is the feedback content generated by the server, and the output is the feedback displayed in the user's field of vision.

[0573] The specific processing unit 290 transmits the result of the specific processing to the headset terminal 314. In the headset terminal 314, the control unit 46A causes the speaker 240 and display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0574] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0575] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and specific processing may also be performed by the headset terminal 314.

[0576] [Fourth Embodiment]

[0577] Figure 7 shows an example of the configuration of the data processing system 410 according to the fourth embodiment.

[0578] As shown in Figure 7, the data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[0579] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0580] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a controlled object 443. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and controlled object 443 are also connected to the bus 52.

[0581] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0582] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0583] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0584] The controlled object 443 includes a display device, LEDs in the eyes, and motors that drive the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the robot 414's emotions can be expressed by controlling these motors. Furthermore, the robot 414's facial expressions can also be expressed by controlling the illumination state of the LEDs in its eyes.

[0585] Figure 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Figure 8, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0586] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0587] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0588] In robot 414, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0589] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0590] The present invention is a system for users to improve their sports form in real time, and a specific embodiment thereof is shown below.

[0591] This system uses sensors mounted on devices such as smart glasses to record the user's movements. The smart glasses' camera records the user's movements from multiple angles, while the G-sensor acquires detailed data such as subtle body movements and angles. This collects the basic data necessary to accurately generate a 3D model of the entire movement.

[0592] The collected data is transmitted wirelessly to a cloud server. The server applies an AI algorithm based on the received data to analyze the user's current form. This analysis can identify how the user's movements deviate from ideal motion data.

[0593] Based on the analysis results, the server generates specific suggestions for improving your form. For example, if the body rotation is insufficient in a golf swing, the server will create improvement instructions for the user such as "increase your shoulder rotation a little." The server then quickly transfers the generated feedback to the user's device.

[0594] The device, based on improvement suggestions sent from the server, presents instructions for correcting movements by overlaying them onto the real world within the user's field of view. The smart glasses' display uses, for example, arrows and highlights to clearly indicate the body parts that need correction. It also shows the direction and angle of movement in real time according to the target ideal form, helping the user adjust their movements on the spot.

[0595] For example, if a tennis player uses this system, they can receive feedback while practicing their serve form if their body weight distribution is incorrect. The user can see at a glance which parts of their body they should pay attention to through the smart glasses, allowing them to efficiently correct their movements.

[0596] In this way, this system supports users in the process of acquiring and improving specific sports skills and achieves performance improvement through its unique form analysis.

[0597] The following describes the processing flow.

[0598] Step 1:

[0599] The user puts on the device and begins their sports activity. The device records the user's movements through the camera and acquires detailed movement data using the G-sensor.

[0600] Step 2:

[0601] The terminal transmits real-time collected video data and sensor information to the server via wireless communication. If edge computing is recommended, the terminal can also perform preliminary data processing.

[0602] Step 3:

[0603] The server analyzes the received raw data. First, it uses an AI algorithm to convert the user's actions into a digitized 3D model. This allows the actual actions to be reproduced in three dimensions.

[0604] Step 4:

[0605] The server compares the generated 3D model with ideal motion data and evaluates each movement. Based on its own criteria, the server identifies deviations in the user's form and determines which parts need improvement and to what extent.

[0606] Step 5:

[0607] The server generates improvement plans based on the analysis results. In this process, it generates specific guidance that directly contributes to improving user performance, including, for example, specific methods for adjusting its operation.

[0608] Step 6:

[0609] The server sends the generated improvement suggestions to the user's device. This data includes information necessary for visual feedback.

[0610] Step 7:

[0611] Based on the received data, the device displays improvement suggestions within the user's field of view. Arrows and text appearing within the field of view indicate specific points for correction, allowing the user to adjust their form in real time.

[0612] (Example 1)

[0613] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0614] This invention relates to a system for assisting users in effectively improving specific sports movements. Conventional systems have struggled to analyze user movements in real time and provide immediate, specific feedback based on that analysis. Furthermore, there have been challenges in providing clear visual representations that allow users to intuitively understand where their movements need correction. In addition, there has been a need for technical improvements to enable efficient and secure data communication and analysis.

[0615] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0616] In this invention, the server includes a communication device that wirelessly transmits acquired motion data, an information processing device that analyzes the motion using a generated AI model based on the received data, and a computing device that compares the analyzed motion data with ideal motion data and identifies areas for improvement in the motion. This makes it possible for users to analyze their sports movements in real time and be presented with specific and visual suggestions for necessary corrections.

[0617] An "information acquisition device" is a device that records a user's actions and collects those movements as data.

[0618] A "communication device" is a device that transmits acquired data to an external source using communication means such as wireless communication.

[0619] An "information processing device" is a device used to perform calculations and analyses based on received data.

[0620] A "generative AI model" is a model that uses machine learning to analyze behavioral data and understand its characteristics.

[0621] A "calculation unit" is a device that compares analyzed data with ideal data and derives improvements to the operation as numerical values ​​or evaluations.

[0622] A "generation device" is a device that generates specific correction instructions for the user based on the results obtained from the computing device.

[0623] A "display device" is a device that presents visual information to the user and delivers instructions for correcting actions.

[0624] This invention provides a system for users to improve their sports form in real time. This system includes integrated technology for accurately recording, analyzing, and providing feedback on the user's physical movements.

[0625] This system utilizes smart glasses and sensor devices as information acquisition devices. These devices are equipped with cameras and accelerometers to accurately record the user's movements. This allows for the collection of detailed data about the user's actions.

[0626] The collected data is transmitted to a cloud server via a communication device. This communication utilizes wireless technologies such as Wi-Fi and Bluetooth to ensure stable and high-speed data transfer.

[0627] The server uses an information processing device to analyze the received data. This analysis employs a method that compares motion data with ideal motion patterns using a generative AI model. The AI ​​model learns the details of the user's movements and automatically detects inappropriate actions.

[0628] Based on the analysis results, the computing unit quantifies areas for improvement in the user's operation and identifies specific correction points. Correction instructions are generated by the generation unit and produced in a format that is easy for the user to understand.

[0629] Finally, the generated instructions are communicated to the user through a display device. The smart glasses' display provides real-time feedback for correcting actions, visually indicating points that need correction using arrows and highlights.

[0630] As a concrete example, when practicing a tennis serve, this system allows the user to instantly correct insufficient weight transfer or shoulder rotation. The input to the generating AI model is a prompt such as, "Generate feedback on weight transfer and shoulder rotation to improve my tennis serve form."

[0631] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0632] Step 1:

[0633] The device records the user's movements. Specifically, the smart glasses' camera captures the user's movements as video, and the accelerometer collects data on body movements. The input is the user's actual movements, and the output is detailed movement data and video data. In this way, information about the entire movement is captured by the device.

[0634] Step 2:

[0635] The terminal sends the collected motion data to the cloud server. The input is the motion data and video data obtained in step 1, and the output is the data sent to the server. Wireless communication technology is used in this process, and the data is appropriately compressed and encrypted before transmission.

[0636] Step 3:

[0637] The server analyzes the received data. The information processing device uses a generative AI model to compare the motion data with ideal motion. The input is the motion data sent in step 2, and the output is the analysis result. Through the analysis, it is possible to obtain numerical values ​​and evaluations of how close the user's motion is to the ideal and any specific differences.

[0638] Step 4:

[0639] The server generates suggestions for improving the operation based on the analysis results. The input is the analysis results from step 3, and the output is specific improvement instructions for the user. The generator uses the prompt sentences obtained from the generated AI model to create instructions such as "You should pull your shoulders back more."

[0640] Step 5:

[0641] The device visually displays the improvement instructions received from the server. The input is the improvement instructions from step 4, and the output is the correction points displayed on the smart glasses' screen. The smart glasses use arrows and highlights to show the user where the behavior needs to be corrected, supporting real-time improvements.

[0642] (Application Example 1)

[0643] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0644] Automated machinery in manufacturing, particularly factory robots, sometimes exhibits inefficient or inaccurate movements. This can lead to decreased production efficiency and variations in product quality. Conventional systems often lack sufficient real-time analysis and improvement instructions, and visual feedback is limited, making rapid correction difficult. Therefore, there is a need for a system that can record, analyze, and efficiently correct factory robot movements in real time.

[0645] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0646] In this invention, the server includes means for using a detection device to record user movements or machine movements, means for using a computer to analyze the recorded movement data by comparing it with ideal movement information, and means for using a display device to present correction instructions to the user or machine based on the analysis results. This enables real-time analysis of factory robot movements, allowing for rapid improvement of operational efficiency and accuracy.

[0647] A "detection device" is a device used to sense and record the actions of a user or machine.

[0648] "Motion data" refers to information about the movements of the user or machine acquired by a detection device.

[0649] "Ideal motion information" refers to data about motion that serves as a standard for improving the efficiency and precision of motion.

[0650] A "computer" is a computer system used to analyze recorded motion data by comparing it with ideal motion information.

[0651] An "auxiliary device" is a device that instructs the user or machine to modify its operation based on the analyzed areas for improvement.

[0652] A "control device" is a device used to record and improve the operation of a machine in real time.

[0653] A "display device" is a device that visually displays instructions for correcting operations to users or operators.

[0654] "Three-dimensional representation" is a format that expresses recorded motion information in a three-dimensional form.

[0655] The system realizing this invention uses a device equipped with multiple sensors and cameras as a detection device to record the movements of the user and the machine. The movement data is transmitted to a server in real time. On the server, based on the received movement data, a generating AI model is used to perform comparative analysis with ideal movement information. Based on the analysis results, the computer identifies areas for improvement and generates specific correction instructions.

[0656] These correction instructions are presented visually through a terminal displayed in the user's field of view or through a display device for the operator. This could include, for example, displaying arrows or highlights on the user's smart glasses. Furthermore, when correcting the operation of a factory robot, this is done via a control device.

[0657] For example, if a robotic arm in a factory makes an angle error while precisely tightening a screw, the server generates an instruction such as "Rotate the arm 5 degrees to the right" and displays it on the display screen. This allows the operator to correct the error immediately.

[0658] An example of a prompt to be input to the generating AI model is, "Generate correction instructions to optimize the robot arm angle and improve the accuracy of screw tightening." Through this form, the present invention can efficiently and effectively improve the operation of the user and the machine, thereby increasing productivity.

[0659] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0660] Step 1:

[0661] The terminal uses detection devices to record user or machine movements in real time. During this process, cameras and multiple sensors are used to acquire video data and angle information of the movements. The input is the user or machine movement itself, and the output is video data and movement data.

[0662] Step 2:

[0663] The terminal wirelessly transmits recorded motion data to the server. This data contains detailed numerical information about the motion, which the server receives. The input consists of previously acquired video data and motion data, forming a dataset ready to be transferred to the server as output.

[0664] Step 3:

[0665] The server inputs the received behavioral data into the generating AI model and analyzes it while comparing it with ideal behavioral information. This process uses prompts to request optimization of the behavior. The input is the behavioral data received by the server, and through processing by the generating AI model, areas for improvement in the behavior are identified as output.

[0666] Step 4:

[0667] The server generates correction instructions for the user or operator based on the areas for improvement identified through analysis. Specifically, it creates instructions that clearly specify which parts need to be corrected and how. The input is the server's analysis results, and the output is the specific correction instructions.

[0668] Step 5:

[0669] Correction instructions generated by the server are visually presented to the user or operator via the terminal's display device. In this case, arrows and highlights are used on the smart glasses' display. The input is correction instructions, and the output is visual information displayed in the user's field of view or on the operator's screen.

[0670] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0671] This invention is a sports form improvement system that combines an emotion engine that recognizes the user's emotions. This configuration is shown below.

[0672] This system incorporates an emotion engine that recognizes the user's emotional state in real time, in addition to existing sports form analysis devices that record and analyze user movements. Devices such as smart glasses collect physiological data such as the user's facial expressions, voice, and heart rate through cameras and sensors.

[0673] The device first sends video and data related to sports movements to a cloud server, where AI is used to analyze the movements. Next, the device separately sends collected emotional data, and the emotion engine evaluates the user's current emotional state. This evaluation is performed using facial expression analysis, voice tone analysis, or physiological data. The emotion engine on the server compares the current emotional data with past emotional history to track changes in the user's emotions.

[0674] The server adjusts the analyzed behavior improvement suggestions based on emotional information obtained from the emotion engine. For example, it optimizes the content and timing of the behavior improvement suggestions so that users receive them more positively. This process is designed to ensure that the feedback provided does not cause unnecessary stress to the user and that motivation is strengthened.

[0675] The device displays emotionally-adjusted feedback within the user's field of vision. This feedback changes to simpler instructions when the user is excited, while providing detailed suggestions for correcting their actions when they are relaxed, displaying information tailored to their individual emotional state.

[0676] For example, if a user is feeling stressed during tennis practice, the device's emotion engine will recognize this state. The server will then prioritize presenting only small, easily actionable improvement suggestions to address that emotion. This allows the user to effectively work on improving their form without being overwhelmed by excessive information. In this way, sports form analysis that takes user emotions into account contributes to improved performance.

[0677] The following describes the processing flow.

[0678] Step 1:

[0679] The user puts on the device and begins their sports activity. The device uses a camera and sensors to simultaneously record the user's movements, facial expressions, and voice.

[0680] Step 2:

[0681] The device transmits recorded motion and emotional data to the server. This data is necessary to simultaneously analyze the user's sports form and emotional state.

[0682] Step 3:

[0683] The server analyzes the received motion data using an AI algorithm and converts the user's current form into a 3D model. This allows for a three-dimensional reproduction of the motion details.

[0684] Step 4:

[0685] The server analyzes the user's emotional data using an emotion engine. Through facial expression analysis, voice tone analysis, and evaluation of physiological data, it identifies the user's current emotional state.

[0686] Step 5:

[0687] The server integrates the results of the motion analysis with the emotional state generated by the emotion engine to generate motion improvement suggestions optimized for the user's state. This includes the timing and content of appropriate feedback.

[0688] Step 6:

[0689] The server sends the generated improvement suggestions to the user's device. The feedback is adjusted to reflect the user's current emotional state.

[0690] Step 7:

[0691] The device displays received feedback in real time within the user's field of view. The explanation of improvement suggestions changes in conciseness or detail depending on the user's emotions. The user then adjusts their actions and improves the form based on this feedback.

[0692] (Example 2)

[0693] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0694] In improving athletic performance in sports and other activities, conventional systems have only analyzed the user's exercise data and suggested areas for improvement. However, when a user's emotional state influences their athletic performance and acceptance of feedback, mere data analysis alone is insufficient to achieve adequate improvement. Therefore, it is necessary to integrate and analyze both exercise and emotional data to enable users to work on improvement more effectively.

[0695] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0696] In this invention, the server includes means for using a detector to record the user's movements, means for using an information processing device to receive the recorded movement data and compare it with ideal movement information, means for using an emotion analysis device to evaluate the user's emotional information, and means for using an auxiliary device to adjust and present the analyzed areas for improvement based on the emotional information. This makes it possible to provide optimal feedback that takes into account the user's movements and emotional state.

[0697] A "user" is the individual who performs the movement, and is the subject that is recorded and analyzed by the system.

[0698] "Exercise" refers to physical movements performed by the user and the series of activities associated with them.

[0699] A "detector" is a device used to record a user's movements, and includes cameras, sensors, and other similar components.

[0700] "Motion data" refers to information about the user's physical movements recorded by the detector.

[0701] An "information processing device" is a device that receives motion data and has a computational function to analyze areas for improvement by comparing it with ideal motion information.

[0702] "Ideal exercise information" refers to a standard that indicates desirable exercise performance and is used for comparison with exercise data.

[0703] "Emotional information" refers to data that indicates the user's psychological or physiological state and is evaluated by an emotion analysis device.

[0704] An "emotion analysis device" is a device used to evaluate a user's emotional information, performing tasks such as facial expression analysis and voice tone analysis.

[0705] An "auxiliary device" is a device that adjusts the analyzed areas for improvement based on emotional information and presents them to the user.

[0706] This invention is a system that supports users in improving their exercise performance by analyzing exercise data and emotional information to provide feedback. The aim of this system is to simultaneously grasp the exercise data and emotional state of the user when they engage in sports or other exercise, and to suggest effective improvement plans.

[0707] First, the device utilizes detectors such as cameras and sensors to collect data from users exercising. This device not only records the user's movements as video but also acquires physiological data such as audio and heart rate. This information records the user's real-time exercise status and is used as foundational data for subsequent analysis.

[0708] Next, the data obtained from this device is sent to a server. The server analyzes this data using AI technology. Specifically, it uses a generative AI model to compare the recorded exercise data with ideal movement information and extract areas for improvement in the exercise. In addition, the server evaluates emotional information from the user's facial expressions, voice tone, and physiological data through an emotion analysis device. This allows the server to understand the emotions the user is experiencing during exercise, such as stress and relaxation.

[0709] The server integrates the analysis results of this movement with emotional information to generate optimal feedback for the user. The assistive device adjusts this feedback in a way that is easily accepted by the user and presents it as a final improvement plan. This improvement plan is adjusted in various ways depending on the user's emotional state, ranging from complex technical explanations to concise advice.

[0710] For example, if a user is feeling stressed during tennis practice, the device might offer a short instruction such as, "Relax your shoulders and take a deep breath," guiding the user towards effective improvement without overwhelming them with excessive information.

[0711] An example of a prompt would be, "Provide the optimal exercise improvement plan for a user experiencing stress during tennis practice." This prompt is used as input for analysis by a generative AI model.

[0712] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0713] Step 1:

[0714] The device records the user's exercise. Specifically, it records video with a camera, collects audio with a microphone, and collects physiological data with a heart rate sensor. The data collected in this process reflects the user's real-time exercise status and emotional state. The input is the user's exercise and physiological responses, and the output is the recorded data of these.

[0715] Step 2:

[0716] The device transmits the collected movement and emotion data to the server. This communication takes place via a high-speed internet connection. The input is the recorded data generated in step 1, and the output is the completion of data transmission to the server.

[0717] Step 3:

[0718] The server analyzes the received motion data. Using a generative AI model, it identifies areas for improvement by comparing the data with ideal motion information. In this analysis, swing speed, angle, trajectory, etc., are compared to a baseline. The input is the motion data transmitted from the terminal, and the output is the analysis results, including the areas for improvement.

[0719] Step 4:

[0720] The server evaluates emotional information using an emotion analysis device. Based on facial expression analysis, voice tone analysis, and physiological data, it classifies the user's emotions with labels such as "stress" or "relaxed." The input is emotion-related data transmitted from the terminal, and the output is the evaluated emotional information.

[0721] Step 5:

[0722] The server integrates movement analysis results and emotional information. It optimizes the analyzed improvement suggestions based on the emotional information and generates feedback that is easily accepted by the user. Depending on the user's state, it adjusts the content to provide detailed explanations or concise advice. The input is the movement analysis results and emotional information, and the output is the adjusted feedback.

[0723] Step 6:

[0724] The terminal presents feedback sent from the server to the user. For example, it may display simple instructions on the screen or detailed explanations in text. The input is feedback information sent from the server, and the output is a visual presentation to the user.

[0725] (Application Example 2)

[0726] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0727] The objective of this invention is to improve the inefficiency caused by failing to consider the user's emotional state when improving user movements. Specifically, it aims to improve efficient exercise form while strengthening motivation by adjusting suggested movements according to whether the user is stressed or relaxed, and by providing optimal feedback to the user.

[0728] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0729] In this invention, the server includes means for collecting data using sensors that record the user's movements; means for using a computer that compares the recorded movement data with ideal movement data and analyzes areas for improvement in the movements; means for using an emotion engine that adjusts the analyzed areas for improvement based on the user's emotional state and generates feedback; and means for using a support device that presents the user with feedback corresponding to the analyzed areas for improvement and emotional state to correct the movements. This makes it possible to provide the user with movement correction feedback that takes emotions into account.

[0730] A "user" refers to an individual who uses the system to analyze and improve its operation.

[0731] "Motion data" refers to digital data that records information about a user's movements, posture, and actions.

[0732] A "sensor" refers to a device that acquires information from the physical environment and converts it into digital data.

[0733] A "computer" refers to an electronic device used to process and analyze digital data.

[0734] "Ideal behavioral data" refers to model data that serves as a standard for evaluating user behavior.

[0735] "Areas for improvement" refers to parts of the operation that need improvement, identified by comparing the current operation data with ideal operation data.

[0736] An "emotion engine" refers to software that recognizes and analyzes a user's emotional state in real time.

[0737] "Feedback" refers to information and instructions provided to the user based on behavioral analysis and emotional state.

[0738] "Support device" refers to equipment used to provide users with information to improve their performance.

[0739] "Emotional state" refers to the user's current psychological or emotional state.

[0740] This invention is a system for integrating and processing user behavior and emotional state. The main components of this system are sensors for recording movement, a computer for processing the data, an emotion engine for analyzing emotions, and a support device for providing feedback.

[0741] The sensors use cameras and biometric devices to collect user movement data and physiological information such as facial expressions, voice, and heart rate. This allows for real-time acquisition of data on the user's movements and emotional state. The device sends this data to the cloud, where a computer compares the movement data to ideal movement data and analyzes areas for improvement.

[0742] The server analyzes user behavior on cloud computing resources and evaluates user emotions. The emotion engine integrates facial expression analysis, voice tone, and physiological information to understand emotional states and tracks changes from past history. Common platforms such as Amazon Rekognition and TensorFlow are used as emotion engines.

[0743] The support device provides the user with feedback tailored to their emotional state. The feedback is simplified and easy to follow when the user is stressed, and provides more detailed instructions when they are relaxed. For example, if the emotional engine determines that a user working in a factory is stressed, a simple instruction such as "We recommend taking a 15-minute break" is provided.

[0744] This system provides feedback to improve user behavior based on their emotional state, enabling a more efficient and comfortable work environment.

[0745] An example of a prompt would be: "To improve work efficiency in the factory, propose emotion-recognition-based feedback methods. In particular, describe specific approaches to stress reduction."

[0746] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0747] Step 1:

[0748] The device collects user motion and physiological data using sensors. It acquires video and heart rate data using cameras and heart rate monitors, recording them as digital data. Inputs include user motion, facial expressions, voice, and physiological data, which are captured by the sensors. Outputs are generated in real time and transmitted to the next processing step.

[0749] Step 2:

[0750] The device sends the acquired data to a cloud server. Using data transfer methods such as Bluetooth or Wi-Fi, the collected behavioral and physiological data are uploaded to the cloud service. The input is the data transmitted from the device, and the output is the data transferred to the cloud server. This data is used for analysis on the server.

[0751] Step 3:

[0752] The server analyzes the behavior using the received data. The computer compares the behavior data to an ideal behavior data model to identify areas for improvement. The input is behavior data uploaded to the cloud. The output is a list of identified behavioral improvements. An AI analysis engine can be utilized in this process.

[0753] Step 4:

[0754] The server uses an emotion engine to evaluate the user's emotions. It integrates received facial expressions, voice, and physiological data to analyze the user's emotional state. The input is physiological data transferred to the cloud. The output is the evaluation result of the user's emotional state. Common AI tools can be used for the emotion engine.

[0755] Step 5:

[0756] The server generates feedback that takes into account emotional states in relation to improvements made to the system's operation. Based on the analysis results, it creates optimal feedback that matches the user's emotions. The input consists of improvements made to the system and the emotional evaluation results. The output is emotionally adjusted feedback for the user. This information is formatted for delivery.

[0757] Step 6:

[0758] The terminal provides feedback to the user. Through the assistive device, feedback tailored to the user's current situation is displayed from the terminal. For example, smart glasses might display "Take a break and prepare to proceed further." The input is the feedback content generated by the server, and the output is the feedback displayed in the user's field of vision.

[0759] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the controlled object 443 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0760] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0761] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the robot 414.

[0762] Furthermore, the emotion identification model 59, acting as an emotion engine, may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to a specific mapping, which is an emotion map (see Figure 9). Similarly, the emotion identification model 59 may also determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[0763] Figure 9 shows an emotion map 400 in which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. The closer to the center of the concentric circles, the more primitive the emotions are located. Further out of the concentric circles, emotions representing states and actions arising from mental states are located. Emotion is a concept that includes feelings and mental states. On the left side of the concentric circles, emotions that are generally generated from reactions occurring in the brain are located. On the right side of the concentric circles, emotions that are generally induced by situational judgment are located. Above and below the concentric circles, emotions that are generally generated from reactions occurring in the brain and induced by situational judgment are located. In addition, the emotion of "pleasure" is located on the upper side of the concentric circles, and the emotion of "displeasure" is located on the lower side. Thus, in the emotion map 400, multiple emotions are mapped based on the structure in which emotions arise, and emotions that are likely to occur simultaneously are mapped close together.

[0764] These emotions are distributed at the 3 o'clock position on the Emotion Map 400, and usually fluctuate between feelings of security and anxiety. In the right half of the Emotion Map 400, situational awareness takes precedence over internal feelings, resulting in a calm impression.

[0765] The inside of the Emotion Map 400 represents inner thoughts, while the outside represents actions. Therefore, the further you go from the outside of the Emotion Map 400, the more visible (expressed in actions) your emotions become.

[0766] Here, human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. Similarly, in robots, cars, motorcycles, etc., emotions can be created based on various balances, such as posture and battery level. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. The emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on a system for analyzing brain physiological signals of speech emotion recognition and emotion, Tokushima University, doctoral dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map contains emotions belonging to a region called "response," where sensation is dominant. The right half of the emotion map contains emotions belonging to a region called "situation," where situational awareness is dominant.

[0767] The emotion map defines two emotions that promote learning. One is the emotion around the middle of the negative "repentance" and "reflection" on the situation side. In other words, it is when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is the emotion around the positive "desire" on the reaction side. In other words, it is when the robot has positive feelings such as "I want more" or "I want to know more."

[0768] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​representing each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple training data sets, which are combinations of user input and emotion values ​​representing each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions located close together have similar values, as shown in the emotion map 900 in Figure 10. Figure 10 shows an example where multiple emotions such as "reassured," "calm," and "confident" have similar emotion values.

[0769] The above description primarily focuses on the functions of the data processing device 12 in relation to this disclosure. However, the system related to this disclosure is not necessarily implemented on a server. The system related to this disclosure may be implemented as a general information processing system. This disclosure may be implemented, for example, as a software program that runs on a personal computer or as an application that runs on a smartphone. The method related to this disclosure may be provided to users in SaaS (Software as a Service) format.

[0770] In the above embodiment, an example was given in which a specific process is performed by a single computer 22. However, the technology of this disclosure is not limited thereto, and a distributed processing of the specific process may be performed by multiple computers, including computer 22. For example, a data generation model 58 may be provided in an external device of the data processing device 12, and the external device may generate data according to the input data.

[0771] In the above embodiment, an example was given in which the specific processing program 56 is stored in the storage 32, but the technology of this disclosure is not limited thereto. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-temporary storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-temporary storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes specific processing according to the specific processing program 56.

[0772] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[0773] Furthermore, it is not necessary to store the entirety of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store the entirety of the specific processing program 56 in the storage 32; it is acceptable to store only a portion of the specific processing program 56.

[0774] The following types of processors can be used as hardware resources to perform specific processing. Examples of processors include a CPU, a general-purpose processor that functions as a hardware resource to perform specific processing by executing software, i.e., a program. Other examples of processors include dedicated electrical circuits, such as FPGAs (Field-Programmable Gate Arrays), PLDs (Programmable Logic Devices), or ASICs (Application Specific Integrated Circuits), which have circuit configurations specifically designed to perform specific processing. All of these processors have built-in or connected memory, and all of them perform specific processing by using memory.

[0775] The hardware resource that performs a specific process may consist of one of these various processors, or it may consist of a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Alternatively, the hardware resource that performs a specific process may consist of a single processor.

[0776] Examples of configurations using a single processor include, firstly, a configuration in which one or more CPUs and software are combined to form a single processor, and this processor functions as a hardware resource that performs a specific process. Secondly, there is a configuration using a processor that realizes the functions of the entire system, including multiple hardware resources that perform a specific process, on a single IC chip, as exemplified by SoCs (System-on-a-chip). In this way, a specific process is realized using one or more of the above types of processors as hardware resources.

[0777] Furthermore, the hardware structure of these various processors can more specifically utilize electrical circuits that combine circuit elements such as semiconductor devices. Also, the specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps can be deleted, new steps added, or the processing order rearranged, as long as it does not deviate from the main purpose.

[0778] The descriptions and illustrations presented above are detailed explanations of the technical aspects of this disclosure and are merely examples of the technical aspects. For example, the above descriptions of the structure, function, operation, and effect are examples of the structure, function, operation, and effect of the technical aspects of this disclosure. Therefore, it goes without saying that you may delete unnecessary parts, add new elements, or replace elements in the descriptions and illustrations presented above, as long as you do not deviate from the essence of the technical aspects of this disclosure. Furthermore, in order to avoid confusion and facilitate understanding of the technical aspects of this disclosure, explanations of common technical knowledge and the like that do not require special explanation to enable the implementation of the technical aspects of this disclosure have been omitted from the descriptions and illustrations presented above.

[0779] All documents, patent applications, and technical standards described herein are incorporated by reference to the same extent as if each individual document, patent application, and technical standard were specifically and individually noted to be incorporated by reference.

[0780] The following is further disclosed regarding the embodiments described above.

[0781] (Claim 1)

[0782] A sensor that records the user's movements,

[0783] A computer that receives recorded motion data and analyzes areas for improvement in motion by comparing it with ideal motion data,

[0784] A support device that presents the analyzed areas for improvement to the user and modifies the operation,

[0785] A system that includes this.

[0786] (Claim 2)

[0787] The system according to claim 1, which converts video data and motion data recorded by a sensor into a three-dimensional model and performs the above-mentioned analysis.

[0788] (Claim 3)

[0789] The system according to claim 1, wherein the support device highlights and presents a specific part of the operation to the user using visual display means.

[0790] "Example 1"

[0791] (Claim 1)

[0792] An information acquisition device that records user actions,

[0793] A communication device that wirelessly transmits the acquired operational data,

[0794] An information processing device that analyzes behavior using an AI model generated based on received data,

[0795] A computing device that compares analyzed motion data with ideal motion data to identify areas for improvement in motion,

[0796] A generating device that generates instructions to modify user behavior based on improvements,

[0797] A display device that visually presents the generated instructions to the user,

[0798] A system that includes this.

[0799] (Claim 2)

[0800] The system according to claim 1, which converts motion data into a 3D model and performs detailed analysis using a generated AI model.

[0801] (Claim 3)

[0802] The system according to claim 1, wherein the display device visually highlights and presents to the user a specific part of the operation using arrows or highlights.

[0803] "Application Example 1"

[0804] (Claim 1)

[0805] A detection device that records user actions,

[0806] A computer that receives recorded motion data and analyzes areas for improvement in motion by comparing it with ideal motion information,

[0807] An auxiliary device that presents the analyzed areas for improvement to the user and corrects the operation,

[0808] A control device for recording and improving the operation of a machine,

[0809] A display device that analyzes user or machine operation data and provides correction instructions in real time,

[0810] A system that includes this.

[0811] (Claim 2)

[0812] The system according to claim 1, which converts video information and motion information recorded by a detection device into a three-dimensional representation and performs the above analysis.

[0813] (Claim 3)

[0814] The system according to claim 1, wherein the auxiliary device highlights and presents a specific part of the operation to the user or machine using visual display means.

[0815] "Example 2 of combining an emotion engine"

[0816] (Claim 1)

[0817] A detector that records the user's movements,

[0818] An information processing device that receives recorded exercise data and analyzes areas for improvement in exercise by comparing it with ideal exercise information,

[0819] A sentiment analysis device that evaluates user emotional information,

[0820] An assistive device that adjusts the analyzed areas for improvement based on emotional information, presents them to the user, and modifies movement.

[0821] A system that includes this.

[0822] (Claim 2)

[0823] The system according to claim 1, which converts recorded video data and motion data into a three-dimensional representation and performs the above-mentioned analysis.

[0824] (Claim 3)

[0825] The system according to claim 1, wherein the assistive device highlights and presents a specific part of the movement to the user using visual display means and provides feedback based on emotional information.

[0826] "Application example 2 when combining with an emotional engine"

[0827] (Claim 1)

[0828] A sensor that records the user's movements,

[0829] A computer that receives recorded motion data and analyzes areas for improvement in motion by comparing it with ideal motion data,

[0830] An emotion engine that adjusts the analyzed improvements based on the user's emotional state and generates feedback,

[0831] An assistance device that corrects operation by presenting the user with feedback tailored to the analyzed improvements and emotional state,

[0832] A system that includes this.

[0833] (Claim 2)

[0834] The system according to claim 1, which converts video data and motion data recorded by a sensor into a three-dimensional model and performs the above-mentioned analysis.

[0835] (Claim 3)

[0836] The system according to claim 1, wherein the support device highlights a specific part of an action to the user using visual display means and provides information corresponding to the user's emotional state. [Explanation of symbols]

[0837] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Devices 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robots< / url:> < / url:> < / url:> < / url:>

Claims

1. A sensor that records the user's movements, A computer that receives recorded motion data and analyzes areas for improvement in motion by comparing it with ideal motion data, A support device that presents the analyzed areas for improvement to the user and modifies the operation, A system that includes this.

2. The system according to claim 1, which converts video data and motion data recorded by a sensor into a three-dimensional model and performs the above-mentioned analysis.

3. The system according to claim 1, wherein the support device highlights and presents a specific part of the operation to the user using visual display means.

Images