system

Abstract

We provide the system. [Solution] To acquire the movement actions of individual users, the system includes image acquisition means and location information acquisition means. A motion analysis means analyzes the image data of motion acquired by the image acquisition means and extracts the user's motion characteristics, An environmental data acquisition method that obtains environmental information in real time, An advice generation means that integrates the data obtained from the motion analysis means and the environmental data acquisition means to generate optimal advice for the user, An audio output means that converts the advice generated by the advice generation means into audio and presents it to the user, A system that includes this.

Description

Technical Field

[0001] The technology of the present disclosure relates to a system.

Background Art

[0002] Patent Document 1 discloses a method for controlling a persona chatbot performed by at least one processor, including steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to an explanation of a chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] There is a problem that it is difficult for a user to obtain appropriate real-time advice according to individual motion characteristics during a golf round. In conventional private lessons, it is difficult to instantaneously provide detailed feedback considering the motion characteristics of each user, and it is also impossible to provide advice reflecting the actual round environment, so there is a problem that the improvement of the user's score does not progress efficiently.

[0005] ​This invention utilizes an image acquisition device and a location information acquisition device to acquire the movement of individual users and analyze their movement characteristics using a motion analysis device employing computer vision technology. Furthermore, it collects weather and wind direction information in real time using an environmental data acquisition device, and generates personalized advice by an advice generation device that comprehensively analyzes this data. The generated advice is then converted into audio using an audio output device and presented to the user in real time, enabling movement improvement tailored to the actual round environment.

[0006] "Image acquisition means" refers to a device or function that captures the user's movement and is a means for recording those movements.

[0007] "Location information acquisition means" refers to means of acquiring the user's location information and characteristics of the environment in which they are moving, using GPS or other sensor technologies.

[0008] "Motion analysis means" refers to a technology that analyzes the user's movement characteristics from acquired image data and extracts specific movement patterns and areas for improvement.

[0009] "Environmental data acquisition means" refers to methods for collecting real-time information such as weather and wind direction in the environment in which the user is playing.

[0010] "Advice generation means" refers to a means of creating user-optimized feedback and advice based on information obtained from motion analysis means and environmental data acquisition means.

[0011] "Voice output means" refers to a device or function that converts generated advice into voice and presents it to the user in real time. [Brief explanation of the drawing]

[0012] [Figure 1] This is a conceptual diagram showing an example of the configuration of a data processing system according to the first embodiment. [Figure 2]This is a conceptual diagram showing an example of the essential functions of a data processing device and a smart device according to the first embodiment. [Figure 3] This is a conceptual diagram showing an example of the configuration of a data processing system according to the second embodiment. [Figure 4] This is a conceptual diagram showing an example of the main functions of a data processing device and smart glasses according to the second embodiment. [Figure 5] This is a conceptual diagram showing an example of the configuration of a data processing system according to the third embodiment. [Figure 6] This is a conceptual diagram showing an example of the main functions of a data processing device and a headset-type terminal according to the third embodiment. [Figure 7] This is a conceptual diagram showing an example of the configuration of a data processing system according to the fourth embodiment. [Figure 8] This is a conceptual diagram showing an example of the main functions of a data processing device and a robot according to the fourth embodiment. [Figure 9] This shows an emotion map where multiple emotions are mapped. [Figure 10] This shows an emotion map where multiple emotions are mapped. [Figure 11] This is a sequence diagram showing the processing flow of the data processing system in Example 1. [Figure 12] This is a sequence diagram showing the processing flow of the data processing system in Application Example 1. [Figure 13] This is a sequence diagram showing the processing flow of the data processing system in Example 2, which incorporates an emotion engine. [Figure 14] This is a sequence diagram showing the processing flow of the data processing system in Application Example 2, which combines an emotion engine. [Modes for carrying out the invention]

[0013] Hereinafter, an example of an embodiment of the system relating to the technology of this disclosure will be described with reference to the attached drawings.

[0014] First, the terms used in the following description will be explained.

[0015] In the following embodiments, a labeled processor (hereinafter simply referred to as "processor") may be a single arithmetic unit or a combination of multiple arithmetic units. Also, the processor may be a single type of arithmetic unit or a combination of multiple types of arithmetic units. Examples of arithmetic units include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), an APU (Accelerated Processing Unit), and the like.

[0016] In the following embodiments, a labeled RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a work memory by the processor.

[0017] In the following embodiments, a labeled storage is one or more non-volatile storage devices that store various programs and various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), or magnetic tapes, etc.

[0018] In the following embodiments, a labeled communication I / F (Interface) is an interface including a communication processor and an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), or Bluetooth (registered trademark), etc.

[0019] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." That is, "A and / or B" means that it may be A alone, or B alone, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" applies when expressing three or more things linked by "and / or."

[0020] [First Embodiment]

[0021] Figure 1 shows an example of the configuration of the data processing system 10 according to the first embodiment.

[0022] As shown in Figure 1, the data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0023] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0024] The smart device 14 comprises a computer 36, a reception device 38, an output device 40, a camera 42, and a communication interface 44. The computer 36 comprises a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The reception device 38, output device 40, and camera 42 are also connected to the bus 52.

[0025] The reception device 38 is equipped with a touch panel 38A and a microphone 38B, etc., and receives user input. The touch panel 38A receives user input by detecting contact with an object (e.g., a pen or finger). The microphone 38B receives user input by detecting the user's voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0026] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form perceptible to the user 20 (e.g., audio and / or text). The display 40A displays visible information such as text and images according to instructions from the processor 46. The speaker 40B outputs audio according to instructions from the processor 46. The camera 42 is a small digital camera equipped with an optical system such as a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0027] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various types of information between processor 46 and processor 28 via network 54.

[0028] Figure 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0029] As shown in Figure 2, in the data processing device 12, a specific processing is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" related to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0030] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0031] In the smart device 14, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The reception output program 60 is used in conjunction with a specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0032] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0033] This invention provides a system that allows users to receive real-time advice on a golf course based on their individual movement characteristics. This system is realized through a series of processes that acquire and analyze the user's movement and provide appropriate advice.

[0034] First, the user launches a smartphone application at a golf driving range or course and films their swing. At this time, the application records video data of the swing using the camera (image acquisition means) built into the smartphone or dedicated device, and simultaneously acquires location information using GPS.

[0035] Next, the terminal sends the acquired data to the server. Here, the server uses advanced computer vision technology to analyze the swing. Specifically, it analyzes the swing speed, angle, and posture to extract detailed user characteristics. The server also collects environmental data such as golf course weather information and wind direction via the internet.

[0036] The server then integrates this data and uses an AI model to generate optimal advice for the user. For example, if the user's swing is delayed compared to their normal swing, it will provide specific advice on how to improve their swing timing.

[0037] Finally, the device converts the advice received from the server into audio via an audio output device and provides real-time feedback through the user's earphones. In addition, the smartphone screen visually displays an image of the swing and the analysis results.

[0038] As a concrete example, suppose a user is playing on a windy day and their shots tend to veer to the right more than usual. In this case, the system takes wind direction data into consideration and provides voice advice such as, "Aim slightly to the left when you swing, and grip the club firmly." In this way, the present invention helps users benefit from immediately applicable improvement measures in their actual round environment.

[0039] The following describes the processing flow.

[0040] Step 1:

[0041] The user launches a smartphone app at a golf course or driving range and records a video with the camera at the moment of their swing. During this process, the app automatically records location information using the device's built-in GPS.

[0042] Step 2:

[0043] The device compresses the acquired swing video and location information and sends it to the server. This involves a data transfer process via an internet connection.

[0044] Step 3:

[0045] The server analyzes the received video data. This video analysis involves using computer vision algorithms to extract details such as swing speed, angle, and body movements.

[0046] Step 4:

[0047] The server retrieves real-time weather information and wind direction data for the golf course from external web services. This environmental data is then integrated with swing analysis data.

[0048] Step 5:

[0049] The server uses an AI model to calculate based on motion analysis data and environmental data, generating advice to help the user improve their swing. For example, it can create advice on club selection considering wind effects and the amount of force to use in a swing.

[0050] Step 6:

[0051] The server generates advice, converts it into text, and then generates and sends this information along with any necessary image videos to the terminal.

[0052] Step 7:

[0053] The advice information received by the device is converted into speech using a speech synthesis engine and output in real time through the user's earphones. In addition, detailed advice and illustrative images are displayed on the smartphone screen.

[0054] Step 8:

[0055] Based on the advice provided by the user, they adjust their swing for the next shot and attempt to improve their play.

[0056] (Example 1)

[0057] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0058] In sports, particularly golf, a challenge has been the difficulty for individual users to obtain real-time feedback on their own movement patterns. Traditional methods required significant time and effort to provide specific advice that took individual movement characteristics into account, and failed to adequately incorporate environmental factors. As a result, improvements were not immediately reflected, and it took time for users' performance to improve.

[0059] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0060] In this invention, the server includes a means for capturing images, a means for acquiring location information, a means for extracting features, a means for collecting situational data, a means for generating advice, a means for presenting voice, and a means for displaying information. This enables individual users to receive real-time feedback on their own movement and to immediately benefit from optimal improvement measures that take environmental factors into consideration.

[0061] A "shooting device" is a mechanism used to record a user's movement and is a device that has the function of acquiring image data.

[0062] A "location information acquisition means" is a mechanism for determining the user's current location, and is a device that acquires location data using technologies such as GPS.

[0063] A "feature extraction means" is a mechanism that analyzes acquired image information and quantifies the characteristics of the user's motor movements.

[0064] A "situational data collection means" is a mechanism for collecting environmental information in real time, and is a device that acquires external data such as weather information and wind direction information.

[0065] The "advice generation mechanism" is a system that generates optimal advice for the user using a generation AI model based on acquired exercise data and environmental data.

[0066] A "voice presentation means" is a mechanism for converting generated advice into voice and conveying it to the user.

[0067] A "display means" is a mechanism for visually displaying information about the user's movement and analysis results.

[0068] This invention is a system designed to help users receive real-time advice at the golf course. The user uses a smartphone application to film and analyze their movements. Specifically, the user films their swing using their smartphone camera, and location information is obtained using GPS technology. The data thus obtained is transmitted from the terminal to a server.

[0069] The server analyzes the captured video data using computer vision technology. This analysis utilizes libraries such as TENSORFLOW® to extract motion characteristics such as swing speed, angle, and posture through image processing. Meanwhile, real-time environmental information is collected via the internet, including weather data and wind direction data.

[0070] The server integrates the acquired data and uses a generative AI model to generate specific advice for the user. The generated advice is sent to the device and conveyed through the user's earphones using an audio presentation system. In addition, the analysis results are visually displayed on the smartphone screen, allowing the user to immediately check their swing performance.

[0071] A concrete example is a scenario where a strong wind causes the ball to veer to the right. In this case, the system provides advice that takes the wind direction into account, such as "aim slightly to the left and grip the club firmly." An example of this prompt would be, "Please provide optimal swing advice for a golf course with strong winds." This allows users to receive real-time improvement suggestions tailored to their current environment.

[0072] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0073] Step 1:

[0074] The user launches a smartphone app at the golf course and records a video of their swing. During this process, the smartphone's camera records the movement, while simultaneously acquiring location information using GPS. The input consists of camera footage and GPS location data. The output is a video file of the swing and location information stored in the device's internal storage.

[0075] Step 2:

[0076] The device transmits the acquired video data and location information to the server. The input consists of the video file and location data stored on the device. The output is the data transferred to the server via the internet.

[0077] Step 3:

[0078] The server analyzes the received video data using computer vision technology. Specifically, it uses TensorFlow to perform image processing and extract motion characteristics such as swing speed, angle, and posture from each frame of the video. The input is the video file sent to the server. The output is data in which the motion characteristics are quantified.

[0079] Step 4:

[0080] The server collects environmental data in real time. This includes obtaining weather information and wind direction data from external sources on the internet. Inputs include weather data APIs accessed via the internet. Outputs are environmental information data stored on the server.

[0081] Step 5:

[0082] The server integrates extracted movement characteristics data and environmental information, and uses a generative AI model to generate advice for the user. The AI ​​model receives the prompt "Generate optimal golf advice based on the user's movement characteristics." The output is improvement advice tailored to the user.

[0083] Step 6:

[0084] The server sends the generated advice to the terminal. The input is the generated advice data. The output is the advice data sent to the terminal via the internet.

[0085] Step 7:

[0086] The device converts received advice into audio and provides feedback through the user's earphones. It also visually displays the analysis results on the smartphone screen. The input is advice data sent from the server. The output consists of audio feedback and visually displayed analysis results.

[0087] (Application Example 1)

[0088] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0089] Traditional golf practice methods lacked real-time feedback, making it difficult for users to make immediate improvements. Furthermore, there were insufficient means to receive appropriate advice that took environmental factors such as wind direction into account. This hindered effective skill development.

[0090] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0091] In this invention, the server includes image acquisition means and location information acquisition means to acquire the motor movements of individual users. It also includes advice generation means that integrates data obtained from motion analysis means and environmental data acquisition means to generate optimal advice, and feedback means that evaluate in real time and provide individual feedback. This makes it possible for users to receive individually optimized advice in real time.

[0092] "Image acquisition means" refers to devices and technologies that record a user's movement as visual data and provide that information in an analyzable format.

[0093] "Location information acquisition means" refers to technology for acquiring geographical information about the location where a user is exercising.

[0094] "Motion analysis means" refers to a technology that analyzes acquired visual data to scientifically evaluate the characteristics and patterns of a user's movements.

[0095] "Environmental data acquisition methods" refer to technologies for collecting data on the external environment when users exercise, such as weather and wind direction.

[0096] "Advice generation means" refers to technology that creates specific advice for users aimed at improving their athletic performance based on analyzed data.

[0097] "Voice output means" refers to devices or technologies for communicating generated advice to the user in voice.

[0098] A "feedback mechanism" refers to a technology or device that evaluates a user's movements in real time and provides appropriate feedback based on the results.

[0099] This system is designed to allow golfers to evaluate their swings in real time and receive personalized, optimized feedback. Users first film their swing using a dedicated smart device (smartphone or dedicated camera). This utilizes a high-performance camera sensor (e.g., Sony IMX sensor).

[0100] The device transfers the captured video data to the server via Bluetooth or Wi-Fi. The server uses machine learning libraries such as TensorFlow to perform motion analysis based on computer vision technology. This analysis extracts detailed data such as swing speed, angle, and posture.

[0101] Furthermore, the server utilizes APIs such as OpenWeatherMap to acquire real-time weather information and wind direction data. This operational and environmental data is integrated, and optimal advice is generated using a generative AI model. This generative AI model uses user-customized prompts to form specific advice tailored to the situation and swing type.

[0102] Finally, the suggestions generated by the advice generation system are converted into speech using the Google® Text-to-Speech API and fed back to the user in real time through earphones. In addition, the analysis results are visually displayed on the smart device's screen, and slow-motion swing playback is also performed to further enhance understanding.

[0103] As a concrete example, consider a situation where you are playing golf on a windy day. The generative AI model provides practical advice such as, "Taking the wind into account, I recommend adding a little more ballast for your next shot." In this way, the system provides the user with useful feedback in real time.

[0104] An example of a prompt for the generating AI model is, "Consider the user's swing data, current weather information, and wind direction data to generate optimal golf swing advice." In response to this prompt, the system will produce specific and effective advice tailored to the situation.

[0105] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0106] Step 1:

[0107] The user films their swing using a smart device. A camera sensor is used to capture high-resolution video. The input is the user's swing motion, and the output is video data of this motion.

[0108] Step 2:

[0109] The device saves the acquired video data to local storage and transfers it to a remote server via Bluetooth or Wi-Fi. The input is video data, and the output is data transfer in a format accessible to the server.

[0110] Step 3:

[0111] The server analyzes the received video data. Using computer vision technology based on TensorFlow, it calculates swing speed, angle, and posture. The input is video data, and the output is detailed data on swing characteristics.

[0112] Step 4:

[0113] The server uses APIs such as OpenWeatherMap to obtain real-time weather information and wind direction data. The input is location information sent to the API, and the output is weather information and wind direction information.

[0114] Step 5:

[0115] The server integrates motion analysis results and environmental data. Using prompts, it generates advice tailored to the user's situation into the generated AI model. The input consists of swing characteristic data and environmental data, while the output is specific advice.

[0116] Step 6:

[0117] The server converts advice generated via the Google Text-to-Speech API into speech. The input is the text data of the generated advice, and the output is the speech data.

[0118] Step 7:

[0119] The terminal provides audio data from the server to the user in real time via earphones. The analysis results are also visually displayed on the smart device's screen. Input consists of audio data and analysis result data, while output consists of user-recognizable audio and visual information.

[0120] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0121] This invention provides a system for golfers to receive personalized, real-time advice during actual rounds, and is characterized by its consideration of the user's emotional state. It not only understands the user's motor characteristics and provides advice that takes environmental conditions into account, but also recognizes emotional elements such as stress and satisfaction during play, and enables feedback based on these.

[0122] First, the user films their golf swing using a dedicated smartphone application. The video includes the swing form and the flow of force. This video is temporarily stored on the device, while location information is also recorded simultaneously. Furthermore, audio and facial expression data are also captured and processed by an emotion engine.

[0123] The device sends this data to a server in the cloud. The server uses computer vision technology to analyze the video data and identify the user's movement characteristics. In addition to this movement analysis, it performs voice analysis and facial recognition to determine the user's emotional state. For example, if the tone of voice or facial expression differs from the usual pattern, the emotion engine detects this change.

[0124] Next, the server retrieves weather information and wind direction data from an external database in real time and analyzes this data along with the user's movement characteristics and emotional state. Based on this analysis, the AI ​​generates optimal advice for the user. For example, if the AI ​​determines that the user is feeling stressed, the advice may include suggestions to take deep breaths to relax.

[0125] Finally, the device transcribes the generated advice into text and provides it to the user in real time through a speech synthesis system. Simultaneously, the smartphone screen displays additional visual information, including details of the advice and information tailored to the user's emotional state.

[0126] For example, if a user is unable to perform at their usual level on a windy day, the system will provide feedback such as, "Try to swing through firmly without being defeated by the wind. Relax a little and concentrate on the next shot." In this way, users can efficiently improve their scores while also being able to manage their motivation and emotions.

[0127] The following describes the processing flow.

[0128] Step 1:

[0129] The user launches a dedicated smartphone app at a golf course or driving range and records a video of their swing. The device's camera is used to record the user's swing, while the device's microphone collects audio data and records location information.

[0130] Step 2:

[0131] The device compresses the acquired swing video, audio data, and location information and securely transmits it to the server. This data includes the user's swing characteristics and emotional indicators.

[0132] Step 3:

[0133] The server analyzes video data using computer vision algorithms to identify the user's swing characteristics, form, and force application. Furthermore, it analyzes audio data with an emotion engine to determine the user's emotional state from their tone and word choice.

[0134] Step 4:

[0135] The server retrieves weather and wind direction information in real time from an external database via the internet. This environmental data is then integrated and analyzed with swing characteristics and emotional state.

[0136] Step 5:

[0137] The server utilizes an AI model to generate optimal advice that takes into account swing analysis data, emotional state, and weather information. If the user is feeling stressed, it will also include advice on breathing techniques to calm down and points for concentration.

[0138] Step 6:

[0139] The device receives advice from the server and converts it into voice feedback using a speech synthesis engine. This feedback is presented in real time through the user's earphones, and the screen displays visual feedback and detailed information tailored to the user's emotional state.

[0140] Step 7:

[0141] Based on the audio advice and visual information provided, users can adjust their swing and compose themselves before taking the next shot. This allows for improved play quality and a better golfing experience.

[0142] (Example 2)

[0143] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0144] There is a need to provide appropriate and personalized advice to individual users by evaluating their performance and emotional state during exercise in real time, and by taking environmental conditions into consideration. However, conventional technologies have limited systems that can provide feedback that fully considers the user's emotional state and environmental conditions, resulting in a lack of means to efficiently improve performance. As a result, it has been difficult for users to improve their performance more effectively and engage in activities in a mentally stable state.

[0145] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0146] In this invention, the server includes a camera and location information device that acquires the actions of individual users, an analysis device that analyzes the video information of actions acquired by the camera and extracts the user's action characteristics and emotional state, and an environmental information device that acquires environmental information in real time. This makes it possible to provide users with optimal and personalized real-time advice.

[0147] A "recording device" refers to hardware or software used to record a user's actions in real time, and is responsible for acquiring image or video data.

[0148] A "location information device" is a technology used to determine a user's current location. It is a device that acquires location information in real time using GPS or other location information technologies.

[0149] "Video information" refers to image or video data acquired by a recording device, and serves as basic data for analyzing the user's behavioral characteristics using an analysis device.

[0150] "Analysis device" refers to software or hardware that analyzes the user's behavioral characteristics and emotional state in real time based on data obtained from a camera and a location information device.

[0151] "Motional characteristics" refer to the characteristics and performance tendencies of the user's movements and behaviors, and the analysis device evaluates the data based on these characteristics.

[0152] "Emotional state" refers to the user's mental health and emotional changes, and is a psychological state analyzed from voice and facial expression data.

[0153] An "environmental information device" is a device that acquires environmental information in real time, such as weather, temperature, and wind direction, regarding external conditions when users are engaged in activities.

[0154] A "support generation device" is a device that integrates data obtained from analysis devices and environmental information devices to generate optimal advice for the user.

[0155] A "voice presentation device" refers to a system that converts generated advice into voice and presents it to the user in an easy-to-understand manner, and includes speakers and speech synthesis software.

[0156] This system aims to evaluate users' motor performance and emotional state in real time and provide support based on that evaluation. The specific implementation method is described below.

[0157] First, the user uses a mobile device with a dedicated application installed to record their movements. This device has a built-in camera and location tracking device, and location information is acquired along with the recorded video data. The video records the user's movement form and force flow, and simultaneously captures audio and facial expression data.

[0158] Next, the device sends the acquired data to a server in the cloud. This communication uses a secure protocol to ensure security. The server analyzes the received video data using computer vision technology and utilizes software such as OpenCV and TensorFlow to identify the user's behavioral characteristics. It also analyzes audio and facial expression data to determine the user's emotional state.

[0159] The server calls external APIs to obtain real-time weather and wind direction information from external data sources. It integrates this data and uses a generative AI model to generate optimal advice for the user. This advice includes specific actions to improve athletic performance and suggestions to support emotional management. Accurate feedback is obtained by inputting the analysis results as prompts into the generative AI model.

[0160] Finally, the device uses speech synthesis technology to convert the generated advice into speech and provides it to the user in real time. The device's screen also displays detailed advice and visual information tailored to the user's emotional state. This allows the user to improve their performance and feel more at ease while working.

[0161] For example, the system could provide advice to a user on a windy day such as, "Try to swing through firmly without being defeated by the wind. Relax a little and concentrate on the next shot." Another example of a prompt to input to the generating AI model is, "Please provide advice to improve the user's golf performance, taking into account the stressful situation the user is experiencing."

[0162] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0163] Step 1:

[0164] Users record their actions using a dedicated smartphone app. Input consists of video, audio, and facial expression data obtained through the camera and microphone. Users utilize this data and save it to their device. Specifically, after activating the camera, users perform a series of actions such as a swing or a shot, and the video is recorded on the device.

[0165] Step 2:

[0166] The device acquires location information along with stored video data and sends the data to the server. Inputs include video, audio, and facial expression data stored on the device, as well as acquired location information. These are aggregated and transferred to the server using a secure protocol. Specifically, the device utilizes its GPS sensor and network communication capabilities to package and transmit the data.

[0167] Step 3:

[0168] The server uses computer vision technology to analyze the transmitted video data. The input is video data received from the terminal, and the output is the result of extracting the user's movement characteristics. Specifically, the server uses OpenCV and TensorFlow to analyze joint movements and swing trajectories, and converts the user's movement characteristics into data.

[0169] Step 4:

[0170] The server uses voice analysis and facial recognition technology to determine the user's emotional state. Input is voice and facial expression data, and output is the result of the assessment of the user's emotional state. Specifically, the server evaluates the tone and speed of the voice and changes in facial expression, and based on this, analyzes stress levels and emotional fluctuations.

[0171] Step 5:

[0172] The server calls an external API to obtain environmental data in real time. The input is the endpoint of the environmental data API, and the output is current weather and wind direction information. Specifically, the server generates an API request and retrieves the necessary data in real time.

[0173] Step 6:

[0174] The server uses integrated data (behavioral characteristics, emotional state, and environmental information) to activate a generative AI model and generate optimal advice for the user. The input is the aforementioned data set, and the output is the content of the advice for the user. Specifically, the server inputs this data as prompts into the model and generates real-time feedback for the user.

[0175] Step 7:

[0176] The terminal converts the output from the server into speech using speech synthesis technology and provides it to the user. The input is the text of advice sent from the server, and the output is the audio and visual information presented to the user. Specifically, the terminal generates speech from the text using a speech synthesis engine and displays the related visuals on the display.

[0177] (Application Example 2)

[0178] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as a "server" and the smart device 14 as a "terminal".

[0179] This invention aims to solve the problem of a lack of support for golfers to manage their technical performance and mental state simultaneously during a round and achieve optimal play. In particular, while conventional systems can provide more personalized advice by considering not only the user's motor characteristics but also their emotional state, such technology does not yet exist.

[0180] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0181] In this invention, the server includes image information acquisition means, audio information acquisition means and location information acquisition means for acquiring the motor movements and emotional states of individual users, motion and emotion analysis means for analyzing the image data and audio data of the motor movements acquired by the image information acquisition means and audio information acquisition means to extract the user's motion characteristics and emotional characteristics, and environmental information acquisition means for acquiring environmental information in real time. This makes it possible to provide users with real-time optimal advice that takes into account their motion characteristics and emotional states, thereby improving technical performance and promoting mental stability.

[0182] "Individual users" refers to specific human users, and those individuals are users of the system.

[0183] "Motor movement" refers to a series of movements or actions that a person performs when moving their body, and includes movements in specific tasks or sports.

[0184] "Emotional state" refers to the psychological or emotional condition a user is experiencing at a given moment, and includes stress, satisfaction, concentration, etc.

[0185] "Means for acquiring image information" refers to devices or technologies for capturing the movement of a user, and includes cameras and video recorders.

[0186] "Voice information acquisition means" refers to a device or technology for recording voices emitted by a user, and includes microphones and the like.

[0187] "Location information acquisition means" refers to a device or technology for determining the geographical location of a user, including location information services such as GPS.

[0188] "Motion and emotion analysis means" refers to a device or technology for analyzing acquired image data and audio data to identify the user's motor and emotional characteristics.

[0189] "Environmental information acquisition means" refers to devices or technologies for acquiring information about the external environment in real time, and includes sensors and information acquisition services via the Internet.

[0190] An "advice generation means" is a device or technology that generates optimal advice for the user based on analyzed data.

[0191] "Voice output means" refers to a device or technology for converting generated advice into voice and conveying it to the user, and includes speakers and speech synthesizers.

[0192] The system for implementing this invention analyzes the user's motor movements and emotional state in real time and provides personalized advice. First, the user acquires video and audio data of their swing using the camera and microphone built into their smartphone. This data also includes the user's location information.

[0193] Next, the device sends the acquired data to a server in the cloud. The server analyzes the video data using computer vision technology (e.g., OpenCV) to understand the user's movement characteristics. Simultaneously, it analyzes the audio data and facial expressions using emotion recognition technology (e.g., Affectiva) to determine the user's emotional state.

[0194] Furthermore, the server uses environmental information, including weather and wind direction data, obtained from external APIs (e.g., OpenWeatherMap), to integrate the analysis results. Based on these results, a generative AI model generates advice tailored to the user's technical performance and emotional state.

[0195] Finally, the device presents the generated advice to the user verbally using speech synthesis technology (e.g., Google Text-to-Speech). This process allows the user to receive both technical and emotional support in real time.

[0196] For example, if a user's shots are inconsistent in strong winds, the system will generate advice such as, "Concentrate to stabilize your swing against the wind." An example of a prompt would be, "Based on the user's swing data and emotional data, generate optimal golf advice for windy conditions."

[0197] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0198] Step 1:

[0199] The user uses their smartphone to film their golf swing and record audio. During this stage, the camera captures video data, the microphone collects audio, and location information is recorded simultaneously. The inputs are video data, audio data, and location information, which are temporarily stored on the device.

[0200] Step 2:

[0201] The device transmits acquired video data, audio data, and location information to a server in the cloud. This process uses a communication module to package the data and send it to the server securely. The input consists of various data stored on the device, and the output is the completion of data transmission to the server.

[0202] Step 3:

[0203] The server analyzes the video data. Computer vision technology (e.g., OpenCV) is used to extract the swing motion characteristics. At this stage, the input is video data, and the output is user motion characteristic information.

[0204] Step 4:

[0205] The server analyzes audio and facial expression data to determine the user's emotional state. Emotion recognition technology (e.g., Affectiva) identifies stress levels and emotions from voice tone and facial expressions. The input is audio and video data, and the output is information about the user's emotional state.

[0206] Step 5:

[0207] The server uses an external API to obtain real-time weather and wind direction information (e.g., OpenWeatherMap). This allows for the collection of up-to-date information about the play environment. The input is a request to obtain weather and wind direction data, and the output is a set of environmental information.

[0208] Step 6:

[0209] The server integrates operational characteristics, emotional states, and environmental information, and uses a generative AI model to generate optimal advice. This analysis quickly generates situation-appropriate advice for the user. The input is an integrated dataset, and the output is the generated advice.

[0210] Step 7:

[0211] The device outputs the generated advice as speech using speech synthesis technology (e.g., Google Text-to-Speech). Simultaneously, it also displays the advice to the user in text. The input is the generated advice, and the output is the audio and text presentation to the user.

[0212] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0213] Data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of data generation model 58 is ChatGPT (registered trademark) (Internet search).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0214] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart device 14.

[0215] [Second Embodiment]

[0216] Figure 3 shows an example of the configuration of the data processing system 210 according to the second embodiment.

[0217] As shown in Figure 3, the data processing system 210 includes a data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0218] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0219] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication interface 44. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, and camera 42 are also connected to the bus 52.

[0220] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0221] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0222] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0223] Figure 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Figure 4, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0224] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0225] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0226] In the smart glasses 214, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0227] Next, the identification processing performed by the identification processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0228] This invention provides a system that allows users to receive real-time advice on a golf course based on their individual movement characteristics. This system is realized through a series of processes that acquire and analyze the user's movement and provide appropriate advice.

[0229] First, the user launches a smartphone application at a golf driving range or course and films their swing. At this time, the application records video data of the swing using the camera (image acquisition means) built into the smartphone or dedicated device, and simultaneously acquires location information using GPS.

[0230] Next, the terminal sends the acquired data to the server. Here, the server uses advanced computer vision technology to analyze the swing. Specifically, it analyzes the swing speed, angle, and posture to extract detailed user characteristics. The server also collects environmental data such as golf course weather information and wind direction via the internet.

[0231] The server then integrates this data and uses an AI model to generate optimal advice for the user. For example, if the user's swing is delayed compared to their normal swing, it will provide specific advice on how to improve their swing timing.

[0232] Finally, the device converts the advice received from the server into audio via an audio output device and provides real-time feedback through the user's earphones. In addition, the smartphone screen visually displays an image of the swing and the analysis results.

[0233] As a concrete example, suppose a user is playing on a windy day and their shots tend to veer to the right more than usual. In this case, the system takes wind direction data into consideration and provides voice advice such as, "Aim slightly to the left when you swing, and grip the club firmly." In this way, the present invention helps users benefit from immediately applicable improvement measures in their actual round environment.

[0234] The following describes the processing flow.

[0235] Step 1:

[0236] The user launches a smartphone app at a golf course or driving range and records a video with the camera at the moment of their swing. During this process, the app automatically records location information using the device's built-in GPS.

[0237] Step 2:

[0238] The device compresses the acquired swing video and location information and sends it to the server. This involves a data transfer process via an internet connection.

[0239] Step 3:

[0240] The server analyzes the received video data. This video analysis involves using computer vision algorithms to extract details such as swing speed, angle, and body movements.

[0241] Step 4:

[0242] The server retrieves real-time weather information and wind direction data for the golf course from external web services. This environmental data is then integrated with swing analysis data.

[0243] Step 5:

[0244] The server uses an AI model to calculate based on motion analysis data and environmental data, generating advice to help the user improve their swing. For example, it can create advice on club selection considering wind effects and the amount of force to use in a swing.

[0245] Step 6:

[0246] The server generates advice, converts it into text, and then generates and sends this information along with any necessary image videos to the terminal.

[0247] Step 7:

[0248] The advice information received by the device is converted into speech using a speech synthesis engine and output in real time through the user's earphones. In addition, detailed advice and illustrative images are displayed on the smartphone screen.

[0249] Step 8:

[0250] Based on the advice provided by the user, they adjust their swing for the next shot and attempt to improve their play.

[0251] (Example 1)

[0252] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0253] In sports, particularly golf, a challenge has been the difficulty for individual users to obtain real-time feedback on their own movement patterns. Traditional methods required significant time and effort to provide specific advice that took individual movement characteristics into account, and failed to adequately incorporate environmental factors. As a result, improvements were not immediately reflected, and it took time for users' performance to improve.

[0254] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0255] In this invention, the server includes a means for capturing images, a means for acquiring location information, a means for extracting features, a means for collecting situational data, a means for generating advice, a means for presenting voice, and a means for displaying information. This enables individual users to receive real-time feedback on their own movement and to immediately benefit from optimal improvement measures that take environmental factors into consideration.

[0256] A "shooting device" is a mechanism used to record a user's movement and is a device that has the function of acquiring image data.

[0257] A "location information acquisition means" is a mechanism for determining the user's current location, and is a device that acquires location data using technologies such as GPS.

[0258] A "feature extraction means" is a mechanism that analyzes acquired image information and quantifies the characteristics of the user's motor movements.

[0259] A "situational data collection means" is a mechanism for collecting environmental information in real time, and is a device that acquires external data such as weather information and wind direction information.

[0260] The "advice generation mechanism" is a system that generates optimal advice for the user using a generation AI model based on acquired exercise data and environmental data.

[0261] A "voice presentation means" is a mechanism for converting generated advice into voice and conveying it to the user.

[0262] A "display means" is a mechanism for visually displaying information about the user's movement and analysis results.

[0263] This invention is a system designed to help users receive real-time advice at the golf course. The user uses a smartphone application to film and analyze their movements. Specifically, the user films their swing using their smartphone camera, and location information is obtained using GPS technology. The data thus obtained is transmitted from the terminal to a server.

[0264] The server analyzes the captured video data using computer vision technology. This analysis utilizes libraries such as TensorFlow to extract motion characteristics such as swing speed, angle, and posture through image processing. Meanwhile, real-time environmental information is collected via the internet, including weather data and wind direction data.

[0265] The server integrates the acquired data and uses a generative AI model to generate specific advice for the user. The generated advice is sent to the device and conveyed through the user's earphones using an audio presentation system. In addition, the analysis results are visually displayed on the smartphone screen, allowing the user to immediately check their swing performance.

[0266] A concrete example is a scenario where a strong wind causes the ball to veer to the right. In this case, the system provides advice that takes the wind direction into account, such as "aim slightly to the left and grip the club firmly." An example of this prompt would be, "Please provide optimal swing advice for a golf course with strong winds." This allows users to receive real-time improvement suggestions tailored to their current environment.

[0267] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0268] Step 1:

[0269] The user launches a smartphone app at the golf course and records a video of their swing. During this process, the smartphone's camera records the movement, while simultaneously acquiring location information using GPS. The input consists of camera footage and GPS location data. The output is a video file of the swing and location information stored in the device's internal storage.

[0270] Step 2:

[0271] The device transmits the acquired video data and location information to the server. The input consists of the video file and location data stored on the device. The output is the data transferred to the server via the internet.

[0272] Step 3:

[0273] The server analyzes the received video data using computer vision technology. Specifically, it uses TensorFlow to perform image processing and extract motion characteristics such as swing speed, angle, and posture from each frame of the video. The input is the video file sent to the server. The output is data in which the motion characteristics are quantified.

[0274] Step 4:

[0275] The server collects environmental data in real time. This includes obtaining weather information and wind direction data from external sources on the internet. Inputs include weather data APIs accessed via the internet. Outputs are environmental information data stored on the server.

[0276] Step 5:

[0277] The server integrates extracted movement characteristics data and environmental information, and uses a generative AI model to generate advice for the user. The AI ​​model receives the prompt "Generate optimal golf advice based on the user's movement characteristics." The output is improvement advice tailored to the user.

[0278] Step 6:

[0279] The server sends the generated advice to the terminal. The input is the generated advice data. The output is the advice data sent to the terminal via the internet.

[0280] Step 7:

[0281] The device converts received advice into audio and provides feedback through the user's earphones. It also visually displays the analysis results on the smartphone screen. The input is advice data sent from the server. The output consists of audio feedback and visually displayed analysis results.

[0282] (Application Example 1)

[0283] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0284] In conventional golf practice, due to the lack of real-time feedback, users had the problem that it was difficult to make immediate improvements. In addition, there were also insufficient means to obtain appropriate advice considering environmental factors such as wind direction. As a result, effective skill improvement was hindered.

[0285] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following respective means.

[0286] In this invention, the server includes an image acquisition means and a position information acquisition means for acquiring the movement actions of individual users. It includes an advice generation means for integrating the data obtained from the movement analysis means and the environmental data acquisition means and generating optimal advice, and a feedback means for evaluating in real time and providing individual feedback. As a result, users can obtain individually optimized advice in real time.

[0287] The "image acquisition means" is a device or technology for recording the movement actions of users as visual data and providing the information in an analyzable form.

[0288] The "position information acquisition means" is a technology for acquiring the geographical information of the location where the user is performing the movement.

[0289] The "movement analysis means" is a technology for analyzing the acquired visual data and scientifically evaluating the characteristics and patterns of the user's movement.

[0290] The "environmental data acquisition means" is a technology for collecting data on the external environment when the user is performing the movement, such as weather and wind direction.

[0291] The "advice generation means" is a technology for creating specific advice for the user to aim at improving movement performance based on the analyzed data.

[0292] "Voice output means" refers to devices or technologies for communicating generated advice to the user in voice.

[0293] A "feedback mechanism" refers to a technology or device that evaluates a user's movements in real time and provides appropriate feedback based on the results.

[0294] This system is designed to allow golfers to evaluate their swings in real time and receive personalized, optimized feedback. Users first film their swing using a dedicated smart device (smartphone or dedicated camera). This utilizes a high-performance camera sensor (e.g., Sony IMX sensor).

[0295] The device transfers the captured video data to the server via Bluetooth or Wi-Fi. The server uses machine learning libraries such as TensorFlow to perform motion analysis based on computer vision technology. This analysis extracts detailed data such as swing speed, angle, and posture.

[0296] Furthermore, the server utilizes APIs such as OpenWeatherMap to acquire real-time weather information and wind direction data. This operational and environmental data is integrated, and optimal advice is generated using a generative AI model. This generative AI model uses user-customized prompts to form specific advice tailored to the situation and swing type.

[0297] Finally, the suggestions generated by the advice generation system are converted into speech using the Google Text-to-Speech API and fed back to the user in real time through earphones. In addition, the analysis results are visually displayed on the smart device's screen, and slow-motion swing playback is also performed to further enhance understanding.

[0298] As a specific example, consider the situation of playing golf on a windy day. The generative AI model provides practical advice such as "Take into account the wind and recommend taking a little more ballast in your next shot." In this way, the system provides useful feedback to the user in real time.

[0299] As an example of a prompt sentence for the generative AI model, there is one such as "Please generate optimal golf swing advice considering the user's swing data, current weather information, and wind direction data." In response to this prompt sentence, the system generates specific and effective advice according to the situation.

[0300] The flow of the specific process in Application Example 1 will be described using FIG. 12.

[0301] Step 1:

[0302] The user uses a smart device to take a picture of their swing. At this time, a high-resolution video is acquired using a camera sensor. The input is the user's swing motion, and the output is the video data of this motion.

[0303] Step 2:

[0304] The terminal saves the acquired video data in local storage and transfers it to a remote server via Bluetooth or Wi-Fi. The input is the video data, and the output is the data transfer in a format accessible by the server.

[0305] Step 3:

[0306] The server analyzes the received video data. Using computer vision technology with TensorFlow, it calculates the speed, angle, posture, etc. of the swing. The input is the video data, and the output is detailed data on swing characteristics.

[0307] Step 4:

[0308] The server uses APIs such as OpenWeatherMap to obtain real-time weather information and wind direction data. The input is location information sent to the API, and the output is weather information and wind direction information.

[0309] Step 5:

[0310] The server integrates motion analysis results and environmental data. Using prompts, it generates advice tailored to the user's situation into the generated AI model. The input consists of swing characteristic data and environmental data, while the output is specific advice.

[0311] Step 6:

[0312] The server converts advice generated via the Google Text-to-Speech API into speech. The input is the text data of the generated advice, and the output is the speech data.

[0313] Step 7:

[0314] The terminal provides audio data from the server to the user in real time via earphones. The analysis results are also visually displayed on the smart device's screen. Input consists of audio data and analysis result data, while output consists of user-recognizable audio and visual information.

[0315] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0316] This invention provides a system for golfers to receive personalized, real-time advice during actual rounds, and is characterized by its consideration of the user's emotional state. It not only understands the user's motor characteristics and provides advice that takes environmental conditions into account, but also recognizes emotional elements such as stress and satisfaction during play, and enables feedback based on these.

[0317] First, the user films their golf swing using a dedicated smartphone application. The video includes the swing form and the flow of force. This video is temporarily stored on the device, while location information is also recorded simultaneously. Furthermore, audio and facial expression data are also captured and processed by an emotion engine.

[0318] The device sends this data to a server in the cloud. The server uses computer vision technology to analyze the video data and identify the user's movement characteristics. In addition to this movement analysis, it performs voice analysis and facial recognition to determine the user's emotional state. For example, if the tone of voice or facial expression differs from the usual pattern, the emotion engine detects this change.

[0319] Next, the server retrieves weather information and wind direction data from an external database in real time and analyzes this data along with the user's movement characteristics and emotional state. Based on this analysis, the AI ​​generates optimal advice for the user. For example, if the AI ​​determines that the user is feeling stressed, the advice may include suggestions to take deep breaths to relax.

[0320] Finally, the device transcribes the generated advice into text and provides it to the user in real time through a speech synthesis system. Simultaneously, the smartphone screen displays additional visual information, including details of the advice and information tailored to the user's emotional state.

[0321] For example, if a user is unable to perform at their usual level on a windy day, the system will provide feedback such as, "Try to swing through firmly without being defeated by the wind. Relax a little and concentrate on the next shot." In this way, users can efficiently improve their scores while also being able to manage their motivation and emotions.

[0322] The following describes the processing flow.

[0323] Step 1:

[0324] The user launches a dedicated smartphone app at a golf course or driving range and records a video of their swing. The device's camera is used to record the user's swing, while the device's microphone collects audio data and records location information.

[0325] Step 2:

[0326] The device compresses the acquired swing video, audio data, and location information and securely transmits it to the server. This data includes the user's swing characteristics and emotional indicators.

[0327] Step 3:

[0328] The server analyzes video data using computer vision algorithms to identify the user's swing characteristics, form, and force application. Furthermore, it analyzes audio data with an emotion engine to determine the user's emotional state from their tone and word choice.

[0329] Step 4:

[0330] The server retrieves weather and wind direction information in real time from an external database via the internet. This environmental data is then integrated and analyzed with swing characteristics and emotional state.

[0331] Step 5:

[0332] The server utilizes an AI model to generate optimal advice that takes into account swing analysis data, emotional state, and weather information. If the user is feeling stressed, it will also include advice on breathing techniques to calm down and points for concentration.

[0333] Step 6:

[0334] The device receives advice from the server and converts it into voice feedback using a speech synthesis engine. This feedback is presented in real time through the user's earphones, and the screen displays visual feedback and detailed information tailored to the user's emotional state.

[0335] Step 7:

[0336] Based on the audio advice and visual information provided, users can adjust their swing and compose themselves before taking the next shot. This allows for improved play quality and a better golfing experience.

[0337] (Example 2)

[0338] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0339] There is a need to provide appropriate and personalized advice to individual users by evaluating their performance and emotional state during exercise in real time, and by taking environmental conditions into consideration. However, conventional technologies have limited systems that can provide feedback that fully considers the user's emotional state and environmental conditions, resulting in a lack of means to efficiently improve performance. As a result, it has been difficult for users to improve their performance more effectively and engage in activities in a mentally stable state.

[0340] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0341] In this invention, the server includes a camera and location information device that acquires the actions of individual users, an analysis device that analyzes the video information of actions acquired by the camera and extracts the user's action characteristics and emotional state, and an environmental information device that acquires environmental information in real time. This makes it possible to provide users with optimal and personalized real-time advice.

[0342] A "recording device" refers to hardware or software used to record a user's actions in real time, and is responsible for acquiring image or video data.

[0343] A "location information device" is a technology used to determine a user's current location. It is a device that acquires location information in real time using GPS or other location information technologies.

[0344] "Video information" refers to image or video data acquired by a recording device, and serves as basic data for analyzing the user's behavioral characteristics using an analysis device.

[0345] "Analysis device" refers to software or hardware that analyzes the user's behavioral characteristics and emotional state in real time based on data obtained from a camera and a location information device.

[0346] "Motional characteristics" refer to the characteristics and performance tendencies of the user's movements and behaviors, and the analysis device evaluates the data based on these characteristics.

[0347] "Emotional state" refers to the user's mental health and emotional changes, and is a psychological state analyzed from voice and facial expression data.

[0348] An "environmental information device" is a device that acquires environmental information in real time, such as weather, temperature, and wind direction, regarding external conditions when users are engaged in activities.

[0349] A "support generation device" is a device that integrates data obtained from analysis devices and environmental information devices to generate optimal advice for the user.

[0350] A "voice presentation device" refers to a system that converts generated advice into voice and presents it to the user in an easy-to-understand manner, and includes speakers and speech synthesis software.

[0351] This system aims to evaluate users' motor performance and emotional state in real time and provide support based on that evaluation. The specific implementation method is described below.

[0352] First, the user uses a mobile device with a dedicated application installed to record their movements. This device has a built-in camera and location tracking device, and location information is acquired along with the recorded video data. The video records the user's movement form and force flow, and simultaneously captures audio and facial expression data.

[0353] Next, the device sends the acquired data to a server in the cloud. This communication uses a secure protocol to ensure security. The server analyzes the received video data using computer vision technology and utilizes software such as OpenCV and TensorFlow to identify the user's behavioral characteristics. It also analyzes audio and facial expression data to determine the user's emotional state.

[0354] The server calls external APIs to obtain real-time weather and wind direction information from external data sources. It integrates this data and uses a generative AI model to generate optimal advice for the user. This advice includes specific actions to improve athletic performance and suggestions to support emotional management. Accurate feedback is obtained by inputting the analysis results as prompts into the generative AI model.

[0355] Finally, the device uses speech synthesis technology to convert the generated advice into speech and provides it to the user in real time. The device's screen also displays detailed advice and visual information tailored to the user's emotional state. This allows the user to improve their performance and feel more at ease while working.

[0356] For example, the system could provide advice to a user on a windy day such as, "Try to swing through firmly without being defeated by the wind. Relax a little and concentrate on the next shot." Another example of a prompt to input to the generating AI model is, "Please provide advice to improve the user's golf performance, taking into account the stressful situation the user is experiencing."

[0357] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0358] Step 1:

[0359] Users record their actions using a dedicated smartphone app. Input consists of video, audio, and facial expression data obtained through the camera and microphone. Users utilize this data and save it to their device. Specifically, after activating the camera, users perform a series of actions such as a swing or a shot, and the video is recorded on the device.

[0360] Step 2:

[0361] The device acquires location information along with stored video data and sends the data to the server. Inputs include video, audio, and facial expression data stored on the device, as well as acquired location information. These are aggregated and transferred to the server using a secure protocol. Specifically, the device utilizes its GPS sensor and network communication capabilities to package and transmit the data.

[0362] Step 3:

[0363] The server uses computer vision technology to analyze the transmitted video data. The input is video data received from the terminal, and the output is the result of extracting the user's movement characteristics. Specifically, the server uses OpenCV and TensorFlow to analyze joint movements and swing trajectories, and converts the user's movement characteristics into data.

[0364] Step 4:

[0365] The server uses voice analysis and facial recognition technology to determine the user's emotional state. Input is voice and facial expression data, and output is the result of the assessment of the user's emotional state. Specifically, the server evaluates the tone and speed of the voice and changes in facial expression, and based on this, analyzes stress levels and emotional fluctuations.

[0366] Step 5:

[0367] The server calls an external API to obtain environmental data in real time. The input is the endpoint of the environmental data API, and the output is current weather and wind direction information. Specifically, the server generates an API request and retrieves the necessary data in real time.

[0368] Step 6:

[0369] The server uses integrated data (behavioral characteristics, emotional state, and environmental information) to activate a generative AI model and generate optimal advice for the user. The input is the aforementioned data set, and the output is the content of the advice for the user. Specifically, the server inputs this data as prompts into the model and generates real-time feedback for the user.

[0370] Step 7:

[0371] The terminal converts the output from the server into speech using speech synthesis technology and provides it to the user. The input is the text of advice sent from the server, and the output is the audio and visual information presented to the user. Specifically, the terminal generates speech from the text using a speech synthesis engine and displays the related visuals on the display.

[0372] (Application Example 2)

[0373] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0374] This invention aims to solve the problem of a lack of support for golfers to manage their technical performance and mental state simultaneously during a round and achieve optimal play. In particular, while conventional systems can provide more personalized advice by considering not only the user's motor characteristics but also their emotional state, such technology does not yet exist.

[0375] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0376] In this invention, the server includes image information acquisition means, audio information acquisition means and location information acquisition means for acquiring the motor movements and emotional states of individual users, motion and emotion analysis means for analyzing the image data and audio data of the motor movements acquired by the image information acquisition means and audio information acquisition means to extract the user's motion characteristics and emotional characteristics, and environmental information acquisition means for acquiring environmental information in real time. This makes it possible to provide users with real-time optimal advice that takes into account their motion characteristics and emotional states, thereby improving technical performance and promoting mental stability.

[0377] "Individual users" refers to specific human users, and those individuals are users of the system.

[0378] "Motor movement" refers to a series of movements or actions that a person performs when moving their body, and includes movements in specific tasks or sports.

[0379] "Emotional state" refers to the psychological or emotional condition a user is experiencing at a given moment, and includes stress, satisfaction, concentration, etc.

[0380] "Means for acquiring image information" refers to devices or technologies for capturing the movement of a user, and includes cameras and video recorders.

[0381] "Voice information acquisition means" refers to a device or technology for recording voices emitted by a user, and includes microphones and the like.

[0382] "Location information acquisition means" refers to a device or technology for determining the geographical location of a user, including location information services such as GPS.

[0383] "Motion and emotion analysis means" refers to a device or technology for analyzing acquired image data and audio data to identify the user's motor and emotional characteristics.

[0384] "Environmental information acquisition means" refers to devices or technologies for acquiring information about the external environment in real time, and includes sensors and information acquisition services via the Internet.

[0385] An "advice generation means" is a device or technology that generates optimal advice for the user based on analyzed data.

[0386] "Voice output means" refers to a device or technology for converting generated advice into voice and conveying it to the user, and includes speakers and speech synthesizers.

[0387] The system for implementing this invention analyzes the user's motor movements and emotional state in real time and provides personalized advice. First, the user acquires video and audio data of their swing using the camera and microphone built into their smartphone. This data also includes the user's location information.

[0388] Next, the device sends the acquired data to a server in the cloud. The server analyzes the video data using computer vision technology (e.g., OpenCV) to understand the user's movement characteristics. Simultaneously, it analyzes the audio data and facial expressions using emotion recognition technology (e.g., Affectiva) to determine the user's emotional state.

[0389] Furthermore, the server uses environmental information, including weather and wind direction data, obtained from external APIs (e.g., OpenWeatherMap), to integrate the analysis results. Based on these results, a generative AI model generates advice tailored to the user's technical performance and emotional state.

[0390] Finally, the device presents the generated advice to the user verbally using speech synthesis technology (e.g., Google Text-to-Speech). This process allows the user to receive both technical and emotional support in real time.

[0391] For example, if a user's shots are inconsistent in strong winds, the system will generate advice such as, "Concentrate to stabilize your swing against the wind." An example of a prompt would be, "Based on the user's swing data and emotional data, generate optimal golf advice for windy conditions."

[0392] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0393] Step 1:

[0394] The user uses their smartphone to film their golf swing and record audio. During this stage, the camera captures video data, the microphone collects audio, and location information is recorded simultaneously. The inputs are video data, audio data, and location information, which are temporarily stored on the device.

[0395] Step 2:

[0396] The device transmits acquired video data, audio data, and location information to a server in the cloud. This process uses a communication module to package the data and send it to the server securely. The input consists of various data stored on the device, and the output is the completion of data transmission to the server.

[0397] Step 3:

[0398] The server analyzes the video data. Computer vision technology (e.g., OpenCV) is used to extract the swing motion characteristics. At this stage, the input is video data, and the output is user motion characteristic information.

[0399] Step 4:

[0400] The server analyzes audio and facial expression data to determine the user's emotional state. Emotion recognition technology (e.g., Affectiva) identifies stress levels and emotions from voice tone and facial expressions. The input is audio and video data, and the output is information about the user's emotional state.

[0401] Step 5:

[0402] The server uses an external API to obtain real-time weather and wind direction information (e.g., OpenWeatherMap). This allows for the collection of up-to-date information about the play environment. The input is a request to obtain weather and wind direction data, and the output is a set of environmental information.

[0403] Step 6:

[0404] The server integrates operational characteristics, emotional states, and environmental information, and uses a generative AI model to generate optimal advice. This analysis quickly generates situation-appropriate advice for the user. The input is an integrated dataset, and the output is the generated advice.

[0405] Step 7:

[0406] The device outputs the generated advice as speech using speech synthesis technology (e.g., Google Text-to-Speech). Simultaneously, it also displays the advice to the user in text. The input is the generated advice, and the output is the audio and text presentation to the user.

[0407] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0408] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0409] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart glasses 214.

[0410] [Third Embodiment]

[0411] Figure 5 shows an example of the configuration of the data processing system 310 according to the third embodiment.

[0412] As shown in Figure 5, the data processing system 310 includes a data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0413] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0414] The headset terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a display 343. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and display 343 are also connected to the bus 52.

[0415] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0416] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0417] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0418] Figure 6 shows an example of the main functions of the data processing device 12 and the headset terminal 314. As shown in Figure 6, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0419] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0420] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0421] In the headset terminal 314, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0422] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the headset terminal 314 will be referred to as the "terminal".

[0423] This invention provides a system that allows users to receive real-time advice on a golf course based on their individual movement characteristics. This system is realized through a series of processes that acquire and analyze the user's movement and provide appropriate advice.

[0424] First, the user launches a smartphone application at a golf driving range or course and films their swing. At this time, the application records video data of the swing using the camera (image acquisition means) built into the smartphone or dedicated device, and simultaneously acquires location information using GPS.

[0425] Next, the terminal sends the acquired data to the server. Here, the server uses advanced computer vision technology to analyze the swing. Specifically, it analyzes the swing speed, angle, and posture to extract detailed user characteristics. The server also collects environmental data such as golf course weather information and wind direction via the internet.

[0426] The server then integrates this data and uses an AI model to generate optimal advice for the user. For example, if the user's swing is delayed compared to their normal swing, it will provide specific advice on how to improve their swing timing.

[0427] Finally, the device converts the advice received from the server into audio via an audio output device and provides real-time feedback through the user's earphones. In addition, the smartphone screen visually displays an image of the swing and the analysis results.

[0428] As a concrete example, suppose a user is playing on a windy day and their shots tend to veer to the right more than usual. In this case, the system takes wind direction data into consideration and provides voice advice such as, "Aim slightly to the left when you swing, and grip the club firmly." In this way, the present invention helps users benefit from immediately applicable improvement measures in their actual round environment.

[0429] The following describes the processing flow.

[0430] Step 1:

[0431] The user launches a smartphone app at a golf course or driving range and records a video with the camera at the moment of their swing. During this process, the app automatically records location information using the device's built-in GPS.

[0432] Step 2:

[0433] The device compresses the acquired swing video and location information and sends it to the server. This involves a data transfer process via an internet connection.

[0434] Step 3:

[0435] The server analyzes the received video data. This video analysis involves using computer vision algorithms to extract details such as swing speed, angle, and body movements.

[0436] Step 4:

[0437] The server retrieves real-time weather information and wind direction data for the golf course from external web services. This environmental data is then integrated with swing analysis data.

[0438] Step 5:

[0439] The server uses an AI model to calculate based on motion analysis data and environmental data, generating advice to help the user improve their swing. For example, it can create advice on club selection considering wind effects and the amount of force to use in a swing.

[0440] Step 6:

[0441] The server generates advice, converts it into text, and then generates and sends this information along with any necessary image videos to the terminal.

[0442] Step 7:

[0443] The advice information received by the device is converted into speech using a speech synthesis engine and output in real time through the user's earphones. In addition, detailed advice and illustrative images are displayed on the smartphone screen.

[0444] Step 8:

[0445] Based on the advice provided by the user, they adjust their swing for the next shot and attempt to improve their play.

[0446] (Example 1)

[0447] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0448] In sports, particularly golf, a challenge has been the difficulty for individual users to obtain real-time feedback on their own movement patterns. Traditional methods required significant time and effort to provide specific advice that took individual movement characteristics into account, and failed to adequately incorporate environmental factors. As a result, improvements were not immediately reflected, and it took time for users' performance to improve.

[0449] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0450] In this invention, the server includes a means for capturing images, a means for acquiring location information, a means for extracting features, a means for collecting situational data, a means for generating advice, a means for presenting voice, and a means for displaying information. This enables individual users to receive real-time feedback on their own movement and to immediately benefit from optimal improvement measures that take environmental factors into consideration.

[0451] A "shooting device" is a mechanism used to record a user's movement and is a device that has the function of acquiring image data.

[0452] A "location information acquisition means" is a mechanism for determining the user's current location, and is a device that acquires location data using technologies such as GPS.

[0453] A "feature extraction means" is a mechanism that analyzes acquired image information and quantifies the characteristics of the user's motor movements.

[0454] A "situational data collection means" is a mechanism for collecting environmental information in real time, and is a device that acquires external data such as weather information and wind direction information.

[0455] The "advice generation mechanism" is a system that generates optimal advice for the user using a generation AI model based on acquired exercise data and environmental data.

[0456] A "voice presentation means" is a mechanism for converting generated advice into voice and conveying it to the user.

[0457] A "display means" is a mechanism for visually displaying information about the user's movement and analysis results.

[0458] This invention is a system designed to help users receive real-time advice at the golf course. The user uses a smartphone application to film and analyze their movements. Specifically, the user films their swing using their smartphone camera, and location information is obtained using GPS technology. The data thus obtained is transmitted from the terminal to a server.

[0459] The server analyzes the captured video data using computer vision technology. This analysis utilizes libraries such as TensorFlow to extract motion characteristics such as swing speed, angle, and posture through image processing. Meanwhile, real-time environmental information is collected via the internet, including weather data and wind direction data.

[0460] The server integrates the acquired data and uses a generative AI model to generate specific advice for the user. The generated advice is sent to the device and conveyed through the user's earphones using an audio presentation system. In addition, the analysis results are visually displayed on the smartphone screen, allowing the user to immediately check their swing performance.

[0461] A concrete example is a scenario where a strong wind causes the ball to veer to the right. In this case, the system provides advice that takes the wind direction into account, such as "aim slightly to the left and grip the club firmly." An example of this prompt would be, "Please provide optimal swing advice for a golf course with strong winds." This allows users to receive real-time improvement suggestions tailored to their current environment.

[0462] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0463] Step 1:

[0464] The user launches a smartphone app at the golf course and records a video of their swing. During this process, the smartphone's camera records the movement, while simultaneously acquiring location information using GPS. The input consists of camera footage and GPS location data. The output is a video file of the swing and location information stored in the device's internal storage.

[0465] Step 2:

[0466] The device transmits the acquired video data and location information to the server. The input consists of the video file and location data stored on the device. The output is the data transferred to the server via the internet.

[0467] Step 3:

[0468] The server analyzes the received video data using computer vision technology. Specifically, it uses TensorFlow to perform image processing and extract motion characteristics such as swing speed, angle, and posture from each frame of the video. The input is the video file sent to the server. The output is data in which the motion characteristics are quantified.

[0469] Step 4:

[0470] The server collects environmental data in real time. This includes obtaining weather information and wind direction data from external sources on the internet. Inputs include weather data APIs accessed via the internet. Outputs are environmental information data stored on the server.

[0471] Step 5:

[0472] The server integrates extracted movement characteristics data and environmental information, and uses a generative AI model to generate advice for the user. The AI ​​model receives the prompt "Generate optimal golf advice based on the user's movement characteristics." The output is improvement advice tailored to the user.

[0473] Step 6:

[0474] The server sends the generated advice to the terminal. The input is the generated advice data. The output is the advice data sent to the terminal via the internet.

[0475] Step 7:

[0476] The device converts received advice into audio and provides feedback through the user's earphones. It also visually displays the analysis results on the smartphone screen. The input is advice data sent from the server. The output consists of audio feedback and visually displayed analysis results.

[0477] (Application Example 1)

[0478] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0479] Traditional golf practice methods lacked real-time feedback, making it difficult for users to make immediate improvements. Furthermore, there were insufficient means to receive appropriate advice that took environmental factors such as wind direction into account. This hindered effective skill development.

[0480] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0481] In this invention, the server includes image acquisition means and location information acquisition means to acquire the motor movements of individual users. It also includes advice generation means that integrates data obtained from motion analysis means and environmental data acquisition means to generate optimal advice, and feedback means that evaluate in real time and provide individual feedback. This makes it possible for users to receive individually optimized advice in real time.

[0482] "Image acquisition means" refers to devices and technologies that record a user's movement as visual data and provide that information in an analyzable format.

[0483] "Location information acquisition means" refers to technology for acquiring geographical information about the location where a user is exercising.

[0484] "Motion analysis means" refers to a technology that analyzes acquired visual data to scientifically evaluate the characteristics and patterns of a user's movements.

[0485] "Environmental data acquisition methods" refer to technologies for collecting data on the external environment when users exercise, such as weather and wind direction.

[0486] "Advice generation means" refers to technology that creates specific advice for users aimed at improving their athletic performance based on analyzed data.

[0487] "Voice output means" refers to devices or technologies for communicating generated advice to the user in voice.

[0488] A "feedback mechanism" refers to a technology or device that evaluates a user's movements in real time and provides appropriate feedback based on the results.

[0489] This system is designed to allow golfers to evaluate their swings in real time and receive personalized, optimized feedback. Users first film their swing using a dedicated smart device (smartphone or dedicated camera). This utilizes a high-performance camera sensor (e.g., Sony IMX sensor).

[0490] The device transfers the captured video data to the server via Bluetooth or Wi-Fi. The server uses machine learning libraries such as TensorFlow to perform motion analysis based on computer vision technology. This analysis extracts detailed data such as swing speed, angle, and posture.

[0491] Furthermore, the server utilizes APIs such as OpenWeatherMap to acquire real-time weather information and wind direction data. This operational and environmental data is integrated, and optimal advice is generated using a generative AI model. This generative AI model uses user-customized prompts to form specific advice tailored to the situation and swing type.

[0492] Finally, the suggestions generated by the advice generation system are converted into speech using the Google Text-to-Speech API and fed back to the user in real time through earphones. In addition, the analysis results are visually displayed on the smart device's screen, and slow-motion swing playback is also performed to further enhance understanding.

[0493] As a concrete example, consider a situation where you are playing golf on a windy day. The generative AI model provides practical advice such as, "Taking the wind into account, I recommend adding a little more ballast for your next shot." In this way, the system provides the user with useful feedback in real time.

[0494] An example of a prompt for the generating AI model is, "Consider the user's swing data, current weather information, and wind direction data to generate optimal golf swing advice." In response to this prompt, the system will produce specific and effective advice tailored to the situation.

[0495] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0496] Step 1:

[0497] The user films their swing using a smart device. A camera sensor is used to capture high-resolution video. The input is the user's swing motion, and the output is video data of this motion.

[0498] Step 2:

[0499] The device saves the acquired video data to local storage and transfers it to a remote server via Bluetooth or Wi-Fi. The input is video data, and the output is data transfer in a format accessible to the server.

[0500] Step 3:

[0501] The server analyzes the received video data. Using computer vision technology based on TensorFlow, it calculates swing speed, angle, and posture. The input is video data, and the output is detailed data on swing characteristics.

[0502] Step 4:

[0503] The server uses APIs such as OpenWeatherMap to obtain real-time weather information and wind direction data. The input is location information sent to the API, and the output is weather information and wind direction information.

[0504] Step 5:

[0505] The server integrates motion analysis results and environmental data. Using prompts, it generates advice tailored to the user's situation into the generated AI model. The input consists of swing characteristic data and environmental data, while the output is specific advice.

[0506] Step 6:

[0507] The server converts advice generated via the Google Text-to-Speech API into speech. The input is the text data of the generated advice, and the output is the speech data.

[0508] Step 7:

[0509] The terminal provides audio data from the server to the user in real time via earphones. The analysis results are also visually displayed on the smart device's screen. Input consists of audio data and analysis result data, while output consists of user-recognizable audio and visual information.

[0510] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0511] This invention provides a system for golfers to receive personalized, real-time advice during actual rounds, and is characterized by its consideration of the user's emotional state. It not only understands the user's motor characteristics and provides advice that takes environmental conditions into account, but also recognizes emotional elements such as stress and satisfaction during play, and enables feedback based on these.

[0512] First, the user films their golf swing using a dedicated smartphone application. The video includes the swing form and the flow of force. This video is temporarily stored on the device, while location information is also recorded simultaneously. Furthermore, audio and facial expression data are also captured and processed by an emotion engine.

[0513] The device sends this data to a server in the cloud. The server uses computer vision technology to analyze the video data and identify the user's movement characteristics. In addition to this movement analysis, it performs voice analysis and facial recognition to determine the user's emotional state. For example, if the tone of voice or facial expression differs from the usual pattern, the emotion engine detects this change.

[0514] Next, the server retrieves weather information and wind direction data from an external database in real time and analyzes this data along with the user's movement characteristics and emotional state. Based on this analysis, the AI ​​generates optimal advice for the user. For example, if the AI ​​determines that the user is feeling stressed, the advice may include suggestions to take deep breaths to relax.

[0515] Finally, the device transcribes the generated advice into text and provides it to the user in real time through a speech synthesis system. Simultaneously, the smartphone screen displays additional visual information, including details of the advice and information tailored to the user's emotional state.

[0516] For example, if a user is unable to perform at their usual level on a windy day, the system will provide feedback such as, "Try to swing through firmly without being defeated by the wind. Relax a little and concentrate on the next shot." In this way, users can efficiently improve their scores while also being able to manage their motivation and emotions.

[0517] The following describes the processing flow.

[0518] Step 1:

[0519] The user launches a dedicated smartphone app at a golf course or driving range and records a video of their swing. The device's camera is used to record the user's swing, while the device's microphone collects audio data and records location information.

[0520] Step 2:

[0521] The device compresses the acquired swing video, audio data, and location information and securely transmits it to the server. This data includes the user's swing characteristics and emotional indicators.

[0522] Step 3:

[0523] The server analyzes video data using computer vision algorithms to identify the user's swing characteristics, form, and force application. Furthermore, it analyzes audio data with an emotion engine to determine the user's emotional state from their tone and word choice.

[0524] Step 4:

[0525] The server retrieves weather and wind direction information in real time from an external database via the internet. This environmental data is then integrated and analyzed with swing characteristics and emotional state.

[0526] Step 5:

[0527] The server utilizes an AI model to generate optimal advice that takes into account swing analysis data, emotional state, and weather information. If the user is feeling stressed, it will also include advice on breathing techniques to calm down and points for concentration.

[0528] Step 6:

[0529] The device receives advice from the server and converts it into voice feedback using a speech synthesis engine. This feedback is presented in real time through the user's earphones, and the screen displays visual feedback and detailed information tailored to the user's emotional state.

[0530] Step 7:

[0531] Based on the audio advice and visual information provided, users can adjust their swing and compose themselves before taking the next shot. This allows for improved play quality and a better golfing experience.

[0532] (Example 2)

[0533] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0534] There is a need to provide appropriate and personalized advice to individual users by evaluating their performance and emotional state during exercise in real time, and by taking environmental conditions into consideration. However, conventional technologies have limited systems that can provide feedback that fully considers the user's emotional state and environmental conditions, resulting in a lack of means to efficiently improve performance. As a result, it has been difficult for users to improve their performance more effectively and engage in activities in a mentally stable state.

[0535] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0536] In this invention, the server includes a camera and location information device that acquires the actions of individual users, an analysis device that analyzes the video information of actions acquired by the camera and extracts the user's action characteristics and emotional state, and an environmental information device that acquires environmental information in real time. This makes it possible to provide users with optimal and personalized real-time advice.

[0537] A "recording device" refers to hardware or software used to record a user's actions in real time, and is responsible for acquiring image or video data.

[0538] A "location information device" is a technology used to determine a user's current location. It is a device that acquires location information in real time using GPS or other location information technologies.

[0539] "Video information" refers to image or video data acquired by a recording device, and serves as basic data for analyzing the user's behavioral characteristics using an analysis device.

[0540] "Analysis device" refers to software or hardware that analyzes the user's behavioral characteristics and emotional state in real time based on data obtained from a camera and a location information device.

[0541] "Motional characteristics" refer to the characteristics and performance tendencies of the user's movements and behaviors, and the analysis device evaluates the data based on these characteristics.

[0542] "Emotional state" refers to the user's mental health and emotional changes, and is a psychological state analyzed from voice and facial expression data.

[0543] An "environmental information device" is a device that acquires environmental information in real time, such as weather, temperature, and wind direction, regarding external conditions when users are engaged in activities.

[0544] A "support generation device" is a device that integrates data obtained from analysis devices and environmental information devices to generate optimal advice for the user.

[0545] A "voice presentation device" refers to a system that converts generated advice into voice and presents it to the user in an easy-to-understand manner, and includes speakers and speech synthesis software.

[0546] This system aims to evaluate users' motor performance and emotional state in real time and provide support based on that evaluation. The specific implementation method is described below.

[0547] First, the user uses a mobile device with a dedicated application installed to record their movements. This device has a built-in camera and location tracking device, and location information is acquired along with the recorded video data. The video records the user's movement form and force flow, and simultaneously captures audio and facial expression data.

[0548] Next, the device sends the acquired data to a server in the cloud. This communication uses a secure protocol to ensure security. The server analyzes the received video data using computer vision technology and utilizes software such as OpenCV and TensorFlow to identify the user's behavioral characteristics. It also analyzes audio and facial expression data to determine the user's emotional state.

[0549] The server calls external APIs to obtain real-time weather and wind direction information from external data sources. It integrates this data and uses a generative AI model to generate optimal advice for the user. This advice includes specific actions to improve athletic performance and suggestions to support emotional management. Accurate feedback is obtained by inputting the analysis results as prompts into the generative AI model.

[0550] Finally, the device uses speech synthesis technology to convert the generated advice into speech and provides it to the user in real time. The device's screen also displays detailed advice and visual information tailored to the user's emotional state. This allows the user to improve their performance and feel more at ease while working.

[0551] For example, the system could provide advice to a user on a windy day such as, "Try to swing through firmly without being defeated by the wind. Relax a little and concentrate on the next shot." Another example of a prompt to input to the generating AI model is, "Please provide advice to improve the user's golf performance, taking into account the stressful situation the user is experiencing."

[0552] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0553] Step 1:

[0554] Users record their actions using a dedicated smartphone app. Input consists of video, audio, and facial expression data obtained through the camera and microphone. Users utilize this data and save it to their device. Specifically, after activating the camera, users perform a series of actions such as a swing or a shot, and the video is recorded on the device.

[0555] Step 2:

[0556] The device acquires location information along with stored video data and sends the data to the server. Inputs include video, audio, and facial expression data stored on the device, as well as acquired location information. These are aggregated and transferred to the server using a secure protocol. Specifically, the device utilizes its GPS sensor and network communication capabilities to package and transmit the data.

[0557] Step 3:

[0558] The server uses computer vision technology to analyze the transmitted video data. The input is video data received from the terminal, and the output is the result of extracting the user's movement characteristics. Specifically, the server uses OpenCV and TensorFlow to analyze joint movements and swing trajectories, and converts the user's movement characteristics into data.

[0559] Step 4:

[0560] The server uses voice analysis and facial recognition technology to determine the user's emotional state. Input is voice and facial expression data, and output is the result of the assessment of the user's emotional state. Specifically, the server evaluates the tone and speed of the voice and changes in facial expression, and based on this, analyzes stress levels and emotional fluctuations.

[0561] Step 5:

[0562] The server calls an external API to obtain environmental data in real time. The input is the endpoint of the environmental data API, and the output is current weather and wind direction information. Specifically, the server generates an API request and retrieves the necessary data in real time.

[0563] Step 6:

[0564] The server uses integrated data (behavioral characteristics, emotional state, and environmental information) to activate a generative AI model and generate optimal advice for the user. The input is the aforementioned data set, and the output is the content of the advice for the user. Specifically, the server inputs this data as prompts into the model and generates real-time feedback for the user.

[0565] Step 7:

[0566] The terminal converts the output from the server into speech using speech synthesis technology and provides it to the user. The input is the text of advice sent from the server, and the output is the audio and visual information presented to the user. Specifically, the terminal generates speech from the text using a speech synthesis engine and displays the related visuals on the display.

[0567] (Application Example 2)

[0568] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0569] This invention aims to solve the problem of a lack of support for golfers to manage their technical performance and mental state simultaneously during a round and achieve optimal play. In particular, while conventional systems can provide more personalized advice by considering not only the user's motor characteristics but also their emotional state, such technology does not yet exist.

[0570] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0571] In this invention, the server includes image information acquisition means, audio information acquisition means and location information acquisition means for acquiring the motor movements and emotional states of individual users, motion and emotion analysis means for analyzing the image data and audio data of the motor movements acquired by the image information acquisition means and audio information acquisition means to extract the user's motion characteristics and emotional characteristics, and environmental information acquisition means for acquiring environmental information in real time. This makes it possible to provide users with real-time optimal advice that takes into account their motion characteristics and emotional states, thereby improving technical performance and promoting mental stability.

[0572] "Individual users" refers to specific human users, and those individuals are users of the system.

[0573] "Motor movement" refers to a series of movements or actions that a person performs when moving their body, and includes movements in specific tasks or sports.

[0574] "Emotional state" refers to the psychological or emotional condition a user is experiencing at a given moment, and includes stress, satisfaction, concentration, etc.

[0575] "Means for acquiring image information" refers to devices or technologies for capturing the movement of a user, and includes cameras and video recorders.

[0576] "Voice information acquisition means" refers to a device or technology for recording voices emitted by a user, and includes microphones and the like.

[0577] "Location information acquisition means" refers to a device or technology for determining the geographical location of a user, including location information services such as GPS.

[0578] "Motion and emotion analysis means" refers to a device or technology for analyzing acquired image data and audio data to identify the user's motor and emotional characteristics.

[0579] "Environmental information acquisition means" refers to devices or technologies for acquiring information about the external environment in real time, and includes sensors and information acquisition services via the Internet.

[0580] An "advice generation means" is a device or technology that generates optimal advice for the user based on analyzed data.

[0581] "Voice output means" refers to a device or technology for converting generated advice into voice and conveying it to the user, and includes speakers and speech synthesizers.

[0582] The system for implementing this invention analyzes the user's motor movements and emotional state in real time and provides personalized advice. First, the user acquires video and audio data of their swing using the camera and microphone built into their smartphone. This data also includes the user's location information.

[0583] Next, the device sends the acquired data to a server in the cloud. The server analyzes the video data using computer vision technology (e.g., OpenCV) to understand the user's movement characteristics. Simultaneously, it analyzes the audio data and facial expressions using emotion recognition technology (e.g., Affectiva) to determine the user's emotional state.

[0584] Furthermore, the server uses environmental information, including weather and wind direction data, obtained from external APIs (e.g., OpenWeatherMap), to integrate the analysis results. Based on these results, a generative AI model generates advice tailored to the user's technical performance and emotional state.

[0585] Finally, the device presents the generated advice to the user verbally using speech synthesis technology (e.g., Google Text-to-Speech). This process allows the user to receive both technical and emotional support in real time.

[0586] For example, if a user's shots are inconsistent in strong winds, the system will generate advice such as, "Concentrate to stabilize your swing against the wind." An example of a prompt would be, "Based on the user's swing data and emotional data, generate optimal golf advice for windy conditions."

[0587] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0588] Step 1:

[0589] The user uses their smartphone to film their golf swing and record audio. During this stage, the camera captures video data, the microphone collects audio, and location information is recorded simultaneously. The inputs are video data, audio data, and location information, which are temporarily stored on the device.

[0590] Step 2:

[0591] The device transmits acquired video data, audio data, and location information to a server in the cloud. This process uses a communication module to package the data and send it to the server securely. The input consists of various data stored on the device, and the output is the completion of data transmission to the server.

[0592] Step 3:

[0593] The server analyzes the video data. Computer vision technology (e.g., OpenCV) is used to extract the swing motion characteristics. At this stage, the input is video data, and the output is user motion characteristic information.

[0594] Step 4:

[0595] The server analyzes audio and facial expression data to determine the user's emotional state. Emotion recognition technology (e.g., Affectiva) identifies stress levels and emotions from voice tone and facial expressions. The input is audio and video data, and the output is information about the user's emotional state.

[0596] Step 5:

[0597] The server uses an external API to obtain real-time weather and wind direction information (e.g., OpenWeatherMap). This allows for the collection of up-to-date information about the play environment. The input is a request to obtain weather and wind direction data, and the output is a set of environmental information.

[0598] Step 6:

[0599] The server integrates operational characteristics, emotional states, and environmental information, and uses a generative AI model to generate optimal advice. This analysis quickly generates situation-appropriate advice for the user. The input is an integrated dataset, and the output is the generated advice.

[0600] Step 7:

[0601] The device outputs the generated advice as speech using speech synthesis technology (e.g., Google Text-to-Speech). Simultaneously, it also displays the advice to the user in text. The input is the generated advice, and the output is the audio and text presentation to the user.

[0602] The specific processing unit 290 transmits the result of the specific processing to the headset terminal 314. In the headset terminal 314, the control unit 46A causes the speaker 240 and display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0603] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0604] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and specific processing may also be performed by the headset terminal 314.

[0605] [Fourth Embodiment]

[0606] Figure 7 shows an example of the configuration of the data processing system 410 according to the fourth embodiment.

[0607] As shown in Figure 7, the data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[0608] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0609] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a controlled object 443. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and controlled object 443 are also connected to the bus 52.

[0610] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0611] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0612] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0613] The controlled object 443 includes a display device, LEDs in the eyes, and motors that drive the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the robot 414's emotions can be expressed by controlling these motors. Furthermore, the robot 414's facial expressions can also be expressed by controlling the illumination state of the LEDs in its eyes.

[0614] Figure 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Figure 8, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0615] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0616] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0617] In robot 414, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0618] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0619] This invention provides a system that allows users to receive real-time advice on a golf course based on their individual movement characteristics. This system is realized through a series of processes that acquire and analyze the user's movement and provide appropriate advice.

[0620] First, the user launches a smartphone application at a golf driving range or course and films their swing. At this time, the application records video data of the swing using the camera (image acquisition means) built into the smartphone or dedicated device, and simultaneously acquires location information using GPS.

[0621] Next, the terminal sends the acquired data to the server. Here, the server uses advanced computer vision technology to analyze the swing. Specifically, it analyzes the swing speed, angle, and posture to extract detailed user characteristics. The server also collects environmental data such as golf course weather information and wind direction via the internet.

[0622] The server then integrates this data and uses an AI model to generate optimal advice for the user. For example, if the user's swing is delayed compared to their normal swing, it will provide specific advice on how to improve their swing timing.

[0623] Finally, the device converts the advice received from the server into audio via an audio output device and provides real-time feedback through the user's earphones. In addition, the smartphone screen visually displays an image of the swing and the analysis results.

[0624] As a concrete example, suppose a user is playing on a windy day and their shots tend to veer to the right more than usual. In this case, the system takes wind direction data into consideration and provides voice advice such as, "Aim slightly to the left when you swing, and grip the club firmly." In this way, the present invention helps users benefit from immediately applicable improvement measures in their actual round environment.

[0625] The following describes the processing flow.

[0626] Step 1:

[0627] The user launches a smartphone app at a golf course or driving range and records a video with the camera at the moment of their swing. During this process, the app automatically records location information using the device's built-in GPS.

[0628] Step 2:

[0629] The device compresses the acquired swing video and location information and sends it to the server. This involves a data transfer process via an internet connection.

[0630] Step 3:

[0631] The server analyzes the received video data. This video analysis involves using computer vision algorithms to extract details such as swing speed, angle, and body movements.

[0632] Step 4:

[0633] The server retrieves real-time weather information and wind direction data for the golf course from external web services. This environmental data is then integrated with swing analysis data.

[0634] Step 5:

[0635] The server uses an AI model to calculate based on motion analysis data and environmental data, generating advice to help the user improve their swing. For example, it can create advice on club selection considering wind effects and the amount of force to use in a swing.

[0636] Step 6:

[0637] The server generates advice, converts it into text, and then generates and sends this information along with any necessary image videos to the terminal.

[0638] Step 7:

[0639] The advice information received by the device is converted into speech using a speech synthesis engine and output in real time through the user's earphones. In addition, detailed advice and illustrative images are displayed on the smartphone screen.

[0640] Step 8:

[0641] Based on the advice provided by the user, they adjust their swing for the next shot and attempt to improve their play.

[0642] (Example 1)

[0643] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0644] In sports, particularly golf, a challenge has been the difficulty for individual users to obtain real-time feedback on their own movement patterns. Traditional methods required significant time and effort to provide specific advice that took individual movement characteristics into account, and failed to adequately incorporate environmental factors. As a result, improvements were not immediately reflected, and it took time for users' performance to improve.

[0645] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0646] In this invention, the server includes a means for capturing images, a means for acquiring location information, a means for extracting features, a means for collecting situational data, a means for generating advice, a means for presenting voice, and a means for displaying information. This enables individual users to receive real-time feedback on their own movement and to immediately benefit from optimal improvement measures that take environmental factors into consideration.

[0647] A "shooting device" is a mechanism used to record a user's movement and is a device that has the function of acquiring image data.

[0648] A "location information acquisition means" is a mechanism for determining the user's current location, and is a device that acquires location data using technologies such as GPS.

[0649] A "feature extraction means" is a mechanism that analyzes acquired image information and quantifies the characteristics of the user's motor movements.

[0650] A "situational data collection means" is a mechanism for collecting environmental information in real time, and is a device that acquires external data such as weather information and wind direction information.

[0651] The "advice generation mechanism" is a system that generates optimal advice for the user using a generation AI model based on acquired exercise data and environmental data.

[0652] A "voice presentation means" is a mechanism for converting generated advice into voice and conveying it to the user.

[0653] A "display means" is a mechanism for visually displaying information about the user's movement and analysis results.

[0654] This invention is a system designed to help users receive real-time advice at the golf course. The user uses a smartphone application to film and analyze their movements. Specifically, the user films their swing using their smartphone camera, and location information is obtained using GPS technology. The data thus obtained is transmitted from the terminal to a server.

[0655] The server analyzes the captured video data using computer vision technology. This analysis utilizes libraries such as TensorFlow to extract motion characteristics such as swing speed, angle, and posture through image processing. Meanwhile, real-time environmental information is collected via the internet, including weather data and wind direction data.

[0656] The server integrates the acquired data and uses a generative AI model to generate specific advice for the user. The generated advice is sent to the device and conveyed through the user's earphones using an audio presentation system. In addition, the analysis results are visually displayed on the smartphone screen, allowing the user to immediately check their swing performance.

[0657] A concrete example is a scenario where a strong wind causes the ball to veer to the right. In this case, the system provides advice that takes the wind direction into account, such as "aim slightly to the left and grip the club firmly." An example of this prompt would be, "Please provide optimal swing advice for a golf course with strong winds." This allows users to receive real-time improvement suggestions tailored to their current environment.

[0658] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0659] Step 1:

[0660] The user launches a smartphone app at the golf course and records a video of their swing. During this process, the smartphone's camera records the movement, while simultaneously acquiring location information using GPS. The input consists of camera footage and GPS location data. The output is a video file of the swing and location information stored in the device's internal storage.

[0661] Step 2:

[0662] The device transmits the acquired video data and location information to the server. The input consists of the video file and location data stored on the device. The output is the data transferred to the server via the internet.

[0663] Step 3:

[0664] The server analyzes the received video data using computer vision technology. Specifically, it uses TensorFlow to perform image processing and extract motion characteristics such as swing speed, angle, and posture from each frame of the video. The input is the video file sent to the server. The output is data in which the motion characteristics are quantified.

[0665] Step 4:

[0666] The server collects environmental data in real time. This includes obtaining weather information and wind direction data from external sources on the internet. Inputs include weather data APIs accessed via the internet. Outputs are environmental information data stored on the server.

[0667] Step 5:

[0668] The server integrates extracted movement characteristics data and environmental information, and uses a generative AI model to generate advice for the user. The AI ​​model receives the prompt "Generate optimal golf advice based on the user's movement characteristics." The output is improvement advice tailored to the user.

[0669] Step 6:

[0670] The server sends the generated advice to the terminal. The input is the generated advice data. The output is the advice data sent to the terminal via the internet.

[0671] Step 7:

[0672] The device converts received advice into audio and provides feedback through the user's earphones. It also visually displays the analysis results on the smartphone screen. The input is advice data sent from the server. The output consists of audio feedback and visually displayed analysis results.

[0673] (Application Example 1)

[0674] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0675] Traditional golf practice methods lacked real-time feedback, making it difficult for users to make immediate improvements. Furthermore, there were insufficient means to receive appropriate advice that took environmental factors such as wind direction into account. This hindered effective skill development.

[0676] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0677] In this invention, the server includes image acquisition means and location information acquisition means to acquire the motor movements of individual users. It also includes advice generation means that integrates data obtained from motion analysis means and environmental data acquisition means to generate optimal advice, and feedback means that evaluate in real time and provide individual feedback. This makes it possible for users to receive individually optimized advice in real time.

[0678] "Image acquisition means" refers to devices and technologies that record a user's movement as visual data and provide that information in an analyzable format.

[0679] "Location information acquisition means" refers to technology for acquiring geographical information about the location where a user is exercising.

[0680] "Motion analysis means" refers to a technology that analyzes acquired visual data to scientifically evaluate the characteristics and patterns of a user's movements.

[0681] "Environmental data acquisition methods" refer to technologies for collecting data on the external environment when users exercise, such as weather and wind direction.

[0682] "Advice generation means" refers to technology that creates specific advice for users aimed at improving their athletic performance based on analyzed data.

[0683] "Voice output means" refers to devices or technologies for communicating generated advice to the user in voice.

[0684] A "feedback mechanism" refers to a technology or device that evaluates a user's movements in real time and provides appropriate feedback based on the results.

[0685] This system is designed to allow golfers to evaluate their swings in real time and receive personalized, optimized feedback. Users first film their swing using a dedicated smart device (smartphone or dedicated camera). This utilizes a high-performance camera sensor (e.g., Sony IMX sensor).

[0686] The device transfers the captured video data to the server via Bluetooth or Wi-Fi. The server uses machine learning libraries such as TensorFlow to perform motion analysis based on computer vision technology. This analysis extracts detailed data such as swing speed, angle, and posture.

[0687] Furthermore, the server utilizes APIs such as OpenWeatherMap to acquire real-time weather information and wind direction data. This operational and environmental data is integrated, and optimal advice is generated using a generative AI model. This generative AI model uses user-customized prompts to form specific advice tailored to the situation and swing type.

[0688] Finally, the suggestions generated by the advice generation system are converted into speech using the Google Text-to-Speech API and fed back to the user in real time through earphones. In addition, the analysis results are visually displayed on the smart device's screen, and slow-motion swing playback is also performed to further enhance understanding.

[0689] As a concrete example, consider a situation where you are playing golf on a windy day. The generative AI model provides practical advice such as, "Taking the wind into account, I recommend adding a little more ballast for your next shot." In this way, the system provides the user with useful feedback in real time.

[0690] An example of a prompt for the generating AI model is, "Consider the user's swing data, current weather information, and wind direction data to generate optimal golf swing advice." In response to this prompt, the system will produce specific and effective advice tailored to the situation.

[0691] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0692] Step 1:

[0693] The user films their swing using a smart device. A camera sensor is used to capture high-resolution video. The input is the user's swing motion, and the output is video data of this motion.

[0694] Step 2:

[0695] The device saves the acquired video data to local storage and transfers it to a remote server via Bluetooth or Wi-Fi. The input is video data, and the output is data transfer in a format accessible to the server.

[0696] Step 3:

[0697] The server analyzes the received video data. Using computer vision technology based on TensorFlow, it calculates swing speed, angle, and posture. The input is video data, and the output is detailed data on swing characteristics.

[0698] Step 4:

[0699] The server uses APIs such as OpenWeatherMap to obtain real-time weather information and wind direction data. The input is location information sent to the API, and the output is weather information and wind direction information.

[0700] Step 5:

[0701] The server integrates motion analysis results and environmental data. Using prompts, it generates advice tailored to the user's situation into the generated AI model. The input consists of swing characteristic data and environmental data, while the output is specific advice.

[0702] Step 6:

[0703] The server converts advice generated via the Google Text-to-Speech API into speech. The input is the text data of the generated advice, and the output is the speech data.

[0704] Step 7:

[0705] The terminal provides audio data from the server to the user in real time via earphones. The analysis results are also visually displayed on the smart device's screen. Input consists of audio data and analysis result data, while output consists of user-recognizable audio and visual information.

[0706] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0707] This invention provides a system for golfers to receive personalized, real-time advice during actual rounds, and is characterized by its consideration of the user's emotional state. It not only understands the user's motor characteristics and provides advice that takes environmental conditions into account, but also recognizes emotional elements such as stress and satisfaction during play, and enables feedback based on these.

[0708] First, the user films their golf swing using a dedicated smartphone application. The video includes the swing form and the flow of force. This video is temporarily stored on the device, while location information is also recorded simultaneously. Furthermore, audio and facial expression data are also captured and processed by an emotion engine.

[0709] The device sends this data to a server in the cloud. The server uses computer vision technology to analyze the video data and identify the user's movement characteristics. In addition to this movement analysis, it performs voice analysis and facial recognition to determine the user's emotional state. For example, if the tone of voice or facial expression differs from the usual pattern, the emotion engine detects this change.

[0710] Next, the server retrieves weather information and wind direction data from an external database in real time and analyzes this data along with the user's movement characteristics and emotional state. Based on this analysis, the AI ​​generates optimal advice for the user. For example, if the AI ​​determines that the user is feeling stressed, the advice may include suggestions to take deep breaths to relax.

[0711] Finally, the device transcribes the generated advice into text and provides it to the user in real time through a speech synthesis system. Simultaneously, the smartphone screen displays additional visual information, including details of the advice and information tailored to the user's emotional state.

[0712] For example, if a user is unable to perform at their usual level on a windy day, the system will provide feedback such as, "Try to swing through firmly without being defeated by the wind. Relax a little and concentrate on the next shot." In this way, users can efficiently improve their scores while also being able to manage their motivation and emotions.

[0713] The following describes the processing flow.

[0714] Step 1:

[0715] The user launches a dedicated smartphone app at a golf course or driving range and records a video of their swing. The device's camera is used to record the user's swing, while the device's microphone collects audio data and records location information.

[0716] Step 2:

[0717] The device compresses the acquired swing video, audio data, and location information and securely transmits it to the server. This data includes the user's swing characteristics and emotional indicators.

[0718] Step 3:

[0719] The server analyzes video data using computer vision algorithms to identify the user's swing characteristics, form, and force application. Furthermore, it analyzes audio data with an emotion engine to determine the user's emotional state from their tone and word choice.

[0720] Step 4:

[0721] The server retrieves weather and wind direction information in real time from an external database via the internet. This environmental data is then integrated and analyzed with swing characteristics and emotional state.

[0722] Step 5:

[0723] The server utilizes an AI model to generate optimal advice that takes into account swing analysis data, emotional state, and weather information. If the user is feeling stressed, it will also include advice on breathing techniques to calm down and points for concentration.

[0724] Step 6:

[0725] The device receives advice from the server and converts it into voice feedback using a speech synthesis engine. This feedback is presented in real time through the user's earphones, and the screen displays visual feedback and detailed information tailored to the user's emotional state.

[0726] Step 7:

[0727] Based on the audio advice and visual information provided, users can adjust their swing and compose themselves before taking the next shot. This allows for improved play quality and a better golfing experience.

[0728] (Example 2)

[0729] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0730] There is a need to provide appropriate and personalized advice to individual users by evaluating their performance and emotional state during exercise in real time, and by taking environmental conditions into consideration. However, conventional technologies have limited systems that can provide feedback that fully considers the user's emotional state and environmental conditions, resulting in a lack of means to efficiently improve performance. As a result, it has been difficult for users to improve their performance more effectively and engage in activities in a mentally stable state.

[0731] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0732] In this invention, the server includes a camera and location information device that acquires the actions of individual users, an analysis device that analyzes the video information of actions acquired by the camera and extracts the user's action characteristics and emotional state, and an environmental information device that acquires environmental information in real time. This makes it possible to provide users with optimal and personalized real-time advice.

[0733] A "recording device" refers to hardware or software used to record a user's actions in real time, and is responsible for acquiring image or video data.

[0734] A "location information device" is a technology used to determine a user's current location. It is a device that acquires location information in real time using GPS or other location information technologies.

[0735] "Video information" refers to image or video data acquired by a recording device, and serves as basic data for analyzing the user's behavioral characteristics using an analysis device.

[0736] "Analysis device" refers to software or hardware that analyzes the user's behavioral characteristics and emotional state in real time based on data obtained from a camera and a location information device.

[0737] "Motional characteristics" refer to the characteristics and performance tendencies of the user's movements and behaviors, and the analysis device evaluates the data based on these characteristics.

[0738] "Emotional state" refers to the user's mental health and emotional changes, and is a psychological state analyzed from voice and facial expression data.

[0739] An "environmental information device" is a device that acquires environmental information in real time, such as weather, temperature, and wind direction, regarding external conditions when users are engaged in activities.

[0740] A "support generation device" is a device that integrates data obtained from analysis devices and environmental information devices to generate optimal advice for the user.

[0741] A "voice presentation device" refers to a system that converts generated advice into voice and presents it to the user in an easy-to-understand manner, and includes speakers and speech synthesis software.

[0742] This system aims to evaluate users' motor performance and emotional state in real time and provide support based on that evaluation. The specific implementation method is described below.

[0743] First, the user uses a mobile device with a dedicated application installed to record their movements. This device has a built-in camera and location tracking device, and location information is acquired along with the recorded video data. The video records the user's movement form and force flow, and simultaneously captures audio and facial expression data.

[0744] Next, the device sends the acquired data to a server in the cloud. This communication uses a secure protocol to ensure security. The server analyzes the received video data using computer vision technology and utilizes software such as OpenCV and TensorFlow to identify the user's behavioral characteristics. It also analyzes audio and facial expression data to determine the user's emotional state.

[0745] The server calls external APIs to obtain real-time weather and wind direction information from external data sources. It integrates this data and uses a generative AI model to generate optimal advice for the user. This advice includes specific actions to improve athletic performance and suggestions to support emotional management. Accurate feedback is obtained by inputting the analysis results as prompts into the generative AI model.

[0746] Finally, the device uses speech synthesis technology to convert the generated advice into speech and provides it to the user in real time. The device's screen also displays detailed advice and visual information tailored to the user's emotional state. This allows the user to improve their performance and feel more at ease while working.

[0747] For example, the system could provide advice to a user on a windy day such as, "Try to swing through firmly without being defeated by the wind. Relax a little and concentrate on the next shot." Another example of a prompt to input to the generating AI model is, "Please provide advice to improve the user's golf performance, taking into account the stressful situation the user is experiencing."

[0748] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0749] Step 1:

[0750] Users record their actions using a dedicated smartphone app. Input consists of video, audio, and facial expression data obtained through the camera and microphone. Users utilize this data and save it to their device. Specifically, after activating the camera, users perform a series of actions such as a swing or a shot, and the video is recorded on the device.

[0751] Step 2:

[0752] The device acquires location information along with stored video data and sends the data to the server. Inputs include video, audio, and facial expression data stored on the device, as well as acquired location information. These are aggregated and transferred to the server using a secure protocol. Specifically, the device utilizes its GPS sensor and network communication capabilities to package and transmit the data.

[0753] Step 3:

[0754] The server uses computer vision technology to analyze the transmitted video data. The input is video data received from the terminal, and the output is the result of extracting the user's movement characteristics. Specifically, the server uses OpenCV and TensorFlow to analyze joint movements and swing trajectories, and converts the user's movement characteristics into data.

[0755] Step 4:

[0756] The server uses voice analysis and facial recognition technology to determine the user's emotional state. Input is voice and facial expression data, and output is the result of the assessment of the user's emotional state. Specifically, the server evaluates the tone and speed of the voice and changes in facial expression, and based on this, analyzes stress levels and emotional fluctuations.

[0757] Step 5:

[0758] The server calls an external API to obtain environmental data in real time. The input is the endpoint of the environmental data API, and the output is current weather and wind direction information. Specifically, the server generates an API request and retrieves the necessary data in real time.

[0759] Step 6:

[0760] The server uses integrated data (behavioral characteristics, emotional state, and environmental information) to activate a generative AI model and generate optimal advice for the user. The input is the aforementioned data set, and the output is the content of the advice for the user. Specifically, the server inputs this data as prompts into the model and generates real-time feedback for the user.

[0761] Step 7:

[0762] The terminal converts the output from the server into speech using speech synthesis technology and provides it to the user. The input is the text of advice sent from the server, and the output is the audio and visual information presented to the user. Specifically, the terminal generates speech from the text using a speech synthesis engine and displays the related visuals on the display.

[0763] (Application Example 2)

[0764] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0765] This invention aims to solve the problem of a lack of support for golfers to manage their technical performance and mental state simultaneously during a round and achieve optimal play. In particular, while conventional systems can provide more personalized advice by considering not only the user's motor characteristics but also their emotional state, such technology does not yet exist.

[0766] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0767] In this invention, the server includes image information acquisition means, audio information acquisition means and location information acquisition means for acquiring the motor movements and emotional states of individual users, motion and emotion analysis means for analyzing the image data and audio data of the motor movements acquired by the image information acquisition means and audio information acquisition means to extract the user's motion characteristics and emotional characteristics, and environmental information acquisition means for acquiring environmental information in real time. This makes it possible to provide users with real-time optimal advice that takes into account their motion characteristics and emotional states, thereby improving technical performance and promoting mental stability.

[0768] "Individual users" refers to specific human users, and those individuals are users of the system.

[0769] "Motor movement" refers to a series of movements or actions that a person performs when moving their body, and includes movements in specific tasks or sports.

[0770] "Emotional state" refers to the psychological or emotional condition a user is experiencing at a given moment, and includes stress, satisfaction, concentration, etc.

[0771] "Means for acquiring image information" refers to devices or technologies for capturing the movement of a user, and includes cameras and video recorders.

[0772] "Voice information acquisition means" refers to a device or technology for recording voices emitted by a user, and includes microphones and the like.

[0773] "Location information acquisition means" refers to a device or technology for determining the geographical location of a user, including location information services such as GPS.

[0774] "Motion and emotion analysis means" refers to a device or technology for analyzing acquired image data and audio data to identify the user's motor and emotional characteristics.

[0775] "Environmental information acquisition means" refers to devices or technologies for acquiring information about the external environment in real time, and includes sensors and information acquisition services via the Internet.

[0776] An "advice generation means" is a device or technology that generates optimal advice for the user based on analyzed data.

[0777] "Voice output means" refers to a device or technology for converting generated advice into voice and conveying it to the user, and includes speakers and speech synthesizers.

[0778] The system for implementing this invention analyzes the user's motor movements and emotional state in real time and provides personalized advice. First, the user acquires video and audio data of their swing using the camera and microphone built into their smartphone. This data also includes the user's location information.

[0779] Next, the device sends the acquired data to a server in the cloud. The server analyzes the video data using computer vision technology (e.g., OpenCV) to understand the user's movement characteristics. Simultaneously, it analyzes the audio data and facial expressions using emotion recognition technology (e.g., Affectiva) to determine the user's emotional state.

[0780] Furthermore, the server uses environmental information, including weather and wind direction data, obtained from external APIs (e.g., OpenWeatherMap), to integrate the analysis results. Based on these results, a generative AI model generates advice tailored to the user's technical performance and emotional state.

[0781] Finally, the device presents the generated advice to the user verbally using speech synthesis technology (e.g., Google Text-to-Speech). This process allows the user to receive both technical and emotional support in real time.

[0782] For example, if a user's shots are inconsistent in strong winds, the system will generate advice such as, "Concentrate to stabilize your swing against the wind." An example of a prompt would be, "Based on the user's swing data and emotional data, generate optimal golf advice for windy conditions."

[0783] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0784] Step 1:

[0785] The user uses their smartphone to film their golf swing and record audio. During this stage, the camera captures video data, the microphone collects audio, and location information is recorded simultaneously. The inputs are video data, audio data, and location information, which are temporarily stored on the device.

[0786] Step 2:

[0787] The device transmits acquired video data, audio data, and location information to a server in the cloud. This process uses a communication module to package the data and send it to the server securely. The input consists of various data stored on the device, and the output is the completion of data transmission to the server.

[0788] Step 3:

[0789] The server analyzes the video data. Computer vision technology (e.g., OpenCV) is used to extract the swing motion characteristics. At this stage, the input is video data, and the output is user motion characteristic information.

[0790] Step 4:

[0791] The server analyzes audio and facial expression data to determine the user's emotional state. Emotion recognition technology (e.g., Affectiva) identifies stress levels and emotions from voice tone and facial expressions. The input is audio and video data, and the output is information about the user's emotional state.

[0792] Step 5:

[0793] The server uses an external API to obtain real-time weather and wind direction information (e.g., OpenWeatherMap). This allows for the collection of up-to-date information about the play environment. The input is a request to obtain weather and wind direction data, and the output is a set of environmental information.

[0794] Step 6:

[0795] The server integrates operational characteristics, emotional states, and environmental information, and uses a generative AI model to generate optimal advice. This analysis quickly generates situation-appropriate advice for the user. The input is an integrated dataset, and the output is the generated advice.

[0796] Step 7:

[0797] The device outputs the generated advice as speech using speech synthesis technology (e.g., Google Text-to-Speech). Simultaneously, it also displays the advice to the user in text. The input is the generated advice, and the output is the audio and text presentation to the user.

[0798] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the controlled object 443 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0799] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0800] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the robot 414.

[0801] Furthermore, the emotion identification model 59, acting as an emotion engine, may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to a specific mapping, which is an emotion map (see Figure 9). Similarly, the emotion identification model 59 may also determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[0802] Figure 9 shows an emotion map 400 in which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. The closer to the center of the concentric circles, the more primitive the emotions are located. Further out of the concentric circles, emotions representing states and actions arising from mental states are located. Emotion is a concept that includes feelings and mental states. On the left side of the concentric circles, emotions that are generally generated from reactions occurring in the brain are located. On the right side of the concentric circles, emotions that are generally induced by situational judgment are located. Above and below the concentric circles, emotions that are generally generated from reactions occurring in the brain and induced by situational judgment are located. In addition, the emotion of "pleasure" is located on the upper side of the concentric circles, and the emotion of "displeasure" is located on the lower side. Thus, in the emotion map 400, multiple emotions are mapped based on the structure in which emotions arise, and emotions that are likely to occur simultaneously are mapped close together.

[0803] These emotions are distributed at the 3 o'clock position on the Emotion Map 400, and usually fluctuate between feelings of security and anxiety. In the right half of the Emotion Map 400, situational awareness takes precedence over internal feelings, resulting in a calm impression.

[0804] The inside of the Emotion Map 400 represents inner thoughts, while the outside represents actions. Therefore, the further you go from the outside of the Emotion Map 400, the more visible (expressed in actions) your emotions become.

[0805] Here, human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. Similarly, in robots, cars, motorcycles, etc., emotions can be created based on various balances, such as posture and battery level. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. The emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on a system for analyzing brain physiological signals of speech emotion recognition and emotion, Tokushima University, doctoral dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map contains emotions belonging to a region called "response," where sensation is dominant. The right half of the emotion map contains emotions belonging to a region called "situation," where situational awareness is dominant.

[0806] The emotion map defines two emotions that promote learning. One is the emotion around the middle of the negative "repentance" and "reflection" on the situation side. In other words, it is when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is the emotion around the positive "desire" on the reaction side. In other words, it is when the robot has positive feelings such as "I want more" or "I want to know more."

[0807] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​representing each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple training data sets, which are combinations of user input and emotion values ​​representing each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions located close together have similar values, as shown in the emotion map 900 in Figure 10. Figure 10 shows an example where multiple emotions such as "reassured," "calm," and "confident" have similar emotion values.

[0808] The above description primarily focuses on the functions of the data processing device 12 in relation to this disclosure. However, the system related to this disclosure is not necessarily implemented on a server. The system related to this disclosure may be implemented as a general information processing system. This disclosure may be implemented, for example, as a software program that runs on a personal computer or as an application that runs on a smartphone. The method related to this disclosure may be provided to users in SaaS (Software as a Service) format.

[0809] In the above embodiment, an example was given in which a specific process is performed by a single computer 22. However, the technology of this disclosure is not limited thereto, and a distributed processing of the specific process may be performed by multiple computers, including computer 22. For example, a data generation model 58 may be provided in an external device of the data processing device 12, and the external device may generate data according to the input data.

[0810] In the above embodiment, an example was given in which the specific processing program 56 is stored in the storage 32, but the technology of this disclosure is not limited thereto. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-temporary storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-temporary storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes specific processing according to the specific processing program 56.

[0811] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[0812] Furthermore, it is not necessary to store the entirety of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store the entirety of the specific processing program 56 in the storage 32; it is acceptable to store only a portion of the specific processing program 56.

[0813] The following types of processors can be used as hardware resources to perform specific processing. Examples of processors include a CPU, a general-purpose processor that functions as a hardware resource to perform specific processing by executing software, i.e., a program. Other examples of processors include dedicated electrical circuits, such as FPGAs (Field-Programmable Gate Arrays), PLDs (Programmable Logic Devices), or ASICs (Application Specific Integrated Circuits), which have circuit configurations specifically designed to perform specific processing. All of these processors have built-in or connected memory, and all of them perform specific processing by using memory.

[0814] The hardware resource that performs a specific process may consist of one of these various processors, or it may consist of a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Alternatively, the hardware resource that performs a specific process may consist of a single processor.

[0815] Examples of configurations using a single processor include, firstly, a configuration in which one or more CPUs and software are combined to form a single processor, and this processor functions as a hardware resource that performs a specific process. Secondly, there is a configuration using a processor that realizes the functions of the entire system, including multiple hardware resources that perform a specific process, on a single IC chip, as exemplified by SoCs (System-on-a-chip). In this way, a specific process is realized using one or more of the above types of processors as hardware resources.

[0816] Furthermore, the hardware structure of these various processors can more specifically utilize electrical circuits that combine circuit elements such as semiconductor devices. Also, the specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps can be deleted, new steps added, or the processing order rearranged, as long as it does not deviate from the main purpose.

[0817] The descriptions and illustrations presented above are detailed explanations of the technical aspects of this disclosure and are merely examples of the technical aspects. For example, the above descriptions of the structure, function, operation, and effect are examples of the structure, function, operation, and effect of the technical aspects of this disclosure. Therefore, it goes without saying that you may delete unnecessary parts, add new elements, or replace elements in the descriptions and illustrations presented above, as long as you do not deviate from the essence of the technical aspects of this disclosure. Furthermore, in order to avoid confusion and facilitate understanding of the technical aspects of this disclosure, explanations of common technical knowledge and the like that do not require special explanation to enable the implementation of the technical aspects of this disclosure have been omitted from the descriptions and illustrations presented above.

[0818] All documents, patent applications, and technical standards described herein are incorporated by reference to the same extent as if each individual document, patent application, and technical standard were specifically and individually noted to be incorporated by reference.

[0819] The following is further disclosed regarding the embodiments described above.

[0820] (Claim 1)

[0821] To acquire the movement actions of individual users, the system includes image acquisition means and location information acquisition means.

[0822] A motion analysis means analyzes the image data of motion acquired by the image acquisition means and extracts the user's motion characteristics,

[0823] An environmental data acquisition method that obtains environmental information in real time,

[0824] An advice generation means that integrates the data obtained from the motion analysis means and the environmental data acquisition means to generate optimal advice for the user,

[0825] An audio output means that converts the advice generated by the advice generation means into audio and presents it to the user,

[0826] A system that includes this.

[0827] (Claim 2)

[0828] The system according to claim 1, wherein the motion analysis means analyzes the characteristics of the user's motor movements using computer vision technology.

[0829] (Claim 3)

[0830] The system according to claim 1, wherein the environmental data acquisition means acquires weather information and wind direction information from an external information source.

[0831] "Example 1"

[0832] (Claim 1)

[0833] A means for capturing the movement of individual users and a means for acquiring location information,

[0834] A feature extraction means processes motion image information acquired by the aforementioned shooting means and extracts the user's motion characteristics,

[0835] A means of collecting situational data to acquire situational information in real time,

[0836] An advice generation means that integrates the information obtained from the feature extraction means and the situation data collection means and generates optimal advice for the user using a generated AI model,

[0837] A voice presentation means that converts the advice generated by the advice generation means into voice and conveys it to the user,

[0838] A display means for visually displaying the content of the aforementioned advice,

[0839] A system that includes this.

[0840] (Claim 2)

[0841] The system according to claim 1, wherein the feature extraction means analyzes the characteristics of the user's motor movements using advanced visual analysis technology.

[0842] (Claim 3)

[0843] The system according to claim 1, wherein the situation data collection means acquires weather information and wind direction information from an external information source.

[0844] "Application Example 1"

[0845] (Claim 1)

[0846] To acquire the movement actions of individual users, the system includes image acquisition means and location information acquisition means.

[0847] A motion analysis means analyzes the image data of motion acquired by the image acquisition means and extracts the user's motion characteristics,

[0848] An environmental data acquisition method that obtains environmental information in real time,

[0849] An advice generation means that integrates the data obtained from the motion analysis means and the environmental data acquisition means to generate optimal advice for the user,

[0850] An audio output means that converts the advice generated by the advice generation means into audio and presents it to the user,

[0851] A feedback mechanism mounted on the robotic device evaluates the user's movements in real time and provides individualized feedback,

[0852] A system that includes this.

[0853] (Claim 2)

[0854] The motion analysis means analyzes the characteristics of the user's motor movements using computer vision technology.

[0855] The aforementioned feedback means analyzes the user's swing in real time while they are playing golf and provides advice that takes wind direction into consideration.

[0856] The system according to claim 1.

[0857] (Claim 3)

[0858] The aforementioned environmental data acquisition means acquires weather information and wind direction information from an external information source.

[0859] The feedback means provides the user with voiced feedback through the voice output means.

[0860] The system according to claim 1.

[0861] "Example 2 of combining an emotion engine"

[0862] (Claim 1)

[0863] To acquire the actions of individual users, the system includes a camera and a location information device.

[0864] An analysis device that analyzes video information of movements acquired by the aforementioned shooting device and extracts the user's movement characteristics and emotional state,

[0865] An environmental information device that acquires environmental information in real time,

[0866] A support generation device that integrates information obtained from the aforementioned analysis device and environmental information device to generate optimal advice for the user,

[0867] A voice presentation device that converts the advice generated by the aforementioned support generation device into audio and presents it to the user,

[0868] A system that includes this.

[0869] (Claim 2)

[0870] The system according to claim 1, wherein the analysis device uses image processing technology to analyze the user's behavioral characteristics and emotional state.

[0871] (Claim 3)

[0872] The system according to claim 1, wherein the environmental information device acquires weather information and wind direction information from an external information provider.

[0873] "Application example 2 when combining with an emotional engine"

[0874] (Claim 1)

[0875] To acquire the motor movements and emotional states of individual users, the system includes means for acquiring image information, means for acquiring audio information, and means for acquiring location information.

[0876] A motion-emotion analysis means analyzes the motion image data and audio data acquired by the aforementioned image information acquisition means and audio information acquisition means to extract the user's motion characteristics and emotional characteristics.

[0877] An environmental information acquisition method that acquires environmental information in real time,

[0878] An advice generation means that integrates data obtained from the aforementioned behavioral emotion analysis means and environmental information acquisition means to generate optimal advice for the user,

[0879] An audio output means that converts the advice generated by the advice generation means into audio and presents it to the user,

[0880] A system that includes this.

[0881] (Claim 2)

[0882] The system according to claim 1, wherein the motion and emotion analysis means analyzes the user's motor characteristics and emotional state using computer vision technology and emotion recognition technology.

[0883] (Claim 3)

[0884] The system according to claim 1, wherein the environmental information acquisition means acquires weather information and wind direction information from an external information source. [Explanation of symbols]

[0885] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Devices 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robots< / url:> < / url:> < / url:> < / url:>

Claims

1. To acquire the movement actions of individual users, the system includes image acquisition means and location information acquisition means. A motion analysis means analyzes the image data of motion acquired by the image acquisition means and extracts the user's motion characteristics, An environmental data acquisition method that obtains environmental information in real time, An advice generation means that integrates the data obtained from the motion analysis means and the environmental data acquisition means to generate optimal advice for the user, An audio output means that converts the advice generated by the advice generation means into audio and presents it to the user, A system that includes this.

2. The system according to claim 1, wherein the motion analysis means analyzes the characteristics of the user's motor movements using computer vision technology.

3. The system according to claim 1, wherein the environmental data acquisition means acquires weather information and wind direction information from an external information source.

Images