system

Abstract

Provide a system. The device comprises smart glasses, which are wearable devices worn by a user; A means for recording a golf swing in real time using a camera mounted on the smart glasses; a terminal device that receives the recorded video data and performs preprocessing; a server device that analyzes the pre-processed video data and generates swing evaluation and feedback; means for providing the generated feedback visually and audibly to the smart glasses; A system including:

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] In golf, maintaining and improving a correct swing form on one's own is difficult, and beginners and intermediate players in particular often experience slow progress and an unstable swing. Many users also find it difficult to find the time and money to attend golf lessons. Even when self-studying using video materials, each instructor has different approaches, making it difficult to understand which advice is best for them. Conventional technology lacks the means to provide individualized and effective advice regardless of location, limiting self-improvement. [Means for solving the problem]

[0005] To solve the above problems, the present invention provides a system including smart glasses, a wearable device worn by a user; a means for recording a golf swing in real time using a camera mounted on the smart glasses; a terminal device for receiving and preprocessing the recorded video data; and a server device for analyzing the preprocessed video data, which includes a means for generating swing evaluation and feedback, and a means for providing the generated feedback visually and audibly to the smart glasses. This system allows users to receive personalized advice in real time regardless of location and continuously improve their swing. Specifically, the system analyzes the user's swing grip, ball placement, ball trajectory, head lift timing, and body movement, and displays the ideal grip position and ball placement as visual feedback and provides audio feedback suggesting areas for improvement, thereby helping the user improve their swing in real time.

[0006] "Smart glasses" are a wearable device worn by a user, and are eyeglass-type portable terminal devices equipped with a built-in camera, display, and audio output device.

[0007] The "camera" is a device installed in the smart glasses that records video data such as golf swings in real time.

[0008] "Recorded data" refers to video information of a user's golf swing recorded by a camera.

[0009] "Preprocessing" refers to processing of captured video data, such as trimming unnecessary parts, converting the data format, and adding metadata.

[0010] "Terminal device" refers to a device that receives and preprocesses data transferred from smart glasses and sends it to the analysis system, and mainly refers to a mobile information terminal such as a smartphone.

[0011] The "server device" is a device that analyzes preprocessed video data sent from a terminal device and generates feedback based on the analysis results.

[0012] "Analysis" refers to the process of using machine learning algorithms to identify and evaluate swing grip, ball placement, swing speed, ball trajectory, and body movement from video data.

[0013] "Feedback" means visual and / or auditory instructions for evaluation and improvement provided to the user based on the analysis results.

[0014] "Visual feedback" refers to visual information such as grip position and ball placement that is displayed on the smart glasses display.

[0015] "Auditory feedback" refers to instructions for improvement provided through audio messages provided through the audio output device of the smart glasses. [Brief explanation of the drawings]

[0016] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0017] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0018] First, the terms used in the following description will be explained.

[0019] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0020] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0021] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0022] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0023] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0024] [First embodiment]

[0025] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0026] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0027] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0028] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0029] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0030] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0031] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0032] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0033] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0034] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0035] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0036] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0037] The golf play assistance service of the present invention uses smart glasses worn by the user to collect video data of the golf swing in real time, analyzes it with an analysis system, and provides visual and auditory feedback. The system consists of smart glasses, a terminal device (such as a smartphone), and a server device.

[0038] Overall system configuration

[0039] 1. Smart Glasses:

[0040] Equipped with a camera: Records your swing from your perspective.

[0041] Display: Provides visual feedback.

[0042] Audio output device: Provides auditory feedback.

[0043] 2. Terminal device (e.g., smartphone):

[0044] Receiving data: Receives recording data from smart glasses.

[0045] Data preprocessing: trimming and formatting data.

[0046] Communication with the server: Send the preprocessed data to the server.

[0047] 3. Server Device:

[0048] Data Analysis: Analyze each element of your swing to generate an evaluation and feedback.

[0049] Sending feedback: Sending the generated feedback back to the end device.

[0050] Program processing

[0051] 1. Smart Glasses Processing:

[0052] A camera is used to collect video data of a user's golf swing.

[0053] The collected video data is transferred to the terminal device.

[0054] 2. Terminal Processing:

[0055] Receives video data transferred from smart glasses.

[0056] Trim unnecessary parts and convert the data format into one that is easy to analyze.

[0057] The preprocessed data is sent to a server device.

[0058] 3. Processing of server equipment:

[0059] The preprocessed video data transmitted from the terminal device is received.

[0060] Machine learning algorithms are used to analyze swing grip, ball placement, swing speed, ball trajectory, and body movement.

[0061] Based on the analysis results, feedback is generated for each user.

[0062] The feedback content is transmitted to the terminal device.

[0063] 4. Terminal device feedback processing:

[0064] Process the feedback received from the server.

[0065] Sending the processed feedback to the smart glasses.

[0066] 5. Smart Glasses Feedback Processing:

[0067] Visual feedback (ideal grip position and ball placement) is displayed on the screen.

[0068] Auditory feedback (voice messages) is played on an audio output device.

[0069] Specific examples

[0070] Beginner user use cases

[0071] User:

[0072] A beginner user wears smart glasses and performs a golf swing. The camera in the smart glasses records the swing and transmits the data to a smartphone in real time.

[0073] Terminal device:

[0074] The smartphone receives the recorded data, performs pre-processing, and then sends the data to the server. Pre-processing includes trimming unnecessary parts and format conversion.

[0075] Server device:

[0076] The server analyzes the pre-processed data and generates evaluation results such as "weak grip" or "ball placement too left." Based on the analysis results, specific feedback is generated and sent to the smartphone.

[0077] Terminal device:

[0078] The smartphone receives the feedback and sends it to the smart glasses.

[0079] Smart Glasses:

[0080] The smart glasses visually display the correct grip position on the HUD and provide audio advice such as "Hold a firm grip and adjust the ball placement to the right."

[0081] Intermediate User Use Cases

[0082] User:

[0083] An intermediate golfer wears the smart glasses while swinging. The camera records the swing and sends the data to a smartphone.

[0084] Terminal device:

[0085] The smartphone pre-processes the recorded data and sends it to the server.

[0086] Server device:

[0087] The server analyzes the preprocessed data and obtains evaluation results such as "the timing of lifting the head is too early" and "the swing speed is unstable." Feedback based on the evaluation results is generated and sent to the smartphone.

[0088] Terminal device:

[0089] The smartphone receives the feedback and sends it to the smart glasses.

[0090] Smart Glasses:

[0091] The smart glasses visually display the appropriate timing to raise your head on the HUD and provide audio guidance such as "be aware of rhythm to stabilize your swing speed."

[0092] Through these processes, the user can understand in real time what corrections to make to their swing and effectively improve their technique.

[0093] The processing flow will be explained below.

[0094] Step 1:

[0095] The user wears the smart glasses and performs a golf swing, with the camera in the smart glasses recording the user's swing in real time.

[0096] Step 2:

[0097] The swing video data recorded by the smart glasses is transferred to the device via Wi-Fi or Bluetooth.

[0098] Step 3:

[0099] The device saves the video data received from the smart glasses. Once saved, it performs pre-processing such as trimming unnecessary parts and converting the data format.

[0100] Step 4:

[0101] The device adds metadata (time information, environmental conditions, etc.) to the preprocessed video data. After preprocessing and adding metadata, the data is uploaded to the server.

[0102] Step 5:

[0103] The server receives the pre-processed video data and queues it for processing. Machine learning algorithms are used to analyze each element of the swing (grip, ball placement, swing speed, ball trajectory, and body movement).

[0104] Step 6:

[0105] The server generates an analysis result, which may include evaluations such as "weak grip" or "bad ball placement," as well as specific suggestions such as "unstable swing speed" or "head lifting timing too early."

[0106] Step 7:

[0107] Based on the analysis results, the server generates individualized feedback for each user, including visual feedback such as data to display the ideal grip position and ball placement, and audio guidance as auditory feedback.

[0108] Step 8:

[0109] The server transmits the generated feedback data to the terminal.

[0110] Step 9:

[0111] The terminal prepares the received feedback data for transmission to the smart glasses and transfers it to the smart glasses.

[0112] Step 10:

[0113] The smart glasses provide visual feedback (e.g., correct grip position and ball placement) on the display.

[0114] Step 11:

[0115] The smart glasses play auditory feedback (voice messages) through an audio output device, instructing the user, for example, to "tighten the grip and adjust the ball placement to the right."

[0116] Step 12:

[0117] The user can then adjust their swing based on the feedback and reflect it in their next swing. Users can take advantage of real-time feedback to continually improve their swing.

[0118] Through these specific steps, users can analyze their swing in real time and receive effective feedback, allowing them to improve their swing technique efficiently, regardless of location.

[0119] Example 1

[0120] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0121] There is a need for a system that can provide real-time feedback to improve golf swing technique. However, conventional systems lack coordination between the data collection, preprocessing, analysis, and feedback stages, making it difficult to provide effective training. Another issue is that it is difficult for users to specifically identify which elements of their swing need improvement.

[0122] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0123] In this invention, the server includes smart glasses, which are wearable devices worn by a user, a means for recording a golf swing in real time using a camera mounted on the smart glasses, a terminal device for receiving the recorded video data, trimming unnecessary portions, and converting the data format, a server device equipped with a machine learning algorithm for analyzing the preprocessed video data, analyzing each element of the swing, evaluating the swing, and generating feedback, and a means for providing the generated feedback visually and audibly to the smart glasses, thereby enabling the user to receive specific and effective feedback in real time.

[0124] "Smart glasses" are wearable devices worn by users that have visual and auditory interfaces.

[0125] The "camera" is a device installed in the smart glasses that records the golf swing from the user's perspective in real time.

[0126] "Recorded data" refers to video data of golf swings collected by the smart glasses camera.

[0127] The "terminal device" is a device that receives video data sent from the smart glasses, trims unnecessary parts, and converts the data into a format that is easy to analyze.

[0128] "Preprocessing" refers to the process of trimming unnecessary parts of the recorded data and converting the data format into one that is easier to analyze.

[0129] The "server device" is a computer system that analyzes the pre-processed video data, analyzes each element of the swing, and generates an evaluation and feedback.

[0130] A "machine learning algorithm" is a program used by the server device, and is a technology for analyzing factors such as swing grip, ball placement, swing speed, ball trajectory, and body movement.

[0131] "Feedback" refers to advice and evaluation information for each user that is generated based on the analysis results and is provided in visual and auditory formats.

[0132] "Visual feedback" refers to information such as the ideal grip position and ball placement that is displayed on the smart glasses display.

[0133] "Auditory feedback" refers to an audio message played through the audio output device of the smart glasses.

[0134] MODE FOR CARRYING OUT THE INVENTION

[0135] This golf play assistance service uses smart glasses worn by the user to collect video data of the golf swing in real time, analyzes it with an analysis system, and provides visual and auditory feedback. The system consists of smart glasses, a terminal device, and a server device.

[0136] Smart Glasses

[0137] Hardware: The smart glasses are equipped with a camera to record the user's swing from their perspective, and also have a display and audio output to provide visual and audio feedback.

[0138] Specific features:

[0139] A camera is used to collect video data of the user's golf swing and transfer it to a terminal device in real time.

[0140] The collected video data is transferred to a terminal device using wireless communication means such as Wi-Fi.

[0141] The analyzed feedback is displayed visually on a display and audibly reproduced through an audio output device.

[0142] terminal device

[0143] Hardware: Terminal devices are general-purpose devices such as smartphones and tablets.

[0144] Software: An application is installed to preprocess video data and communicate with the server.

[0145] Specific features:

[0146] Receives video data sent from smart glasses.

[0147] The received video data is trimmed and converted into a format suitable for analysis (e.g., MP4 format).

[0148] The pre-processed video data is encrypted and sent to a server via the Internet.

[0149] Server device

[0150] Hardware: A high-performance computer system with the processing power to analyze large amounts of data.

[0151] Software: Includes analysis programs that implement machine learning algorithms (e.g., TENSORFLOW (registered trademark), PyTorch).

[0152] Specific features:

[0153] The preprocessed video data transmitted from the terminal device is received.

[0154] Machine learning algorithms are used to analyze swing grip, ball placement, swing speed, ball trajectory, and body movement.

[0155] Generate evaluation and feedback based on the analysis results.

[0156] The generated feedback is transmitted to the terminal device.

[0157] Specific examples

[0158] Beginner user use cases

[0159] User: A beginner user wears the smart glasses and performs a golf swing. The smart glasses' camera records the swing and transmits the data to a smartphone in real time.

[0160] Terminal device: The smartphone receives the recorded data, performs pre-processing, and then sends the data to the server. Pre-processing includes trimming unnecessary parts and format conversion.

[0161] Server device: The server analyzes the pre-processed data and generates evaluation results such as "the grip is weak" or "the ball is placed too left." Based on the analysis results, specific feedback is generated and sent to the smartphone.

[0162] Terminal device: The smartphone receives the feedback and sends it to the smart glasses.

[0163] Smart glasses: The smart glasses visually show the correct grip position on the display and provide audio advice such as "Hold a firmer grip and adjust the ball placement to the right."

[0164] Prompt Sentence Examples

[0165] "Please explain in natural language the process of analyzing a video of a beginner golfer's swing and providing feedback on areas for improvement."

[0166] In this way, the user can understand in real time what corrections to make to their swing and effectively improve their technique.

[0167] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0168] Step 1: Collecting video data with smart glasses

[0169] User: The user wears the smart glasses and performs a golf swing.

[0170] Input: The user's golf swing motion.

[0171] How it works: The smart glasses' camera records the entire swing from the user's perspective.

[0172] Output: Video data stored inside the smart glasses.

[0173] Step 2: Transferring video data from smart glasses to your device

[0174] Smart glasses: Smart glasses transfer recorded video data to the terminal device.

[0175] Input: Video data stored on smart glasses.

[0176] Specific operation: Smart glasses transmit video data to terminal devices in real time via wireless communication such as Wi-Fi.

[0177] Output: Video data transferred to the terminal device.

[0178] Step 3: Receiving and preprocessing video data on the terminal device

[0179] Terminal: The terminal device receives the video data transmitted from the smart glasses.

[0180] Input: Video data sent from smart glasses.

[0181] Specific operation: Trim unnecessary parts of the video data and convert it into a format that is easy to analyze (e.g., MP4 format).

[0182] Output: Preprocessed video data.

[0183] Step 4: Sending data from the terminal to the server

[0184] Terminal: The terminal device transmits the preprocessed video data to the server device.

[0185] Input: Preprocessed video data.

[0186] Specific operation: The terminal device encrypts data and sends it to the server via the Internet.

[0187] Output: Preprocessed video data transferred to the server.

[0188] Step 5: Video data analysis by the server

[0189] Server: The server device analyzes the received preprocessed video data.

[0190] Input: Preprocessed video data.

[0191] Specific behavior: Machine learning algorithms are used to analyze swing grip, ball placement, swing speed, ball trajectory, and body movement.

[0192] Output: Analysis results and feedback information.

[0193] Step 6: Sending feedback from the server to the terminal device

[0194] Server: The server device sends the generated feedback to the terminal device.

[0195] Input: Analysis results and feedback information.

[0196] Specific operations: The feedback information is formatted and sent to the terminal device via the Internet.

[0197] Output: Feedback information transmitted to the terminal device.

[0198] Step 7: End device processes feedback and sends it to the smart glasses

[0199] Terminal: The terminal device processes the feedback information received from the server and sends it to the smart glasses.

[0200] Input: Feedback information received from the server.

[0201] What it does: Feedback information is optimized as visual and auditory feedback and wirelessly transmitted to the smart glasses.

[0202] Output: Feedback information transmitted to the smart glasses.

[0203] Step 8: Providing feedback with smart glasses

[0204] Smart Glasses: Smart glasses provide the received feedback to the user.

[0205] Input: Feedback information sent from the terminal device.

[0206] What it does: Visual feedback is shown on the display, and auditory feedback is played through the audio output device.

[0207] Output: The feedback presented to the user.

[0208] (Application example 1)

[0209] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0210] Accuracy and efficiency are required for robot operations within factories. In particular, if a transport robot delivers packages incorrectly or operates improperly, it can affect overall production efficiency. The purpose of this invention is to solve these issues by analyzing robot operations within factories in real time and providing rapid feedback.

[0211] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0212] In this invention, the server includes a means for generating an evaluation and feedback for the operation, a means for providing the generated feedback visually and audibly to the smart glasses, and a means for improving the operation of the robot in the factory, thereby making it possible to analyze the operation of the robot in the factory in real time and provide appropriate feedback.

[0213] "User" refers to the person who wears and uses this system.

[0214] "Wearable device" refers to a device that can be worn by a user, in this case smart glasses.

[0215] "Smart glasses" refers to eyeglass-type devices equipped with cameras, displays, etc.

[0216] "Camera" refers to a device that captures images or videos.

[0217] "Movement" refers to the series of movements performed by a robot in a factory.

[0218] "Real-time" refers to data being processed immediately, without delay.

[0219] "Recording" refers to the recording of an action using a camera.

[0220] "Means" refers to a method or apparatus for performing a particular function or process.

[0221] "Video data" refers to video information of recorded actions.

[0222] "Receiving" refers to receiving data.

[0223] "Preprocessing" refers to the process of changing data into a form suitable for analysis.

[0224] "Terminal device" refers to a device that receives and pre-processes data, such as a smartphone.

[0225] "Server device" refers to a device that analyzes data and generates feedback.

[0226] "Evaluation" refers to the analysis results of recorded and analyzed behavior.

[0227] "Feedback" refers to suggestions for improvement or instructions provided based on the evaluation results.

[0228] "Visual" refers to providing information in a form that can be seen by the user, such as through a display.

[0229] "Auditory" refers to providing information to the user in an audible form, such as through voice.

[0230] "Accuracy" refers to the degree to which an action is performed as planned.

[0231] "Efficiency" refers to the degree to which resources are used optimally to obtain maximum results.

[0232] "Pickup position" refers to the correct position where the robot will pick up the luggage.

[0233] "Smoothness of movement" refers to the degree to which a robot can perform movements continuously and without interruption.

[0234] This invention relates to a system that monitors and analyzes the operation of robots in factories in real time and provides feedback. This system consists of smart glasses, which are wearable devices worn by users, a camera mounted on the smart glasses, a terminal device that receives and preprocesses the data, and a server device that analyzes the data.

[0235] System configuration

[0236] 1. Smart Glasses

[0237] The smart glasses, a wearable device worn by the user, are equipped with a camera that can record the robot's movements in real time, and also have a display and audio output device that provide visual and auditory feedback.

[0238] 2. Terminal Device

[0239] The recorded video data is sent to a terminal device (e.g., a smartphone). This terminal device performs preprocessing, trimming unnecessary parts and converting the data format to one that is easier to analyze. The preprocessed data is then sent to a server.

[0240] 3. Server Device

[0241] The server analyzes the received preprocessed video data and evaluates the factory robot's operations. Specifically, it has the ability to analyze accuracy, efficiency, and the smoothness of the pickup position and operation of the goods. This is done using machine learning algorithms such as OpenCV and TensorFlow. Based on the evaluation results, it generates feedback to be provided to the user. This feedback is then sent to the terminal device.

[0242] 4. Providing Feedback

[0243] The terminal device transmits the feedback received from the server to the smart glasses, which provide visual feedback (e.g., displaying the correct action position on the HUD) and auditory feedback (e.g., a voice command saying "Proceed to the next action").

[0244] Specific examples

[0245] For example, suppose a delivery robot operating in a factory frequently mis-delivers packages. The smart glasses worn by the user record the robot's movements in real time, and after preprocessing by a terminal device, send the data to a server. The server analyzes the robot's movement data and obtains the evaluation result that "the package is not being picked up in the correct position." Based on this evaluation result, the smart glasses visually display "correct the position to pick up the package correctly" and provide a voice instruction to "proceed to the next action."

[0246] Prompt Sentence Examples

[0247] Prompt: "Generate code for a robotic behavior analysis system in a factory. The server will be implemented in Python and will use the OpenCV and TensorFlow libraries to analyze the robotic behavior. The smart glasses will use Google® Glass® to provide real-time feedback. The end device will be an ANDROID® smartphone."

[0248] In this way, the present invention can improve robotic operation within a factory, increasing overall production efficiency.

[0249] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0250] Step 1:

[0251] The smart glasses worn by the user use a built-in camera to record the factory robot's operations in real time. In this step, the smart glasses capture the operation video and transfer it to a terminal device as video data. The input is the factory robot's operations, and the output is the recorded video data.

[0252] Step 2:

[0253] The terminal device receives the video data sent from the smart glasses. The terminal device then preprocesses the video data. Specifically, it trims unnecessary parts and converts them into a data format suitable for analysis. The input is the recorded video data, and the output is the preprocessed video data.

[0254] Step 3:

[0255] The terminal device transmits the preprocessed video data to the server device. The server device has a means for analyzing the received video data and evaluates the accuracy, efficiency, package pickup position, and smoothness of the robot's movements from the video data. The input is the preprocessed video data, and the output is the evaluation results.

[0256] Step 4:

[0257] The server device uses machine learning algorithms (e.g., OpenCV and TensorFlow) to evaluate the robot's movements. Specifically, the motion analysis algorithm analyzes the video data and identifies the accuracy of each movement and areas for improvement. The input is the preprocessed video data, and the output is the evaluation results.

[0258] Step 5:

[0259] The server generates feedback based on the evaluation results. Using a generative AI model, it creates specific feedback content to provide to the user. In this case, both visual and auditory feedback is generated. The input is the evaluation results, and the output is the feedback content.

[0260] Step 6:

[0261] The server device sends the generated feedback content to the terminal device. The terminal device sends the received feedback to the smart glasses and presents it to the user in an appropriate manner. At this stage, the input is the feedback content and the output is the feedback transfer to the smart glasses.

[0262] Step 7:

[0263] The smart glasses provide feedback to the user using a display device and an audio output device. Visual feedback shows the correct operating position on the display, and audio feedback indicates the next action. The input is the feedback content sent from the terminal device, and the output is the visual and audio feedback received by the user.

[0264] In this way, a system is created that analyzes the operation of robots in a factory in real time and provides rapid feedback.

[0265] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0266] The present invention uses smart glasses, a wearable device worn by the user, to collect video data of the golf swing, which is then analyzed by an analysis system to provide visual and auditory feedback. Furthermore, by combining this with an emotion engine that recognizes the user's emotional state, more personalized advice can be provided.

[0267] Overall system configuration

[0268] 1. Smart Glasses:

[0269] Equipped with a camera: Records the user's swing in real time.

[0270] Facial expression recognition camera: Captures the user's facial expression data required for the emotion engine.

[0271] Microphone: Collects audio data.

[0272] Biometric sensors: Measure heart rate, skin galvanic response, etc.

[0273] Display: Provides visual feedback.

[0274] Audio output device: Provides auditory feedback.

[0275] 2. Terminal device (e.g., smartphone):

[0276] Receiving video data: Receive video data from the smart glasses.

[0277] Receiving emotional data: Receiving facial expressions, voice, and biometric data.

[0278] Data preprocessing: Trimming and format conversion of video data and emotion data.

[0279] Communication with the server: Send the preprocessed data to the server.

[0280] 3. Server Device:

[0281] Swing Data Analysis: Analyze grip, ball placement, swing speed, ball trajectory, and body movement.

[0282] Emotional data analysis: Recognize the user's emotional state and reflect it in the feedback.

[0283] Feedback generation: Generates user-specific visual and auditory feedback.

[0284] Sending feedback: Sending the generated feedback to the end device.

[0285] Program processing

[0286] Smart Glasses Processing

[0287] The system uses a camera to collect video data of the user's golf swing. At the same time, a facial recognition camera records the user's facial expressions, a microphone collects audio data, and biometric sensors capture biometric data such as heart rate and skin galvanic response.

[0288] The collected data is transferred to a terminal device.

[0289] Terminal processing

[0290] The video data and emotion data received from the smart glasses are saved and unnecessary parts are trimmed.

[0291] The video data is converted into an appropriate format, and metadata (time information, environmental conditions, etc.) is added to the emotion data.

[0292] The preprocessed data is uploaded to a server device.

[0293] Server device processing

[0294] The system analyzes pre-processed video data sent from the terminal device to identify and evaluate the swing grip, ball placement, swing speed, ball trajectory, and body movement.

[0295] Emotional data is analyzed to recognize the user's emotional state, for example, to determine whether the user is stressed or focused.

[0296] Generate feedback based on the analysis results. Dynamically adjust the content and presentation of the feedback depending on the user's emotional state. For example, if the user is feeling stressed, provide advice on how to relax.

[0297] The feedback content is transmitted to the terminal device.

[0298] Terminal device feedback processing

[0299] The feedback received from the server is prepared for transmission to the smart glasses and transferred to the smart glasses.

[0300] Feedback processing for smart glasses

[0301] Visual feedback (e.g., correct grip position and ball placement) is provided on the display.

[0302] Auditory feedback (voice messages) is played on the audio output device, instructing the user to, for example, "Hold a tighter grip and correct the ball placement to the right."

[0303] Specific examples

[0304] Beginner user use cases

[0305] User:

[0306] A beginner wears smart glasses and performs a golf swing. The smart glasses' camera records the swing and facial expressions, the microphone collects audio data, and the biometric sensor records the heart rate.

[0307] Terminal device:

[0308] The smartphone receives the recording data and emotion data, and after preprocessing, sends the data to the server.

[0309] Server device:

[0310] The server analyzes the video data and generates feedback that reflects the user's emotional state, such as "Your grip is weak" or "The ball is placed too left." For example, if the user is feeling stressed, the server provides additional feedback such as "Relax and concentrate on your swing."

[0311] Terminal device:

[0312] The smartphone receives the feedback and sends it to the smart glasses.

[0313] Smart Glasses:

[0314] The smart glasses visually display the correct grip position and provide audio instructions such as "Hold a firmer grip and adjust the ball placement to the right," and if the user feels stressed, they can offer additional advice such as "Relax and focus on your swing."

[0315] Intermediate User Use Cases

[0316] User:

[0317] Intermediate golfers wear smart glasses while swinging. A camera records the swing, captures facial expression data, a microphone collects audio, and a biometric sensor records heart rate.

[0318] Terminal device:

[0319] The smartphone preprocesses the recorded data and emotion data and sends them to the server.

[0320] Server device:

[0321] The server analyzes the data and obtains the evaluation result that "the timing of raising the head is too early." In addition to feedback based on the evaluation result, feedback reflecting the user's emotional state is generated.

[0322] Terminal device:

[0323] The smartphone receives the feedback and sends it to the smart glasses.

[0324] Smart Glasses:

[0325] The smart glasses visually indicate the appropriate time to lift your head and provide audio instructions such as, "Be aware of your rhythm to stabilize your swing speed." If the user is concentrating too much, the glasses provide additional advice such as, "Keep your concentration and keep swinging."

[0326] Through these processes, users can analyze their swing in real time and receive personalized feedback. The introduction of an emotion engine allows for more appropriate advice based on the user's psychological state, enabling efficient and effective improvement of swing technique.

[0327] The processing flow will be explained below.

[0328] Step 1:

[0329] The user wears the smart glasses and performs a golf swing. The smart glasses' camera records the user's swing in real time, the facial recognition camera records the user's facial expressions, biometric sensors measure the user's heart rate and galvanic skin response, and a microphone collects audio.

[0330] Step 2:

[0331] The video data of the swing recorded by the smart glasses, as well as emotional data such as facial expressions, voice, and heart rate, are transferred to the device via Wi-Fi or Bluetooth.

[0332] Step 3:

[0333] The device stores the video data and emotion data received from the smart glasses. The stored data is preprocessed, such as trimming unnecessary parts and converting the data format. The video data is processed first, and then the emotion data is preprocessed.

[0334] Step 4:

[0335] The device adds metadata (time information, environmental conditions, etc.) to the preprocessed video data, and after preprocessing and adding the metadata, uploads this data to the server.

[0336] Step 5:

[0337] The server receives the pre-processed video data and queues it for processing. Machine learning algorithms are used to analyze the swing grip, ball placement, swing speed, ball trajectory, and body movement.

[0338] Step 6:

[0339] The server receives the pre-processed emotion data and uses emotion recognition algorithms to identify the user's emotional state, for example, assessing stress levels and concentration levels based on the user's facial expressions and heart rate.

[0340] Step 7:

[0341] The server generates the analysis results. From the swing data, it makes evaluations such as "weak grip" or "bad ball placement," and from the emotional data, it makes evaluations such as "high stress level" or "concentrated."

[0342] Step 8:

[0343] The server generates personalized feedback for each user based on the analysis results. The content and presentation of the feedback is adjusted according to the user's emotional state. For example, if the user is feeling stressed, the server will provide additional feedback such as "Relax and concentrate on your swing."

[0344] Step 9:

[0345] The server transmits the generated feedback data to the terminal. The feedback data includes data for displaying the ideal grip position and ball placement as visual feedback, and audio guidance as auditory feedback.

[0346] Step 10:

[0347] The terminal receives the feedback data and prepares it for transmission to the smart glasses, after which the feedback data is transferred to the smart glasses.

[0348] Step 11:

[0349] The smart glasses provide visual feedback (e.g., correct grip position and ball placement) on the display, and auditory feedback (voice messages) through a voice output device, instructing the user to "hold the grip tighter and adjust the ball placement to the right." They also provide additional advice based on the user's emotional state.

[0350] Step 12:

[0351] The user can then adjust their swing based on the feedback and reflect it in their next swing. Users can take advantage of real-time feedback to continually improve their swing.

[0352] Through these specific steps, users can receive real-time analysis of their swing and personalized feedback. The introduction of an emotion engine allows for more appropriate advice based on the user's psychological state, enabling efficient and effective improvement of swing technique.

[0353] Example 2

[0354] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0355] Conventional golf swing analysis systems can analyze a user's swing and provide feedback, but they cannot provide personalized feedback that takes the user's emotional state into account. As a result, the advice given ignores the user's psychological state, making it difficult to perform optimal practice. To solve this problem, a new system is required to measure the user's emotional state in real time and provide feedback that reflects this in the analysis.

[0356] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0357] In this invention, the server includes smart glasses, which are wearable devices worn by a user, a means for recording and measuring a golf swing and a user's emotional state in real time using a camera and a sensor mounted on the smart glasses, a terminal device for receiving the recorded video data and emotional data and trimming and formatting the data, a means for analyzing the preprocessed video data and emotional data and generating an evaluation and feedback based on the swing and the user's emotional state, and a means for providing the generated feedback visually and audibly to the smart glasses, thereby enabling personalized feedback that takes the user's emotional state into consideration.

[0358] "Smart glasses" are wearable devices worn by users that have built-in cameras and sensors and collect video data of golf swings and biometric data of the user.

[0359] The "camera" is a photographic device installed in the smart glasses that records the user's golf swing and facial expressions in real time.

[0360] A "sensor" is a measuring device installed in smart glasses that acquires biometric data such as the user's heart rate and skin galvanic response.

[0361] "Video data" refers to video data of a golf swing recorded by the smart glasses camera.

[0362] "Emotional data" refers to data that indicates the user's emotional state, acquired through the smart glasses' facial recognition camera, microphone, and biometric sensors.

[0363] A "terminal device" is a device that receives video data and emotion data sent from smart glasses and performs data preprocessing (trimming and format conversion).

[0364] A "server" is a device that analyzes the pre-processed video data and emotional data and generates evaluations and feedback based on swing and emotional state.

[0365] "Preprocessing" refers to the process of trimming and formatting data performed by the terminal device.

[0366] "Analysis" means the processing of data by the Server for the purposes of evaluating the data and generating feedback.

[0367] "Feedback" refers to visual and audible advice provided to the user based on the evaluation results generated by the server.

[0368] "Visual feedback" refers to visual information displayed on the smart glasses screen that shows corrections to your golf swing and the ideal swing form.

[0369] "Auditory feedback" refers to advice in the form of audio messages played through the audio output device of the smart glasses.

[0370] This invention uses smart glasses, a wearable device worn by the user, to collect video data of the golf swing, analyze it with an analysis system, and provide visual and auditory feedback.Furthermore, by combining it with an emotion engine, it is possible to provide more personalized feedback based on the user's emotional state.

[0371] Overall system configuration

[0372] 1. Smart Glasses:

[0373] Equipped with a camera: Records the user's swing in real time.

[0374] Facial expression recognition camera: Captures the user's facial expression data required for the emotion engine.

[0375] Microphone: Collects audio data.

[0376] Biometric sensors: measure biometric data such as heart rate and galvanic skin response.

[0377] Display: Provides visual feedback.

[0378] Audio output device: Provides auditory feedback.

[0379] 2. Terminal device (e.g., smartphone):

[0380] Receiving video data: Receive video data from the smart glasses.

[0381] Receiving emotional data: Receiving facial expressions, voice, and biometric data.

[0382] Data preprocessing: Trimming and format conversion of video data and emotion data.

[0383] Communication with the server: Send the preprocessed data to the server.

[0384] 3. Server Device:

[0385] Swing Data Analysis: Analyze grip, ball placement, swing speed, ball trajectory, and body movement.

[0386] Emotional data analysis: Recognize the user's emotional state and reflect it in the feedback.

[0387] Feedback generation: Generates user-specific visual and auditory feedback.

[0388] Sending feedback: Sending the generated feedback to the end device.

[0389] Program processing

[0390] Smart Glasses Processing

[0391] The system uses a camera to collect video data of the user's golf swing. At the same time, a facial recognition camera records the user's facial expressions, a microphone collects audio data, and biometric sensors capture biometric data such as heart rate and skin galvanic response.

[0392] The collected data is transferred to a terminal device.

[0393] Terminal processing

[0394] The video data and emotion data received from the smart glasses are saved and unnecessary parts are trimmed.

[0395] The video data is converted into an appropriate format, and metadata (time information, environmental conditions, etc.) is added to the emotion data.

[0396] The preprocessed data is uploaded to a server device.

[0397] Server device processing

[0398] The system analyzes pre-processed video data sent from the terminal device to identify and evaluate the swing grip, ball placement, swing speed, ball trajectory, and body movement.

[0399] Emotional data is analyzed to recognize the user's emotional state, for example, to determine whether the user is stressed or focused.

[0400] Generate feedback based on the analysis results. Dynamically adjust the content and presentation of the feedback depending on the user's emotional state. For example, if the user is feeling stressed, provide advice on how to relax.

[0401] The feedback content is transmitted to the terminal device.

[0402] Terminal device feedback processing

[0403] The feedback received from the server is prepared for transmission to the smart glasses and transferred to the smart glasses.

[0404] Feedback processing for smart glasses

[0405] Visual feedback (e.g., correct grip position and ball placement) is provided on the display.

[0406] Auditory feedback (voice messages) is played on the audio output device, instructing the user to, for example, "Hold a tighter grip and correct the ball placement to the right."

[0407] Specific examples

[0408] Beginner user use cases

[0409] 1. A user wears smart glasses and performs a golf swing. The smart glasses' camera records the swing and facial expressions, the microphone collects audio data, and the biometric sensor records the heart rate.

[0410] 2. The terminal device receives the recorded data and emotion data, preprocesses the data (trimming unnecessary parts, format conversion, adding metadata), and sends it to the server.

[0411] 3. The server analyzes the video data and generates feedback that reflects the user's emotional state, such as "Your grip is weak" or "The ball is placed too left." For example, if the user is feeling stressed, the server provides additional feedback such as "Relax and concentrate on your swing."

[0412] 4. The terminal device receives the feedback and sends it to the smart glasses.

[0413] 5. The smart glasses visually display the correct grip position and provide audio instructions such as "Hold a firmer grip and correct the ball placement to the right." If the user feels stressed, the glasses also provide additional advice such as "Relax and focus on your swing."

[0414] Intermediate User Use Cases

[0415] 1. The user wears the smart glasses while swinging. The camera records the swing, captures facial expression data, collects audio with a microphone, and records heart rate with a biometric sensor.

[0416] 2. The terminal device preprocesses the recorded data and emotion data and sends them to the server.

[0417] 3. The server analyzes the data and obtains the evaluation result that "the timing of raising the head is too early." In addition to feedback based on the evaluation result, feedback reflecting the user's emotional state is generated.

[0418] 4. The terminal device receives the feedback and sends it to the smart glasses.

[0419] 5. The smart glasses visually indicate the appropriate time to raise your head and provide audio instructions such as, "Be aware of your rhythm to stabilize your swing speed." If the user is concentrating too much, the smart glasses provide additional advice such as, "Keep your concentration and keep swinging."

[0420] Prompt Sentence Examples

[0421] Example prompts to be input to the generative AI model:

[0422] Your task is to write a program to operate smart glasses for a golf swing analysis system. The system will collect data by combining cameras, facial recognition, voice recognition, and biometric sensors to analyze the user's swing and emotional state. It will then provide appropriate feedback in real time. Through the process from data collection to feedback provision, we aim to improve the user's swing technique and care for their emotions.

[0423] This prompt statement enables the generative AI model to understand the overall flow of the system and generate the necessary program code.

[0424] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0425] Step 1:

[0426] The user wears smart glasses and performs a golf swing. The smart glasses collect video data of the user's golf swing using a built-in camera. At the same time, the facial recognition camera captures the user's facial expression data, and the microphone collects audio data. The smart glasses also use biometric sensors to capture biometric data such as heart rate and galvanic skin response. This data is stored in the smart glasses' internal memory.

[0427] input:

[0428] User's golf swing

[0429] User's facial expression

[0430] User Voice

[0431] User's biometric information

[0432] output:

[0433] Golf swing video data

[0434] User facial expression data

[0435] User voice data

[0436] User biometric data

[0437] Step 2:

[0438] All data collected by the smart glasses is transferred to the terminal device in real time using wireless communication such as Bluetooth or Wi-Fi.

[0439] input:

[0440] Golf swing video data

[0441] User facial expression data

[0442] User voice data

[0443] User biometric data

[0444] output:

[0445] Transferred video data

[0446] Transferred facial expression data

[0447] Transferred audio data

[0448] Transmitted biometric data

[0449] Step 3:

[0450] The terminal device stores the data received from the smart glasses and trims any unnecessary parts. For example, unnecessary parts before and after a swing are cut out. The video data is also converted into an appropriate format for analysis. Metadata such as time information and environmental conditions are added to the emotion data. After this preprocessing, the data is stored in the terminal device's internal memory.

[0451] input:

[0452] Transferred video data

[0453] Transferred facial expression data

[0454] Transferred audio data

[0455] Transmitted biometric data

[0456] output:

[0457] Video data after trimming

[0458] Format converted video data

[0459] Emotion data with added metadata

[0460] Step 4:

[0461] The terminal device uploads the pre-processed data to the server, which transfers the data via the Internet. Once uploaded, the data is stored on the server for analysis.

[0462] input:

[0463] Video data after trimming

[0464] Format converted video data

[0465] Emotion data with added metadata

[0466] output:

[0467] Preprocessed data stored on the server

[0468] Step 5:

[0469] The server analyzes the pre-processed video data and emotional data. It uses AI algorithms to identify and evaluate swing grip, ball placement, swing speed, ball trajectory, and body movement. It also analyzes the emotional data to recognize the user's emotional state (e.g., stress, concentration).

[0470] input:

[0471] Preprocessed video data

[0472] Emotion data with added metadata

[0473] output:

[0474] Swing data analysis results

[0475] Emotion data analysis results

[0476] Step 6:

[0477] The server generates personalized visual and auditory feedback for the user based on the analysis results. The content and presentation of the feedback are dynamically adjusted according to the user's emotional state. For example, if the user is feeling stressed, advice to help them relax is generated. The generated feedback is stored on the server.

[0478] input:

[0479] Swing data analysis results

[0480] Emotion data analysis results

[0481] output:

[0482] Individual user feedback

[0483] Step 7:

[0484] The server transmits the generated feedback to the terminal device, with data transfer occurring over the Internet.

[0485] input:

[0486] Individual user feedback

[0487] output:

[0488] Feedback sent to the device

[0489] Step 8:

[0490] The terminal device receives the feedback and prepares it for transmission to the smart glasses. The feedback is transferred to the smart glasses via Bluetooth or Wi-Fi.

[0491] input:

[0492] Feedback sent to the device

[0493] output:

[0494] Feedback transmitted to smart glasses

[0495] Step 9:

[0496] The smart glasses display visual feedback (e.g., correct grip position and ball placement) on the display, and also play auditory feedback (e.g., "Hold your grip tighter and correct the ball placement to the right") through an audio output device, allowing the user to receive feedback in real time.

[0497] input:

[0498] Feedback transmitted to smart glasses

[0499] output:

[0500] Visual and auditory feedback provided to the user

[0501] (Application example 2)

[0502] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0503] Currently, many factory robot and worker motion analysis systems have difficulty detecting motion errors and efficiency declines in real time and providing appropriate feedback. Furthermore, few systems consider the psychological state and stress level of workers to improve their motion. Therefore, there is a need for more effective and adaptive feedback systems to improve production efficiency and reduce errors.

[0504] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0505] In this invention, the server includes a means for evaluating and generating feedback on operations, placement, and efficiency, a means for analyzing emotional data and recognizing the stress and concentration of workers, and a means for providing the generated feedback visually and audibly. This makes it possible to analyze the operations of factory robots and workers in real time and provide appropriate feedback. It is also possible to provide more personalized feedback by taking into account the psychological state of the workers.

[0506] A "user" is a person or operator who uses a motion analysis system for golf swings or robotic work.

[0507] A "wearable device" is a device that can be worn by a user and is capable of collecting and displaying data.

[0508] "Smart glasses" are wearable devices equipped with cameras, sensors, displays, etc., that function as an interface for visual and auditory data.

[0509] A "camera" is a device that takes and records images and videos.

[0510] A "golf swing" refers to a golf hitting motion, and is an act for collecting data on a user's swing motion.

[0511] "Real time" refers to processing events as they occur, without any time lag.

[0512] "Recording means" refers to a method or device for recording video data using a camera mounted on smart glasses.

[0513] "Video data" refers to data that displays a series of images recorded by a camera in a time-series sequence.

[0514] "Emotion data" is data that indicates the emotional state of the user based on biological reactions such as the user's heart rate and facial expressions.

[0515] "Preprocessing" refers to the process of processing or converting collected data to make it easier to analyze.

[0516] A "terminal device" is an electronic device capable of data communication and processing, including smartphones and tablets.

[0517] A "server device" is a computer system for storing, managing, and analyzing data.

[0518] "Evaluation" is the act of judging the quality of the results of data analysis based on specific criteria.

[0519] "Feedback" refers to instructions or advice provided to the user based on the evaluation results.

[0520] "Visual feedback" is information provided to a user using a visual medium such as a display.

[0521] "Auditory feedback" refers to spoken messages or instructions provided to a user through an audio output device.

[0522] "Efficiency of action" is a measure of how effectively an action is performed to achieve a goal.

[0523] "Layout" refers to the state of how things or people are arranged.

[0524] An "error rate" is the rate at which errors occur during a particular task or activity.

[0525] "Physical responses" are biometric data that indicate the user's physical changes and condition.

[0526] System Configuration

[0527] Hardware

[0528] 1. Smart Glasses

[0529] camera

[0530] Facial Recognition Camera

[0531] microphone

[0532] Biometric Sensor

[0533] display

[0534] Audio output device

[0535] 2. Terminal Device

[0536] Smartphone or tablet

[0537] 3. Server Device

[0538] Computer system for data analysis

[0539] software

[0540] 1. Video analysis software

[0541] (e.g. OpenCV)

[0542] 2. Facial Recognition Software

[0543] 3. Audio Analysis Software

[0544] 4. Biological Data Analysis Software

[0545] 5. Emotion Engine

[0546] 6. Feedback Generator

[0547] Overall system processing

[0548] 1. Data Collection and Preprocessing

[0549] Smart Glasses:

[0550] The camera installed in the smart glasses captures the user's work and movements in real time.

[0551] A facial expression recognition camera captures the user's facial expression data, and a microphone collects voice data.

[0552] Biosensors are used to measure biometric data such as heart rate and skin galvanic response.

[0553] Terminal device:

[0554] Receive video data and emotion data transmitted from the smart glasses.

[0555] Trim out any unwanted parts and convert to the appropriate format.

[0556] The preprocessed data is transferred to a server device.

[0557] 2. Data analysis and feedback generation

[0558] Server device:

[0559] Using video analysis software, we analyze the efficiency, placement, and error rate of movements.

[0560] Using facial expression recognition software and biometric data analysis software, the system analyzes the user's emotional state, thereby identifying the user's stress and concentration level.

[0561] The emotional engine generates feedback based on the results of motion analysis, dynamically adjusting the visual and auditory feedback according to the user's emotional state.

[0562] Generate feedback:

[0563] Based on the analysis results, feedback is generated that provides visual and auditory indications of work improvements and points to note.

[0564] The server transmits the feedback content to the terminal device.

[0565] 3. Providing Feedback

[0566] Terminal device:

[0567] The received feedback is forwarded to the smart glasses.

[0568] Smart Glasses:

[0569] Visual feedback is provided on the display, such as ideal operating positions and areas requiring attention.

[0570] The auditory feedback is played on an audio output device to instruct the user as a voice message.

[0571] Specific examples

[0572] Use cases for beginner users:

[0573] A user wears smart glasses and works on a factory production line. The smart glasses' camera captures their movements and facial expressions, the microphone collects audio data, and the biometric sensor records their heart rate. The terminal device preprocesses this data and sends it to a server. Based on the analysis results, the server provides visual and auditory feedback on areas for improvement and important points to note.

[0574] Example prompt sentence:

[0575] "Analyze the robot's movements during work and generate feedback to maximize efficiency. Also, recognize the worker's stress and concentration levels and provide advice based on that."

[0576] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0577] Step 1:

[0578] Data collection using smart glasses

[0579] Input: User movements, facial expressions, voice data, biometric data (heart rate, skin galvanic response)

[0580] How it works: A camera on the smart glasses captures the user's movements in real time, a facial recognition camera records the user's facial expressions, a microphone collects audio data, and biometric sensors measure heart rate and galvanic skin response.

[0581] Output: Acquisition of video data, facial expression data, audio data, and biometric data

[0582] Step 2:

[0583] Data reception and preprocessing by the terminal device

[0584] Input: Video data, facial expression data, audio data, and biometric data sent from smart glasses

[0585] Specific operation: The terminal device (smartphone or tablet) receives the data sent from the smart glasses, trims unnecessary parts, converts the video data into an appropriate format, and adds metadata (time information, environmental conditions, etc.) to the emotion data.

[0586] Output: Preprocessed video data and emotion data

[0587] Step 3:

[0588] Data transmission from terminal device to server

[0589] Input: Preprocessed video data and emotion data

[0590] Specific operations: The terminal device uploads the preprocessed data to the server device.

[0591] Output: Preprocessed data sent to the server

[0592] Step 4:

[0593] Data analysis by server

[0594] Input: Preprocessed data sent from the terminal

[0595] Specific operations: The server analyzes the video data and evaluates the efficiency, placement, and error rate of the movements. It also uses facial expression recognition software and biometric data analysis software to analyze the user's emotional data and recognize their stress and concentration levels.

[0596] Output: Movement evaluation results and emotional state analysis results

[0597] Step 5:

[0598] Feedback Generation

[0599] Input: Movement evaluation results and emotional state analysis results

[0600] Specific behavior: The server utilizes an emotion engine to generate feedback based on the behavior analysis results and the user's emotional state. The visual and auditory feedback content is dynamically adjusted.

[0601] Output: Visual and auditory feedback

[0602] Step 6:

[0603] Sending feedback to terminal devices

[0604] Input: Server-generated feedback

[0605] Specific operations: The server device transmits the generated feedback to the terminal device.

[0606] Output: Feedback sent to the terminal

[0607] Step 7:

[0608] Sending feedback from the device to the smart glasses

[0609] Input: Feedback sent to the terminal

[0610] Specific operation: The terminal device transmits the feedback content to the smart glasses.

[0611] Output: Feedback sent to the smart glasses

[0612] Step 8:

[0613] Feedback provided by smart glasses

[0614] Input: Feedback sent to the smart glasses

[0615] Specific operation: Smart glasses display visual feedback on the display and play auditory feedback through the audio output device. For example, instructions for correcting movement placement or advice on preventing mistakes are provided via both visual and audio feedback.

[0616] Output: The visual and auditory feedback the user receives.

[0617] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0618] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search engine).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0619] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0620] [Second embodiment]

[0621] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0622] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0623] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0624] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0625] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0626] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0627] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0628] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0629] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0630] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0631] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0632] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0633] The golf play assistance service of the present invention uses smart glasses worn by the user to collect video data of the golf swing in real time, analyzes it with an analysis system, and provides visual and auditory feedback. The system consists of smart glasses, a terminal device (such as a smartphone), and a server device.

[0634] Overall system configuration

[0635] 1. Smart Glasses:

[0636] Equipped with a camera: Records your swing from your perspective.

[0637] Display: Provides visual feedback.

[0638] Audio output device: Provides auditory feedback.

[0639] 2. Terminal device (e.g., smartphone):

[0640] Receiving data: Receives recording data from smart glasses.

[0641] Data preprocessing: trimming and formatting data.

[0642] Communication with the server: Send the preprocessed data to the server.

[0643] 3. Server Device:

[0644] Data Analysis: Analyze each element of your swing to generate an evaluation and feedback.

[0645] Sending feedback: Sending the generated feedback back to the end device.

[0646] Program processing

[0647] 1. Smart Glasses Processing:

[0648] A camera is used to collect video data of a user's golf swing.

[0649] The collected video data is transferred to the terminal device.

[0650] 2. Terminal Processing:

[0651] Receives video data transferred from smart glasses.

[0652] Trim unnecessary parts and convert the data format into one that is easy to analyze.

[0653] The preprocessed data is sent to a server device.

[0654] 3. Processing of server equipment:

[0655] The preprocessed video data transmitted from the terminal device is received.

[0656] Machine learning algorithms are used to analyze swing grip, ball placement, swing speed, ball trajectory, and body movement.

[0657] Based on the analysis results, feedback is generated for each user.

[0658] The feedback content is transmitted to the terminal device.

[0659] 4. Terminal device feedback processing:

[0660] Process the feedback received from the server.

[0661] Sending the processed feedback to the smart glasses.

[0662] 5. Smart Glasses Feedback Processing:

[0663] Visual feedback (ideal grip position and ball placement) is displayed on the screen.

[0664] Auditory feedback (voice messages) is played on an audio output device.

[0665] Specific examples

[0666] Beginner user use cases

[0667] User:

[0668] A beginner user wears smart glasses and performs a golf swing. The camera in the smart glasses records the swing and transmits the data to a smartphone in real time.

[0669] Terminal device:

[0670] The smartphone receives the recorded data, performs pre-processing, and then sends the data to the server. Pre-processing includes trimming unnecessary parts and format conversion.

[0671] Server device:

[0672] The server analyzes the pre-processed data and generates evaluation results such as "weak grip" or "ball placement too left." Based on the analysis results, specific feedback is generated and sent to the smartphone.

[0673] Terminal device:

[0674] The smartphone receives the feedback and sends it to the smart glasses.

[0675] Smart Glasses:

[0676] The smart glasses visually display the correct grip position on the HUD and provide audio advice such as "Hold a firm grip and adjust the ball placement to the right."

[0677] Intermediate User Use Cases

[0678] User:

[0679] An intermediate golfer wears the smart glasses while swinging. The camera records the swing and sends the data to a smartphone.

[0680] Terminal device:

[0681] The smartphone pre-processes the recorded data and sends it to the server.

[0682] Server device:

[0683] The server analyzes the preprocessed data and obtains evaluation results such as "the timing of lifting the head is too early" and "the swing speed is unstable." Feedback based on the evaluation results is generated and sent to the smartphone.

[0684] Terminal device:

[0685] The smartphone receives the feedback and sends it to the smart glasses.

[0686] Smart Glasses:

[0687] The smart glasses visually display the appropriate timing to raise your head on the HUD and provide audio guidance such as "be aware of rhythm to stabilize your swing speed."

[0688] Through these processes, the user can understand in real time what corrections to make to their swing and effectively improve their technique.

[0689] The processing flow will be explained below.

[0690] Step 1:

[0691] The user wears the smart glasses and performs a golf swing, with the camera in the smart glasses recording the user's swing in real time.

[0692] Step 2:

[0693] The swing video data recorded by the smart glasses is transferred to the device via Wi-Fi or Bluetooth.

[0694] Step 3:

[0695] The device saves the video data received from the smart glasses. Once saved, it performs pre-processing such as trimming unnecessary parts and converting the data format.

[0696] Step 4:

[0697] The device adds metadata (time information, environmental conditions, etc.) to the preprocessed video data. After preprocessing and adding metadata, the data is uploaded to the server.

[0698] Step 5:

[0699] The server receives the pre-processed video data and queues it for processing. Machine learning algorithms are used to analyze each element of the swing (grip, ball placement, swing speed, ball trajectory, and body movement).

[0700] Step 6:

[0701] The server generates an analysis result, which may include evaluations such as "weak grip" or "bad ball placement," as well as specific suggestions such as "unstable swing speed" or "head lifting timing too early."

[0702] Step 7:

[0703] Based on the analysis results, the server generates individualized feedback for each user, including visual feedback such as data to display the ideal grip position and ball placement, and audio guidance as auditory feedback.

[0704] Step 8:

[0705] The server transmits the generated feedback data to the terminal.

[0706] Step 9:

[0707] The terminal prepares the received feedback data for transmission to the smart glasses and transfers it to the smart glasses.

[0708] Step 10:

[0709] The smart glasses provide visual feedback (e.g., correct grip position and ball placement) on the display.

[0710] Step 11:

[0711] The smart glasses play auditory feedback (voice messages) through an audio output device, instructing the user, for example, to "tighten the grip and adjust the ball placement to the right."

[0712] Step 12:

[0713] The user can then adjust their swing based on the feedback and reflect it in their next swing. Users can take advantage of real-time feedback to continually improve their swing.

[0714] Through these specific steps, users can analyze their swing in real time and receive effective feedback, allowing them to improve their swing technique efficiently, regardless of location.

[0715] Example 1

[0716] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0717] There is a need for a system that can provide real-time feedback to improve golf swing technique. However, conventional systems lack coordination between the data collection, preprocessing, analysis, and feedback stages, making it difficult to provide effective training. Another issue is that it is difficult for users to specifically identify which elements of their swing need improvement.

[0718] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0719] In this invention, the server includes smart glasses, which are wearable devices worn by a user, a means for recording a golf swing in real time using a camera mounted on the smart glasses, a terminal device for receiving the recorded video data, trimming unnecessary portions, and converting the data format, a server device equipped with a machine learning algorithm for analyzing the preprocessed video data, analyzing each element of the swing, evaluating the swing, and generating feedback, and a means for providing the generated feedback visually and audibly to the smart glasses, thereby enabling the user to receive specific and effective feedback in real time.

[0720] "Smart glasses" are wearable devices worn by users that have visual and auditory interfaces.

[0721] The "camera" is a device installed in the smart glasses that records the golf swing from the user's perspective in real time.

[0722] "Recorded data" refers to video data of golf swings collected by the smart glasses camera.

[0723] The "terminal device" is a device that receives video data sent from the smart glasses, trims unnecessary parts, and converts the data into a format that is easy to analyze.

[0724] "Preprocessing" refers to the process of trimming unnecessary parts of the recorded data and converting the data format into one that is easier to analyze.

[0725] The "server device" is a computer system that analyzes the pre-processed video data, analyzes each element of the swing, and generates an evaluation and feedback.

[0726] A "machine learning algorithm" is a program used by the server device, and is a technology for analyzing factors such as swing grip, ball placement, swing speed, ball trajectory, and body movement.

[0727] "Feedback" refers to advice and evaluation information for each user that is generated based on the analysis results and is provided in visual and auditory formats.

[0728] "Visual feedback" refers to information such as the ideal grip position and ball placement that is displayed on the smart glasses display.

[0729] "Auditory feedback" refers to an audio message played through the audio output device of the smart glasses.

[0730] MODE FOR CARRYING OUT THE INVENTION

[0731] This golf play assistance service uses smart glasses worn by the user to collect video data of the golf swing in real time, analyzes it with an analysis system, and provides visual and auditory feedback. The system consists of smart glasses, a terminal device, and a server device.

[0732] Smart Glasses

[0733] Hardware: The smart glasses are equipped with a camera to record the user's swing from their perspective, and also have a display and audio output to provide visual and audio feedback.

[0734] Specific features:

[0735] A camera is used to collect video data of the user's golf swing and transfer it to a terminal device in real time.

[0736] The collected video data is transferred to a terminal device using wireless communication means such as Wi-Fi.

[0737] The analyzed feedback is displayed visually on a display and audibly reproduced through an audio output device.

[0738] terminal device

[0739] Hardware: Terminal devices are general-purpose devices such as smartphones and tablets.

[0740] Software: An application is installed to preprocess video data and communicate with the server.

[0741] Specific features:

[0742] Receives video data sent from smart glasses.

[0743] The received video data is trimmed and converted into a format suitable for analysis (e.g., MP4 format).

[0744] The pre-processed video data is encrypted and sent to a server via the Internet.

[0745] Server device

[0746] Hardware: A high-performance computer system with the processing power to analyze large amounts of data.

[0747] Software: Includes analysis programs that implement machine learning algorithms (e.g., TensorFlow, PyTorch).

[0748] Specific features:

[0749] The preprocessed video data transmitted from the terminal device is received.

[0750] Machine learning algorithms are used to analyze swing grip, ball placement, swing speed, ball trajectory, and body movement.

[0751] Generate evaluation and feedback based on the analysis results.

[0752] The generated feedback is transmitted to the terminal device.

[0753] Specific examples

[0754] Beginner user use cases

[0755] User: A beginner user wears the smart glasses and performs a golf swing. The smart glasses' camera records the swing and transmits the data to a smartphone in real time.

[0756] Terminal device: The smartphone receives the recorded data, performs pre-processing, and then sends the data to the server. Pre-processing includes trimming unnecessary parts and format conversion.

[0757] Server device: The server analyzes the pre-processed data and generates evaluation results such as "the grip is weak" or "the ball is placed too left." Based on the analysis results, specific feedback is generated and sent to the smartphone.

[0758] Terminal device: The smartphone receives the feedback and sends it to the smart glasses.

[0759] Smart glasses: The smart glasses visually show the correct grip position on the display and provide audio advice such as "Hold a firmer grip and adjust the ball placement to the right."

[0760] Prompt Sentence Examples

[0761] "Please explain in natural language the process of analyzing a video of a beginner golfer's swing and providing feedback on areas for improvement."

[0762] In this way, the user can understand in real time what corrections to make to their swing and effectively improve their technique.

[0763] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0764] Step 1: Collecting video data with smart glasses

[0765] User: The user wears the smart glasses and performs a golf swing.

[0766] Input: The user's golf swing motion.

[0767] How it works: The smart glasses' camera records the entire swing from the user's perspective.

[0768] Output: Video data stored inside the smart glasses.

[0769] Step 2: Transferring video data from smart glasses to your device

[0770] Smart glasses: Smart glasses transfer recorded video data to the terminal device.

[0771] Input: Video data stored on smart glasses.

[0772] Specific operation: Smart glasses transmit video data to terminal devices in real time via wireless communication such as Wi-Fi.

[0773] Output: Video data transferred to the terminal device.

[0774] Step 3: Receiving and preprocessing video data on the terminal device

[0775] Terminal: The terminal device receives the video data transmitted from the smart glasses.

[0776] Input: Video data sent from smart glasses.

[0777] Specific operation: Trim unnecessary parts of the video data and convert it into a format that is easy to analyze (e.g., MP4 format).

[0778] Output: Preprocessed video data.

[0779] Step 4: Sending data from the terminal to the server

[0780] Terminal: The terminal device transmits the preprocessed video data to the server device.

[0781] Input: Preprocessed video data.

[0782] Specific operation: The terminal device encrypts data and sends it to the server via the Internet.

[0783] Output: Preprocessed video data transferred to the server.

[0784] Step 5: Video data analysis by the server

[0785] Server: The server device analyzes the received preprocessed video data.

[0786] Input: Preprocessed video data.

[0787] Specific behavior: Machine learning algorithms are used to analyze swing grip, ball placement, swing speed, ball trajectory, and body movement.

[0788] Output: Analysis results and feedback information.

[0789] Step 6: Sending feedback from the server to the terminal device

[0790] Server: The server device sends the generated feedback to the terminal device.

[0791] Input: Analysis results and feedback information.

[0792] Specific operations: The feedback information is formatted and sent to the terminal device via the Internet.

[0793] Output: Feedback information transmitted to the terminal device.

[0794] Step 7: End device processes feedback and sends it to the smart glasses

[0795] Terminal: The terminal device processes the feedback information received from the server and sends it to the smart glasses.

[0796] Input: Feedback information received from the server.

[0797] What it does: Feedback information is optimized as visual and auditory feedback and wirelessly transmitted to the smart glasses.

[0798] Output: Feedback information transmitted to the smart glasses.

[0799] Step 8: Providing feedback with smart glasses

[0800] Smart Glasses: Smart glasses provide the received feedback to the user.

[0801] Input: Feedback information sent from the terminal device.

[0802] What it does: Visual feedback is shown on the display, and auditory feedback is played through the audio output device.

[0803] Output: The feedback presented to the user.

[0804] (Application example 1)

[0805] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0806] Accuracy and efficiency are required for robot operations within factories. In particular, if a transport robot delivers packages incorrectly or operates improperly, it can affect overall production efficiency. The purpose of this invention is to solve these issues by analyzing robot operations within factories in real time and providing rapid feedback.

[0807] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0808] In this invention, the server includes a means for generating an evaluation and feedback for the operation, a means for providing the generated feedback visually and audibly to the smart glasses, and a means for improving the operation of the robot in the factory, thereby making it possible to analyze the operation of the robot in the factory in real time and provide appropriate feedback.

[0809] "User" refers to the person who wears and uses this system.

[0810] "Wearable device" refers to a device that can be worn by a user, in this case smart glasses.

[0811] "Smart glasses" refers to eyeglass-type devices equipped with cameras, displays, etc.

[0812] "Camera" refers to a device that captures images or videos.

[0813] "Movement" refers to the series of movements performed by a robot in a factory.

[0814] "Real-time" refers to data being processed immediately, without delay.

[0815] "Recording" refers to the recording of an action using a camera.

[0816] "Means" refers to a method or apparatus for performing a particular function or process.

[0817] "Video data" refers to video information of recorded actions.

[0818] "Receiving" refers to receiving data.

[0819] "Preprocessing" refers to the process of changing data into a form suitable for analysis.

[0820] "Terminal device" refers to a device that receives and pre-processes data, such as a smartphone.

[0821] "Server device" refers to a device that analyzes data and generates feedback.

[0822] "Evaluation" refers to the analysis results of recorded and analyzed behavior.

[0823] "Feedback" refers to suggestions for improvement or instructions provided based on the evaluation results.

[0824] "Visual" refers to providing information in a form that can be seen by the user, such as through a display.

[0825] "Auditory" refers to providing information to the user in an audible form, such as through voice.

[0826] "Accuracy" refers to the degree to which an action is performed as planned.

[0827] "Efficiency" refers to the degree to which resources are used optimally to obtain maximum results.

[0828] "Pickup position" refers to the correct position where the robot will pick up the luggage.

[0829] "Smoothness of movement" refers to the degree to which a robot can perform movements continuously and without interruption.

[0830] This invention relates to a system that monitors and analyzes the operation of robots in factories in real time and provides feedback. This system consists of smart glasses, which are wearable devices worn by users, a camera mounted on the smart glasses, a terminal device that receives and preprocesses the data, and a server device that analyzes the data.

[0831] System configuration

[0832] 1. Smart Glasses

[0833] The smart glasses, a wearable device worn by the user, are equipped with a camera that can record the robot's movements in real time, and also have a display and audio output device that provide visual and auditory feedback.

[0834] 2. Terminal Device

[0835] The recorded video data is sent to a terminal device (e.g., a smartphone). This terminal device performs preprocessing, trimming unnecessary parts and converting the data format to one that is easier to analyze. The preprocessed data is then sent to a server.

[0836] 3. Server Device

[0837] The server analyzes the received preprocessed video data and evaluates the factory robot's operations. Specifically, it has the ability to analyze accuracy, efficiency, and the smoothness of the pickup position and operation of the goods. This is done using machine learning algorithms such as OpenCV and TensorFlow. Based on the evaluation results, it generates feedback to be provided to the user. This feedback is then sent to the terminal device.

[0838] 4. Providing Feedback

[0839] The terminal device transmits the feedback received from the server to the smart glasses, which provide visual feedback (e.g., displaying the correct action position on the HUD) and auditory feedback (e.g., a voice command saying "Proceed to the next action").

[0840] Specific examples

[0841] For example, suppose a delivery robot operating in a factory frequently mis-delivers packages. The smart glasses worn by the user record the robot's movements in real time, and after preprocessing by a terminal device, send the data to a server. The server analyzes the robot's movement data and obtains the evaluation result that "the package is not being picked up in the correct position." Based on this evaluation result, the smart glasses visually display "correct the position to pick up the package correctly" and provide a voice instruction to "proceed to the next action."

[0842] Prompt Sentence Examples

[0843] Prompt: "Generate code for a robotic behavior analysis system in a factory. The server will be implemented in Python and will use the OpenCV and TensorFlow libraries to analyze the robotic behavior. The smart glasses will use Google Glass to provide real-time feedback. The end device will be an Android smartphone."

[0844] In this way, the present invention can improve robotic operation within a factory, increasing overall production efficiency.

[0845] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0846] Step 1:

[0847] The smart glasses worn by the user use a built-in camera to record the factory robot's operations in real time. In this step, the smart glasses capture the operation video and transfer it to a terminal device as video data. The input is the factory robot's operations, and the output is the recorded video data.

[0848] Step 2:

[0849] The terminal device receives the video data sent from the smart glasses. The terminal device then preprocesses the video data. Specifically, it trims unnecessary parts and converts them into a data format suitable for analysis. The input is the recorded video data, and the output is the preprocessed video data.

[0850] Step 3:

[0851] The terminal device transmits the preprocessed video data to the server device. The server device has a means for analyzing the received video data and evaluates the accuracy, efficiency, package pickup position, and smoothness of the robot's movements from the video data. The input is the preprocessed video data, and the output is the evaluation results.

[0852] Step 4:

[0853] The server device uses machine learning algorithms (e.g., OpenCV and TensorFlow) to evaluate the robot's movements. Specifically, the motion analysis algorithm analyzes the video data and identifies the accuracy of each movement and areas for improvement. The input is the preprocessed video data, and the output is the evaluation results.

[0854] Step 5:

[0855] The server generates feedback based on the evaluation results. Using a generative AI model, it creates specific feedback content to provide to the user. In this case, both visual and auditory feedback is generated. The input is the evaluation results, and the output is the feedback content.

[0856] Step 6:

[0857] The server device sends the generated feedback content to the terminal device. The terminal device sends the received feedback to the smart glasses and presents it to the user in an appropriate manner. At this stage, the input is the feedback content and the output is the feedback transfer to the smart glasses.

[0858] Step 7:

[0859] The smart glasses provide feedback to the user using a display device and an audio output device. Visual feedback shows the correct operating position on the display, and audio feedback indicates the next action. The input is the feedback content sent from the terminal device, and the output is the visual and audio feedback received by the user.

[0860] In this way, a system is created that analyzes the operation of robots in a factory in real time and provides rapid feedback.

[0861] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0862] The present invention uses smart glasses, a wearable device worn by the user, to collect video data of the golf swing, which is then analyzed by an analysis system to provide visual and auditory feedback. Furthermore, by combining this with an emotion engine that recognizes the user's emotional state, more personalized advice can be provided.

[0863] Overall system configuration

[0864] 1. Smart Glasses:

[0865] Equipped with a camera: Records the user's swing in real time.

[0866] Facial expression recognition camera: Captures the user's facial expression data required for the emotion engine.

[0867] Microphone: Collects audio data.

[0868] Biometric sensors: Measure heart rate, skin galvanic response, etc.

[0869] Display: Provides visual feedback.

[0870] Audio output device: Provides auditory feedback.

[0871] 2. Terminal device (e.g., smartphone):

[0872] Receiving video data: Receive video data from the smart glasses.

[0873] Receiving emotional data: Receiving facial expressions, voice, and biometric data.

[0874] Data preprocessing: Trimming and format conversion of video data and emotion data.

[0875] Communication with the server: Send the preprocessed data to the server.

[0876] 3. Server Device:

[0877] Swing Data Analysis: Analyze grip, ball placement, swing speed, ball trajectory, and body movement.

[0878] Emotional data analysis: Recognize the user's emotional state and reflect it in the feedback.

[0879] Feedback generation: Generates user-specific visual and auditory feedback.

[0880] Sending feedback: Sending the generated feedback to the end device.

[0881] Program processing

[0882] Smart Glasses Processing

[0883] The system uses a camera to collect video data of the user's golf swing. At the same time, a facial recognition camera records the user's facial expressions, a microphone collects audio data, and biometric sensors capture biometric data such as heart rate and skin galvanic response.

[0884] The collected data is transferred to a terminal device.

[0885] Terminal processing

[0886] The video data and emotion data received from the smart glasses are saved and unnecessary parts are trimmed.

[0887] The video data is converted into an appropriate format, and metadata (time information, environmental conditions, etc.) is added to the emotion data.

[0888] The preprocessed data is uploaded to a server device.

[0889] Server device processing

[0890] The system analyzes pre-processed video data sent from the terminal device to identify and evaluate the swing grip, ball placement, swing speed, ball trajectory, and body movement.

[0891] Emotional data is analyzed to recognize the user's emotional state, for example, to determine whether the user is stressed or focused.

[0892] Generate feedback based on the analysis results. Dynamically adjust the content and presentation of the feedback depending on the user's emotional state. For example, if the user is feeling stressed, provide advice on how to relax.

[0893] The feedback content is transmitted to the terminal device.

[0894] Terminal device feedback processing

[0895] The feedback received from the server is prepared for transmission to the smart glasses and transferred to the smart glasses.

[0896] Feedback processing for smart glasses

[0897] Visual feedback (e.g., correct grip position and ball placement) is provided on the display.

[0898] Auditory feedback (voice messages) is played on the audio output device, instructing the user to, for example, "Hold a tighter grip and correct the ball placement to the right."

[0899] Specific examples

[0900] Beginner user use cases

[0901] User:

[0902] A beginner wears smart glasses and performs a golf swing. The smart glasses' camera records the swing and facial expressions, the microphone collects audio data, and the biometric sensor records the heart rate.

[0903] Terminal device:

[0904] The smartphone receives the recording data and emotion data, and after preprocessing, sends the data to the server.

[0905] Server device:

[0906] The server analyzes the video data and generates feedback that reflects the user's emotional state, such as "Your grip is weak" or "The ball is placed too left." For example, if the user is feeling stressed, the server provides additional feedback such as "Relax and concentrate on your swing."

[0907] Terminal device:

[0908] The smartphone receives the feedback and sends it to the smart glasses.

[0909] Smart Glasses:

[0910] The smart glasses visually display the correct grip position and provide audio instructions such as "Hold a firmer grip and adjust the ball placement to the right," and if the user feels stressed, they can offer additional advice such as "Relax and focus on your swing."

[0911] Intermediate User Use Cases

[0912] User:

[0913] Intermediate golfers wear smart glasses while swinging. A camera records the swing, captures facial expression data, a microphone collects audio, and a biometric sensor records heart rate.

[0914] Terminal device:

[0915] The smartphone preprocesses the recorded data and emotion data and sends them to the server.

[0916] Server device:

[0917] The server analyzes the data and obtains the evaluation result that "the timing of raising the head is too early." In addition to feedback based on the evaluation result, feedback reflecting the user's emotional state is generated.

[0918] Terminal device:

[0919] The smartphone receives the feedback and sends it to the smart glasses.

[0920] Smart Glasses:

[0921] The smart glasses visually indicate the appropriate time to lift your head and provide audio instructions such as, "Be aware of your rhythm to stabilize your swing speed." If the user is concentrating too much, the glasses provide additional advice such as, "Keep your concentration and keep swinging."

[0922] Through these processes, users can analyze their swing in real time and receive personalized feedback. The introduction of an emotion engine allows for more appropriate advice based on the user's psychological state, enabling efficient and effective improvement of swing technique.

[0923] The processing flow will be explained below.

[0924] Step 1:

[0925] The user wears the smart glasses and performs a golf swing. The smart glasses' camera records the user's swing in real time, the facial recognition camera records the user's facial expressions, biometric sensors measure the user's heart rate and galvanic skin response, and a microphone collects audio.

[0926] Step 2:

[0927] The video data of the swing recorded by the smart glasses, as well as emotional data such as facial expressions, voice, and heart rate, are transferred to the device via Wi-Fi or Bluetooth.

[0928] Step 3:

[0929] The device stores the video data and emotion data received from the smart glasses. The stored data is preprocessed, such as trimming unnecessary parts and converting the data format. The video data is processed first, and then the emotion data is preprocessed.

[0930] Step 4:

[0931] The device adds metadata (time information, environmental conditions, etc.) to the preprocessed video data, and after preprocessing and adding the metadata, uploads this data to the server.

[0932] Step 5:

[0933] The server receives the pre-processed video data and queues it for processing. Machine learning algorithms are used to analyze the swing grip, ball placement, swing speed, ball trajectory, and body movement.

[0934] Step 6:

[0935] The server receives the pre-processed emotion data and uses emotion recognition algorithms to identify the user's emotional state, for example, assessing stress levels and concentration levels based on the user's facial expressions and heart rate.

[0936] Step 7:

[0937] The server generates the analysis results. From the swing data, it makes evaluations such as "weak grip" or "bad ball placement," and from the emotional data, it makes evaluations such as "high stress level" or "concentrated."

[0938] Step 8:

[0939] The server generates personalized feedback for each user based on the analysis results. The content and presentation of the feedback is adjusted according to the user's emotional state. For example, if the user is feeling stressed, the server will provide additional feedback such as "Relax and concentrate on your swing."

[0940] Step 9:

[0941] The server transmits the generated feedback data to the terminal. The feedback data includes data for displaying the ideal grip position and ball placement as visual feedback, and audio guidance as auditory feedback.

[0942] Step 10:

[0943] The terminal receives the feedback data and prepares it for transmission to the smart glasses, after which the feedback data is transferred to the smart glasses.

[0944] Step 11:

[0945] The smart glasses provide visual feedback (e.g., correct grip position and ball placement) on the display, and auditory feedback (voice messages) through a voice output device, instructing the user to "hold the grip tighter and adjust the ball placement to the right." They also provide additional advice based on the user's emotional state.

[0946] Step 12:

[0947] The user can then adjust their swing based on the feedback and reflect it in their next swing. Users can take advantage of real-time feedback to continually improve their swing.

[0948] Through these specific steps, users can receive real-time analysis of their swing and personalized feedback. The introduction of an emotion engine allows for more appropriate advice based on the user's psychological state, enabling efficient and effective improvement of swing technique.

[0949] Example 2

[0950] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0951] Conventional golf swing analysis systems can analyze a user's swing and provide feedback, but they cannot provide personalized feedback that takes the user's emotional state into account. As a result, the advice given ignores the user's psychological state, making it difficult to perform optimal practice. To solve this problem, a new system is required to measure the user's emotional state in real time and provide feedback that reflects this in the analysis.

[0952] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0953] In this invention, the server includes smart glasses, which are wearable devices worn by a user, a means for recording and measuring a golf swing and a user's emotional state in real time using a camera and a sensor mounted on the smart glasses, a terminal device for receiving the recorded video data and emotional data and trimming and formatting the data, a means for analyzing the preprocessed video data and emotional data and generating an evaluation and feedback based on the swing and the user's emotional state, and a means for providing the generated feedback visually and audibly to the smart glasses, thereby enabling personalized feedback that takes the user's emotional state into consideration.

[0954] "Smart glasses" are wearable devices worn by users that have built-in cameras and sensors and collect video data of golf swings and biometric data of the user.

[0955] The "camera" is a photographic device installed in the smart glasses that records the user's golf swing and facial expressions in real time.

[0956] A "sensor" is a measuring device installed in smart glasses that acquires biometric data such as the user's heart rate and skin galvanic response.

[0957] "Video data" refers to video data of a golf swing recorded by the smart glasses camera.

[0958] "Emotional data" refers to data that indicates the user's emotional state, acquired through the smart glasses' facial recognition camera, microphone, and biometric sensors.

[0959] A "terminal device" is a device that receives video data and emotion data sent from smart glasses and performs data preprocessing (trimming and format conversion).

[0960] A "server" is a device that analyzes the pre-processed video data and emotional data and generates evaluations and feedback based on swing and emotional state.

[0961] "Preprocessing" refers to the process of trimming and formatting data performed by the terminal device.

[0962] "Analysis" means the processing of data by the Server for the purposes of evaluating the data and generating feedback.

[0963] "Feedback" refers to visual and audible advice provided to the user based on the evaluation results generated by the server.

[0964] "Visual feedback" refers to visual information displayed on the smart glasses screen that shows corrections to your golf swing and the ideal swing form.

[0965] "Auditory feedback" refers to advice in the form of audio messages played through the audio output device of the smart glasses.

[0966] This invention uses smart glasses, a wearable device worn by the user, to collect video data of the golf swing, analyze it with an analysis system, and provide visual and auditory feedback.Furthermore, by combining it with an emotion engine, it is possible to provide more personalized feedback based on the user's emotional state.

[0967] Overall system configuration

[0968] 1. Smart Glasses:

[0969] Equipped with a camera: Records the user's swing in real time.

[0970] Facial expression recognition camera: Captures the user's facial expression data required for the emotion engine.

[0971] Microphone: Collects audio data.

[0972] Biometric sensors: measure biometric data such as heart rate and galvanic skin response.

[0973] Display: Provides visual feedback.

[0974] Audio output device: Provides auditory feedback.

[0975] 2. Terminal device (e.g., smartphone):

[0976] Receiving video data: Receive video data from the smart glasses.

[0977] Receiving emotional data: Receiving facial expressions, voice, and biometric data.

[0978] Data preprocessing: Trimming and format conversion of video data and emotion data.

[0979] Communication with the server: Send the preprocessed data to the server.

[0980] 3. Server Device:

[0981] Swing Data Analysis: Analyze grip, ball placement, swing speed, ball trajectory, and body movement.

[0982] Emotional data analysis: Recognize the user's emotional state and reflect it in the feedback.

[0983] Feedback generation: Generates user-specific visual and auditory feedback.

[0984] Sending feedback: Sending the generated feedback to the end device.

[0985] Program processing

[0986] Smart Glasses Processing

[0987] The system uses a camera to collect video data of the user's golf swing. At the same time, a facial recognition camera records the user's facial expressions, a microphone collects audio data, and biometric sensors capture biometric data such as heart rate and skin galvanic response.

[0988] The collected data is transferred to a terminal device.

[0989] Terminal processing

[0990] The video data and emotion data received from the smart glasses are saved and unnecessary parts are trimmed.

[0991] The video data is converted into an appropriate format, and metadata (time information, environmental conditions, etc.) is added to the emotion data.

[0992] The preprocessed data is uploaded to a server device.

[0993] Server device processing

[0994] The system analyzes pre-processed video data sent from the terminal device to identify and evaluate the swing grip, ball placement, swing speed, ball trajectory, and body movement.

[0995] Emotional data is analyzed to recognize the user's emotional state, for example, to determine whether the user is stressed or focused.

[0996] Generate feedback based on the analysis results. Dynamically adjust the content and presentation of the feedback depending on the user's emotional state. For example, if the user is feeling stressed, provide advice on how to relax.

[0997] The feedback content is transmitted to the terminal device.

[0998] Terminal device feedback processing

[0999] The feedback received from the server is prepared for transmission to the smart glasses and transferred to the smart glasses.

[1000] Feedback processing for smart glasses

[1001] Visual feedback (e.g., correct grip position and ball placement) is provided on the display.

[1002] Auditory feedback (voice messages) is played on the audio output device, instructing the user to, for example, "Hold a tighter grip and correct the ball placement to the right."

[1003] Specific examples

[1004] Beginner user use cases

[1005] 1. A user wears smart glasses and performs a golf swing. The smart glasses' camera records the swing and facial expressions, the microphone collects audio data, and the biometric sensor records the heart rate.

[1006] 2. The terminal device receives the recorded data and emotion data, preprocesses the data (trimming unnecessary parts, format conversion, adding metadata), and sends it to the server.

[1007] 3. The server analyzes the video data and generates feedback that reflects the user's emotional state, such as "Your grip is weak" or "The ball is placed too left." For example, if the user is feeling stressed, the server provides additional feedback such as "Relax and concentrate on your swing."

[1008] 4. The terminal device receives the feedback and sends it to the smart glasses.

[1009] 5. The smart glasses visually display the correct grip position and provide audio instructions such as "Hold a firmer grip and correct the ball placement to the right." If the user feels stressed, the glasses also provide additional advice such as "Relax and focus on your swing."

[1010] Intermediate User Use Cases

[1011] 1. The user wears the smart glasses while swinging. The camera records the swing, captures facial expression data, collects audio with a microphone, and records heart rate with a biometric sensor.

[1012] 2. The terminal device preprocesses the recorded data and emotion data and sends them to the server.

[1013] 3. The server analyzes the data and obtains the evaluation result that "the timing of raising the head is too early." In addition to feedback based on the evaluation result, feedback reflecting the user's emotional state is generated.

[1014] 4. The terminal device receives the feedback and sends it to the smart glasses.

[1015] 5. The smart glasses visually indicate the appropriate time to raise your head and provide audio instructions such as, "Be aware of your rhythm to stabilize your swing speed." If the user is concentrating too much, the smart glasses provide additional advice such as, "Keep your concentration and keep swinging."

[1016] Prompt Sentence Examples

[1017] Example prompts to be input to the generative AI model:

[1018] Your task is to write a program to operate smart glasses for a golf swing analysis system. The system will collect data by combining cameras, facial recognition, voice recognition, and biometric sensors to analyze the user's swing and emotional state. It will then provide appropriate feedback in real time. Through the process from data collection to feedback provision, we aim to improve the user's swing technique and care for their emotions.

[1019] This prompt statement enables the generative AI model to understand the overall flow of the system and generate the necessary program code.

[1020] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1021] Step 1:

[1022] The user wears smart glasses and performs a golf swing. The smart glasses collect video data of the user's golf swing using a built-in camera. At the same time, the facial recognition camera captures the user's facial expression data, and the microphone collects audio data. The smart glasses also use biometric sensors to capture biometric data such as heart rate and galvanic skin response. This data is stored in the smart glasses' internal memory.

[1023] input:

[1024] User's golf swing

[1025] User's facial expression

[1026] User Voice

[1027] User's biometric information

[1028] output:

[1029] Golf swing video data

[1030] User facial expression data

[1031] User voice data

[1032] User biometric data

[1033] Step 2:

[1034] All data collected by the smart glasses is transferred to the terminal device in real time using wireless communication such as Bluetooth or Wi-Fi.

[1035] input:

[1036] Golf swing video data

[1037] User facial expression data

[1038] User voice data

[1039] User biometric data

[1040] output:

[1041] Transferred video data

[1042] Transferred facial expression data

[1043] Transferred audio data

[1044] Transmitted biometric data

[1045] Step 3:

[1046] The terminal device stores the data received from the smart glasses and trims any unnecessary parts. For example, unnecessary parts before and after a swing are cut out. The video data is also converted into an appropriate format for analysis. Metadata such as time information and environmental conditions are added to the emotion data. After this preprocessing, the data is stored in the terminal device's internal memory.

[1047] input:

[1048] Transferred video data

[1049] Transferred facial expression data

[1050] Transferred audio data

[1051] Transmitted biometric data

[1052] output:

[1053] Video data after trimming

[1054] Format converted video data

[1055] Emotion data with added metadata

[1056] Step 4:

[1057] The terminal device uploads the pre-processed data to the server, which transfers the data via the Internet. Once uploaded, the data is stored on the server for analysis.

[1058] input:

[1059] Video data after trimming

[1060] Format converted video data

[1061] Emotion data with added metadata

[1062] output:

[1063] Preprocessed data stored on the server

[1064] Step 5:

[1065] The server analyzes the pre-processed video data and emotional data. It uses AI algorithms to identify and evaluate swing grip, ball placement, swing speed, ball trajectory, and body movement. It also analyzes the emotional data to recognize the user's emotional state (e.g., stress, concentration).

[1066] input:

[1067] Preprocessed video data

[1068] Emotion data with added metadata

[1069] output:

[1070] Swing data analysis results

[1071] Emotion data analysis results

[1072] Step 6:

[1073] The server generates personalized visual and auditory feedback for the user based on the analysis results. The content and presentation of the feedback are dynamically adjusted according to the user's emotional state. For example, if the user is feeling stressed, advice to help them relax is generated. The generated feedback is stored on the server.

[1074] input:

[1075] Swing data analysis results

[1076] Emotion data analysis results

[1077] output:

[1078] Individual user feedback

[1079] Step 7:

[1080] The server transmits the generated feedback to the terminal device, with data transfer occurring over the Internet.

[1081] input:

[1082] Individual user feedback

[1083] output:

[1084] Feedback sent to the device

[1085] Step 8:

[1086] The terminal device receives the feedback and prepares it for transmission to the smart glasses. The feedback is transferred to the smart glasses via Bluetooth or Wi-Fi.

[1087] input:

[1088] Feedback sent to the device

[1089] output:

[1090] Feedback transmitted to smart glasses

[1091] Step 9:

[1092] The smart glasses display visual feedback (e.g., correct grip position and ball placement) on the display, and also play auditory feedback (e.g., "Hold your grip tighter and correct the ball placement to the right") through an audio output device, allowing the user to receive feedback in real time.

[1093] input:

[1094] Feedback transmitted to smart glasses

[1095] output:

[1096] Visual and auditory feedback provided to the user

[1097] (Application example 2)

[1098] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1099] Currently, many factory robot and worker motion analysis systems have difficulty detecting motion errors and efficiency declines in real time and providing appropriate feedback. Furthermore, few systems consider the psychological state and stress level of workers to improve their motion. Therefore, there is a need for more effective and adaptive feedback systems to improve production efficiency and reduce errors.

[1100] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1101] In this invention, the server includes a means for evaluating and generating feedback on operations, placement, and efficiency, a means for analyzing emotional data and recognizing the stress and concentration of workers, and a means for providing the generated feedback visually and audibly. This makes it possible to analyze the operations of factory robots and workers in real time and provide appropriate feedback. It is also possible to provide more personalized feedback by taking into account the psychological state of the workers.

[1102] A "user" is a person or operator who uses a motion analysis system for golf swings or robotic work.

[1103] A "wearable device" is a device that can be worn by a user and is capable of collecting and displaying data.

[1104] "Smart glasses" are wearable devices equipped with cameras, sensors, displays, etc., that function as an interface for visual and auditory data.

[1105] A "camera" is a device that takes and records images and videos.

[1106] A "golf swing" refers to a golf hitting motion, and is an act for collecting data on a user's swing motion.

[1107] "Real time" refers to processing events as they occur, without any time lag.

[1108] "Recording means" refers to a method or device for recording video data using a camera mounted on smart glasses.

[1109] "Video data" refers to data that displays a series of images recorded by a camera in a time-series sequence.

[1110] "Emotion data" is data that indicates the emotional state of the user based on biological reactions such as the user's heart rate and facial expressions.

[1111] "Preprocessing" refers to the process of processing or converting collected data to make it easier to analyze.

[1112] A "terminal device" is an electronic device capable of data communication and processing, including smartphones and tablets.

[1113] A "server device" is a computer system for storing, managing, and analyzing data.

[1114] "Evaluation" is the act of judging the quality of the results of data analysis based on specific criteria.

[1115] "Feedback" refers to instructions or advice provided to the user based on the evaluation results.

[1116] "Visual feedback" is information provided to a user using a visual medium such as a display.

[1117] "Auditory feedback" refers to spoken messages or instructions provided to a user through an audio output device.

[1118] "Efficiency of action" is a measure of how effectively an action is performed to achieve a goal.

[1119] "Layout" refers to the state of how things or people are arranged.

[1120] An "error rate" is the rate at which errors occur during a particular task or activity.

[1121] "Physical responses" are biometric data that indicate the user's physical changes and condition.

[1122] System Configuration

[1123] Hardware

[1124] 1. Smart Glasses

[1125] camera

[1126] Facial Recognition Camera

[1127] microphone

[1128] Biometric Sensor

[1129] display

[1130] Audio output device

[1131] 2. Terminal Device

[1132] Smartphone or tablet

[1133] 3. Server Device

[1134] Computer system for data analysis

[1135] software

[1136] 1. Video analysis software

[1137] (e.g. OpenCV)

[1138] 2. Facial Recognition Software

[1139] 3. Audio Analysis Software

[1140] 4. Biological Data Analysis Software

[1141] 5. Emotion Engine

[1142] 6. Feedback Generator

[1143] Overall system processing

[1144] 1. Data Collection and Preprocessing

[1145] Smart Glasses:

[1146] The camera installed in the smart glasses captures the user's work and movements in real time.

[1147] A facial expression recognition camera captures the user's facial expression data, and a microphone collects voice data.

[1148] Biosensors are used to measure biometric data such as heart rate and skin galvanic response.

[1149] Terminal device:

[1150] Receive video data and emotion data transmitted from the smart glasses.

[1151] Trim out any unwanted parts and convert to the appropriate format.

[1152] The preprocessed data is transferred to a server device.

[1153] 2. Data analysis and feedback generation

[1154] Server device:

[1155] Using video analysis software, we analyze the efficiency, placement, and error rate of movements.

[1156] Using facial expression recognition software and biometric data analysis software, the system analyzes the user's emotional state, thereby identifying the user's stress and concentration level.

[1157] The emotional engine generates feedback based on the results of motion analysis, dynamically adjusting the visual and auditory feedback according to the user's emotional state.

[1158] Generate feedback:

[1159] Based on the analysis results, feedback is generated that provides visual and auditory indications of work improvements and points to note.

[1160] The server transmits the feedback content to the terminal device.

[1161] 3. Providing Feedback

[1162] Terminal device:

[1163] The received feedback is forwarded to the smart glasses.

[1164] Smart Glasses:

[1165] Visual feedback is provided on the display, such as ideal operating positions and areas requiring attention.

[1166] The auditory feedback is played on an audio output device to instruct the user as a voice message.

[1167] Specific examples

[1168] Use cases for beginner users:

[1169] A user wears smart glasses and works on a factory production line. The smart glasses' camera captures their movements and facial expressions, the microphone collects audio data, and the biometric sensor records their heart rate. The terminal device preprocesses this data and sends it to a server. Based on the analysis results, the server provides visual and auditory feedback on areas for improvement and important points to note.

[1170] Example prompt sentence:

[1171] "Analyze the robot's movements during work and generate feedback to maximize efficiency. Also, recognize the worker's stress and concentration levels and provide advice based on that."

[1172] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1173] Step 1:

[1174] Data collection using smart glasses

[1175] Input: User movements, facial expressions, voice data, biometric data (heart rate, skin galvanic response)

[1176] How it works: A camera on the smart glasses captures the user's movements in real time, a facial recognition camera records the user's facial expressions, a microphone collects audio data, and biometric sensors measure heart rate and galvanic skin response.

[1177] Output: Acquisition of video data, facial expression data, audio data, and biometric data

[1178] Step 2:

[1179] Data reception and preprocessing by the terminal device

[1180] Input: Video data, facial expression data, audio data, and biometric data sent from smart glasses

[1181] Specific operation: The terminal device (smartphone or tablet) receives the data sent from the smart glasses, trims unnecessary parts, converts the video data into an appropriate format, and adds metadata (time information, environmental conditions, etc.) to the emotion data.

[1182] Output: Preprocessed video data and emotion data

[1183] Step 3:

[1184] Data transmission from terminal device to server

[1185] Input: Preprocessed video data and emotion data

[1186] Specific operations: The terminal device uploads the preprocessed data to the server device.

[1187] Output: Preprocessed data sent to the server

[1188] Step 4:

[1189] Data analysis by server

[1190] Input: Preprocessed data sent from the terminal

[1191] Specific operations: The server analyzes the video data and evaluates the efficiency, placement, and error rate of the movements. It also uses facial expression recognition software and biometric data analysis software to analyze the user's emotional data and recognize their stress and concentration levels.

[1192] Output: Movement evaluation results and emotional state analysis results

[1193] Step 5:

[1194] Feedback Generation

[1195] Input: Movement evaluation results and emotional state analysis results

[1196] Specific behavior: The server utilizes an emotion engine to generate feedback based on the behavior analysis results and the user's emotional state. The visual and auditory feedback content is dynamically adjusted.

[1197] Output: Visual and auditory feedback

[1198] Step 6:

[1199] Sending feedback to terminal devices

[1200] Input: Server-generated feedback

[1201] Specific operations: The server device transmits the generated feedback to the terminal device.

[1202] Output: Feedback sent to the terminal

[1203] Step 7:

[1204] Sending feedback from the device to the smart glasses

[1205] Input: Feedback sent to the terminal

[1206] Specific operation: The terminal device transmits the feedback content to the smart glasses.

[1207] Output: Feedback sent to the smart glasses

[1208] Step 8:

[1209] Feedback provided by smart glasses

[1210] Input: Feedback sent to the smart glasses

[1211] Specific operation: Smart glasses display visual feedback on the display and play auditory feedback through the audio output device. For example, instructions for correcting movement placement or advice on preventing mistakes are provided via both visual and audio feedback.

[1212] Output: The visual and auditory feedback the user receives.

[1213] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1214] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1215] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[1216] [Third embodiment]

[1217] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[1218] 5, the data processing system 310 includes the data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[1219] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1220] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[1221] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1222] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1223] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1224] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1225] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1226] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1227] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1228] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[1229] The golf play assistance service of the present invention uses smart glasses worn by the user to collect video data of the golf swing in real time, analyzes it with an analysis system, and provides visual and auditory feedback. The system consists of smart glasses, a terminal device (such as a smartphone), and a server device.

[1230] Overall system configuration

[1231] 1. Smart Glasses:

[1232] Equipped with a camera: Records your swing from your perspective.

[1233] Display: Provides visual feedback.

[1234] Audio output device: Provides auditory feedback.

[1235] 2. Terminal device (e.g., smartphone):

[1236] Receiving data: Receives recording data from smart glasses.

[1237] Data preprocessing: trimming and formatting data.

[1238] Communication with the server: Send the preprocessed data to the server.

[1239] 3. Server Device:

[1240] Data Analysis: Analyze each element of your swing to generate an evaluation and feedback.

[1241] Sending feedback: Sending the generated feedback back to the end device.

[1242] Program processing

[1243] 1. Smart Glasses Processing:

[1244] A camera is used to collect video data of a user's golf swing.

[1245] The collected video data is transferred to the terminal device.

[1246] 2. Terminal Processing:

[1247] Receives video data transferred from smart glasses.

[1248] Trim unnecessary parts and convert the data format into one that is easy to analyze.

[1249] The preprocessed data is sent to a server device.

[1250] 3. Processing of server equipment:

[1251] The preprocessed video data transmitted from the terminal device is received.

[1252] Machine learning algorithms are used to analyze swing grip, ball placement, swing speed, ball trajectory, and body movement.

[1253] Based on the analysis results, feedback is generated for each user.

[1254] The feedback content is transmitted to the terminal device.

[1255] 4. Terminal device feedback processing:

[1256] Process the feedback received from the server.

[1257] Sending the processed feedback to the smart glasses.

[1258] 5. Smart Glasses Feedback Processing:

[1259] Visual feedback (ideal grip position and ball placement) is displayed on the screen.

[1260] Auditory feedback (voice messages) is played on an audio output device.

[1261] Specific examples

[1262] Beginner user use cases

[1263] User:

[1264] A beginner user wears smart glasses and performs a golf swing. The camera in the smart glasses records the swing and transmits the data to a smartphone in real time.

[1265] Terminal device:

[1266] The smartphone receives the recorded data, performs pre-processing, and then sends the data to the server. Pre-processing includes trimming unnecessary parts and format conversion.

[1267] Server device:

[1268] The server analyzes the pre-processed data and generates evaluation results such as "weak grip" or "ball placement too left." Based on the analysis results, specific feedback is generated and sent to the smartphone.

[1269] Terminal device:

[1270] The smartphone receives the feedback and sends it to the smart glasses.

[1271] Smart Glasses:

[1272] The smart glasses visually display the correct grip position on the HUD and provide audio advice such as "Hold a firm grip and adjust the ball placement to the right."

[1273] Intermediate User Use Cases

[1274] User:

[1275] An intermediate golfer wears the smart glasses while swinging. The camera records the swing and sends the data to a smartphone.

[1276] Terminal device:

[1277] The smartphone pre-processes the recorded data and sends it to the server.

[1278] Server device:

[1279] The server analyzes the preprocessed data and obtains evaluation results such as "the timing of lifting the head is too early" and "the swing speed is unstable." Feedback based on the evaluation results is generated and sent to the smartphone.

[1280] Terminal device:

[1281] The smartphone receives the feedback and sends it to the smart glasses.

[1282] Smart Glasses:

[1283] The smart glasses visually display the appropriate timing to raise your head on the HUD and provide audio guidance such as "be aware of rhythm to stabilize your swing speed."

[1284] Through these processes, the user can understand in real time what corrections to make to their swing and effectively improve their technique.

[1285] The processing flow will be explained below.

[1286] Step 1:

[1287] The user wears the smart glasses and performs a golf swing, with the camera in the smart glasses recording the user's swing in real time.

[1288] Step 2:

[1289] The swing video data recorded by the smart glasses is transferred to the device via Wi-Fi or Bluetooth.

[1290] Step 3:

[1291] The device saves the video data received from the smart glasses. Once saved, it performs pre-processing such as trimming unnecessary parts and converting the data format.

[1292] Step 4:

[1293] The device adds metadata (time information, environmental conditions, etc.) to the preprocessed video data. After preprocessing and adding metadata, the data is uploaded to the server.

[1294] Step 5:

[1295] The server receives the pre-processed video data and queues it for processing. Machine learning algorithms are used to analyze each element of the swing (grip, ball placement, swing speed, ball trajectory, and body movement).

[1296] Step 6:

[1297] The server generates an analysis result, which may include evaluations such as "weak grip" or "bad ball placement," as well as specific suggestions such as "unstable swing speed" or "head lifting timing too early."

[1298] Step 7:

[1299] Based on the analysis results, the server generates individualized feedback for each user, including visual feedback such as data to display the ideal grip position and ball placement, and audio guidance as auditory feedback.

[1300] Step 8:

[1301] The server transmits the generated feedback data to the terminal.

[1302] Step 9:

[1303] The terminal prepares the received feedback data for transmission to the smart glasses and transfers it to the smart glasses.

[1304] Step 10:

[1305] The smart glasses provide visual feedback (e.g., correct grip position and ball placement) on the display.

[1306] Step 11:

[1307] The smart glasses play auditory feedback (voice messages) through an audio output device, instructing the user, for example, to "tighten the grip and adjust the ball placement to the right."

[1308] Step 12:

[1309] The user can then adjust their swing based on the feedback and reflect it in their next swing. Users can take advantage of real-time feedback to continually improve their swing.

[1310] Through these specific steps, users can analyze their swing in real time and receive effective feedback, allowing them to improve their swing technique efficiently, regardless of location.

[1311] Example 1

[1312] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1313] There is a need for a system that can provide real-time feedback to improve golf swing technique. However, conventional systems lack coordination between the data collection, preprocessing, analysis, and feedback stages, making it difficult to provide effective training. Another issue is that it is difficult for users to specifically identify which elements of their swing need improvement.

[1314] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1315] In this invention, the server includes smart glasses, which are wearable devices worn by a user, a means for recording a golf swing in real time using a camera mounted on the smart glasses, a terminal device for receiving the recorded video data, trimming unnecessary portions, and converting the data format, a server device equipped with a machine learning algorithm for analyzing the preprocessed video data, analyzing each element of the swing, evaluating the swing, and generating feedback, and a means for providing the generated feedback visually and audibly to the smart glasses, thereby enabling the user to receive specific and effective feedback in real time.

[1316] "Smart glasses" are wearable devices worn by users that have visual and auditory interfaces.

[1317] The "camera" is a device installed in the smart glasses that records the golf swing from the user's perspective in real time.

[1318] "Recorded data" refers to video data of golf swings collected by the smart glasses camera.

[1319] The "terminal device" is a device that receives video data sent from the smart glasses, trims unnecessary parts, and converts the data into a format that is easy to analyze.

[1320] "Preprocessing" refers to the process of trimming unnecessary parts of the recorded data and converting the data format into one that is easier to analyze.

[1321] The "server device" is a computer system that analyzes the pre-processed video data, analyzes each element of the swing, and generates an evaluation and feedback.

[1322] A "machine learning algorithm" is a program used by the server device, and is a technology for analyzing factors such as swing grip, ball placement, swing speed, ball trajectory, and body movement.

[1323] "Feedback" refers to advice and evaluation information for each user that is generated based on the analysis results and is provided in visual and auditory formats.

[1324] "Visual feedback" refers to information such as the ideal grip position and ball placement that is displayed on the smart glasses display.

[1325] "Auditory feedback" refers to an audio message played through the audio output device of the smart glasses.

[1326] MODE FOR CARRYING OUT THE INVENTION

[1327] This golf play assistance service uses smart glasses worn by the user to collect video data of the golf swing in real time, analyzes it with an analysis system, and provides visual and auditory feedback. The system consists of smart glasses, a terminal device, and a server device.

[1328] Smart Glasses

[1329] Hardware: The smart glasses are equipped with a camera to record the user's swing from their perspective, and also have a display and audio output to provide visual and audio feedback.

[1330] Specific features:

[1331] A camera is used to collect video data of the user's golf swing and transfer it to a terminal device in real time.

[1332] The collected video data is transferred to a terminal device using wireless communication means such as Wi-Fi.

[1333] The analyzed feedback is displayed visually on a display and audibly reproduced through an audio output device.

[1334] terminal device

[1335] Hardware: Terminal devices are general-purpose devices such as smartphones and tablets.

[1336] Software: An application is installed to preprocess video data and communicate with the server.

[1337] Specific features:

[1338] Receives video data sent from smart glasses.

[1339] The received video data is trimmed and converted into a format suitable for analysis (e.g., MP4 format).

[1340] The pre-processed video data is encrypted and sent to a server via the Internet.

[1341] Server device

[1342] Hardware: A high-performance computer system with the processing power to analyze large amounts of data.

[1343] Software: Includes analysis programs that implement machine learning algorithms (e.g., TensorFlow, PyTorch).

[1344] Specific features:

[1345] The preprocessed video data transmitted from the terminal device is received.

[1346] Machine learning algorithms are used to analyze swing grip, ball placement, swing speed, ball trajectory, and body movement.

[1347] Generate evaluation and feedback based on the analysis results.

[1348] The generated feedback is transmitted to the terminal device.

[1349] Specific examples

[1350] Beginner user use cases

[1351] User: A beginner user wears the smart glasses and performs a golf swing. The smart glasses' camera records the swing and transmits the data to a smartphone in real time.

[1352] Terminal device: The smartphone receives the recorded data, performs pre-processing, and then sends the data to the server. Pre-processing includes trimming unnecessary parts and format conversion.

[1353] Server device: The server analyzes the pre-processed data and generates evaluation results such as "the grip is weak" or "the ball is placed too left." Based on the analysis results, specific feedback is generated and sent to the smartphone.

[1354] Terminal device: The smartphone receives the feedback and sends it to the smart glasses.

[1355] Smart glasses: The smart glasses visually show the correct grip position on the display and provide audio advice such as "Hold a firmer grip and adjust the ball placement to the right."

[1356] Prompt Sentence Examples

[1357] "Please explain in natural language the process of analyzing a video of a beginner golfer's swing and providing feedback on areas for improvement."

[1358] In this way, the user can understand in real time what corrections to make to their swing and effectively improve their technique.

[1359] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1360] Step 1: Collecting video data with smart glasses

[1361] User: The user wears the smart glasses and performs a golf swing.

[1362] Input: The user's golf swing motion.

[1363] How it works: The smart glasses' camera records the entire swing from the user's perspective.

[1364] Output: Video data stored inside the smart glasses.

[1365] Step 2: Transferring video data from smart glasses to your device

[1366] Smart glasses: Smart glasses transfer recorded video data to the terminal device.

[1367] Input: Video data stored on smart glasses.

[1368] Specific operation: Smart glasses transmit video data to terminal devices in real time via wireless communication such as Wi-Fi.

[1369] Output: Video data transferred to the terminal device.

[1370] Step 3: Receiving and preprocessing video data on the terminal device

[1371] Terminal: The terminal device receives the video data transmitted from the smart glasses.

[1372] Input: Video data sent from smart glasses.

[1373] Specific operation: Trim unnecessary parts of the video data and convert it into a format that is easy to analyze (e.g., MP4 format).

[1374] Output: Preprocessed video data.

[1375] Step 4: Sending data from the terminal to the server

[1376] Terminal: The terminal device transmits the preprocessed video data to the server device.

[1377] Input: Preprocessed video data.

[1378] Specific operation: The terminal device encrypts data and sends it to the server via the Internet.

[1379] Output: Preprocessed video data transferred to the server.

[1380] Step 5: Video data analysis by the server

[1381] Server: The server device analyzes the received preprocessed video data.

[1382] Input: Preprocessed video data.

[1383] Specific behavior: Machine learning algorithms are used to analyze swing grip, ball placement, swing speed, ball trajectory, and body movement.

[1384] Output: Analysis results and feedback information.

[1385] Step 6: Sending feedback from the server to the terminal device

[1386] Server: The server device sends the generated feedback to the terminal device.

[1387] Input: Analysis results and feedback information.

[1388] Specific operations: The feedback information is formatted and sent to the terminal device via the Internet.

[1389] Output: Feedback information transmitted to the terminal device.

[1390] Step 7: End device processes feedback and sends it to the smart glasses

[1391] Terminal: The terminal device processes the feedback information received from the server and sends it to the smart glasses.

[1392] Input: Feedback information received from the server.

[1393] What it does: Feedback information is optimized as visual and auditory feedback and wirelessly transmitted to the smart glasses.

[1394] Output: Feedback information transmitted to the smart glasses.

[1395] Step 8: Providing feedback with smart glasses

[1396] Smart Glasses: Smart glasses provide the received feedback to the user.

[1397] Input: Feedback information sent from the terminal device.

[1398] What it does: Visual feedback is shown on the display, and auditory feedback is played through the audio output device.

[1399] Output: The feedback presented to the user.

[1400] (Application example 1)

[1401] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1402] Accuracy and efficiency are required for robot operations within factories. In particular, if a transport robot delivers packages incorrectly or operates improperly, it can affect overall production efficiency. The purpose of this invention is to solve these issues by analyzing robot operations within factories in real time and providing rapid feedback.

[1403] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1404] In this invention, the server includes a means for generating an evaluation and feedback for the operation, a means for providing the generated feedback visually and audibly to the smart glasses, and a means for improving the operation of the robot in the factory, thereby making it possible to analyze the operation of the robot in the factory in real time and provide appropriate feedback.

[1405] "User" refers to the person who wears and uses this system.

[1406] "Wearable device" refers to a device that can be worn by a user, in this case smart glasses.

[1407] "Smart glasses" refers to eyeglass-type devices equipped with cameras, displays, etc.

[1408] "Camera" refers to a device that captures images or videos.

[1409] "Movement" refers to the series of movements performed by a robot in a factory.

[1410] "Real-time" refers to data being processed immediately, without delay.

[1411] "Recording" refers to the recording of an action using a camera.

[1412] "Means" refers to a method or apparatus for performing a particular function or process.

[1413] "Video data" refers to video information of recorded actions.

[1414] "Receiving" refers to receiving data.

[1415] "Preprocessing" refers to the process of changing data into a form suitable for analysis.

[1416] "Terminal device" refers to a device that receives and pre-processes data, such as a smartphone.

[1417] "Server device" refers to a device that analyzes data and generates feedback.

[1418] "Evaluation" refers to the analysis results of recorded and analyzed behavior.

[1419] "Feedback" refers to suggestions for improvement or instructions provided based on the evaluation results.

[1420] "Visual" refers to providing information in a form that can be seen by the user, such as through a display.

[1421] "Auditory" refers to providing information to the user in an audible form, such as through voice.

[1422] "Accuracy" refers to the degree to which an action is performed as planned.

[1423] "Efficiency" refers to the degree to which resources are used optimally to obtain maximum results.

[1424] "Pickup position" refers to the correct position where the robot will pick up the luggage.

[1425] "Smoothness of movement" refers to the degree to which a robot can perform movements continuously and without interruption.

[1426] This invention relates to a system that monitors and analyzes the operation of robots in factories in real time and provides feedback. This system consists of smart glasses, which are wearable devices worn by users, a camera mounted on the smart glasses, a terminal device that receives and preprocesses the data, and a server device that analyzes the data.

[1427] System configuration

[1428] 1. Smart Glasses

[1429] The smart glasses, a wearable device worn by the user, are equipped with a camera that can record the robot's movements in real time, and also have a display and audio output device that provide visual and auditory feedback.

[1430] 2. Terminal Device

[1431] The recorded video data is sent to a terminal device (e.g., a smartphone). This terminal device performs preprocessing, trimming unnecessary parts and converting the data format to one that is easier to analyze. The preprocessed data is then sent to a server.

[1432] 3. Server Device

[1433] The server analyzes the received preprocessed video data and evaluates the factory robot's operations. Specifically, it has the ability to analyze accuracy, efficiency, and the smoothness of the pickup position and operation of the goods. This is done using machine learning algorithms such as OpenCV and TensorFlow. Based on the evaluation results, it generates feedback to be provided to the user. This feedback is then sent to the terminal device.

[1434] 4. Providing Feedback

[1435] The terminal device transmits the feedback received from the server to the smart glasses, which provide visual feedback (e.g., displaying the correct action position on the HUD) and auditory feedback (e.g., a voice command saying "Proceed to the next action").

[1436] Specific examples

[1437] For example, suppose a delivery robot operating in a factory frequently mis-delivers packages. The smart glasses worn by the user record the robot's movements in real time, and after preprocessing by a terminal device, send the data to a server. The server analyzes the robot's movement data and obtains the evaluation result that "the package is not being picked up in the correct position." Based on this evaluation result, the smart glasses visually display "correct the position to pick up the package correctly" and provide a voice instruction to "proceed to the next action."

[1438] Prompt Sentence Examples

[1439] Prompt: "Generate code for a robotic behavior analysis system in a factory. The server will be implemented in Python and will use the OpenCV and TensorFlow libraries to analyze the robotic behavior. The smart glasses will use Google Glass to provide real-time feedback. The end device will be an Android smartphone."

[1440] In this way, the present invention can improve robotic operation within a factory, increasing overall production efficiency.

[1441] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1442] Step 1:

[1443] The smart glasses worn by the user use a built-in camera to record the factory robot's operations in real time. In this step, the smart glasses capture the operation video and transfer it to a terminal device as video data. The input is the factory robot's operations, and the output is the recorded video data.

[1444] Step 2:

[1445] The terminal device receives the video data sent from the smart glasses. The terminal device then preprocesses the video data. Specifically, it trims unnecessary parts and converts them into a data format suitable for analysis. The input is the recorded video data, and the output is the preprocessed video data.

[1446] Step 3:

[1447] The terminal device transmits the preprocessed video data to the server device. The server device has a means for analyzing the received video data and evaluates the accuracy, efficiency, package pickup position, and smoothness of the robot's movements from the video data. The input is the preprocessed video data, and the output is the evaluation results.

[1448] Step 4:

[1449] The server device uses machine learning algorithms (e.g., OpenCV and TensorFlow) to evaluate the robot's movements. Specifically, the motion analysis algorithm analyzes the video data and identifies the accuracy of each movement and areas for improvement. The input is the preprocessed video data, and the output is the evaluation results.

[1450] Step 5:

[1451] The server generates feedback based on the evaluation results. Using a generative AI model, it creates specific feedback content to provide to the user. In this case, both visual and auditory feedback is generated. The input is the evaluation results, and the output is the feedback content.

[1452] Step 6:

[1453] The server device sends the generated feedback content to the terminal device. The terminal device sends the received feedback to the smart glasses and presents it to the user in an appropriate manner. At this stage, the input is the feedback content and the output is the feedback transfer to the smart glasses.

[1454] Step 7:

[1455] The smart glasses provide feedback to the user using a display device and an audio output device. Visual feedback shows the correct operating position on the display, and audio feedback indicates the next action. The input is the feedback content sent from the terminal device, and the output is the visual and audio feedback received by the user.

[1456] In this way, a system is created that analyzes the operation of robots in a factory in real time and provides rapid feedback.

[1457] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1458] The present invention uses smart glasses, a wearable device worn by the user, to collect video data of the golf swing, which is then analyzed by an analysis system to provide visual and auditory feedback. Furthermore, by combining this with an emotion engine that recognizes the user's emotional state, more personalized advice can be provided.

[1459] Overall system configuration

[1460] 1. Smart Glasses:

[1461] Equipped with a camera: Records the user's swing in real time.

[1462] Facial expression recognition camera: Captures the user's facial expression data required for the emotion engine.

[1463] Microphone: Collects audio data.

[1464] Biometric sensors: Measure heart rate, skin galvanic response, etc.

[1465] Display: Provides visual feedback.

[1466] Audio output device: Provides auditory feedback.

[1467] 2. Terminal device (e.g., smartphone):

[1468] Receiving video data: Receive video data from the smart glasses.

[1469] Receiving emotional data: Receiving facial expressions, voice, and biometric data.

[1470] Data preprocessing: Trimming and format conversion of video data and emotion data.

[1471] Communication with the server: Send the preprocessed data to the server.

[1472] 3. Server Device:

[1473] Swing Data Analysis: Analyze grip, ball placement, swing speed, ball trajectory, and body movement.

[1474] Emotional data analysis: Recognize the user's emotional state and reflect it in the feedback.

[1475] Feedback generation: Generates user-specific visual and auditory feedback.

[1476] Sending feedback: Sending the generated feedback to the end device.

[1477] Program processing

[1478] Smart Glasses Processing

[1479] The system uses a camera to collect video data of the user's golf swing. At the same time, a facial recognition camera records the user's facial expressions, a microphone collects audio data, and biometric sensors capture biometric data such as heart rate and skin galvanic response.

[1480] The collected data is transferred to a terminal device.

[1481] Terminal processing

[1482] The video data and emotion data received from the smart glasses are saved and unnecessary parts are trimmed.

[1483] The video data is converted into an appropriate format, and metadata (time information, environmental conditions, etc.) is added to the emotion data.

[1484] The preprocessed data is uploaded to a server device.

[1485] Server device processing

[1486] The system analyzes pre-processed video data sent from the terminal device to identify and evaluate the swing grip, ball placement, swing speed, ball trajectory, and body movement.

[1487] Emotional data is analyzed to recognize the user's emotional state, for example, to determine whether the user is stressed or focused.

[1488] Generate feedback based on the analysis results. Dynamically adjust the content and presentation of the feedback depending on the user's emotional state. For example, if the user is feeling stressed, provide advice on how to relax.

[1489] The feedback content is transmitted to the terminal device.

[1490] Terminal device feedback processing

[1491] The feedback received from the server is prepared for transmission to the smart glasses and transferred to the smart glasses.

[1492] Feedback processing for smart glasses

[1493] Visual feedback (e.g., correct grip position and ball placement) is provided on the display.

[1494] Auditory feedback (voice messages) is played on the audio output device, instructing the user to, for example, "Hold a tighter grip and correct the ball placement to the right."

[1495] Specific examples

[1496] Beginner user use cases

[1497] User:

[1498] A beginner wears smart glasses and performs a golf swing. The smart glasses' camera records the swing and facial expressions, the microphone collects audio data, and the biometric sensor records the heart rate.

[1499] Terminal device:

[1500] The smartphone receives the recording data and emotion data, and after preprocessing, sends the data to the server.

[1501] Server device:

[1502] The server analyzes the video data and generates feedback that reflects the user's emotional state, such as "Your grip is weak" or "The ball is placed too left." For example, if the user is feeling stressed, the server provides additional feedback such as "Relax and concentrate on your swing."

[1503] Terminal device:

[1504] The smartphone receives the feedback and sends it to the smart glasses.

[1505] Smart Glasses:

[1506] The smart glasses visually display the correct grip position and provide audio instructions such as "Hold a firmer grip and adjust the ball placement to the right," and if the user feels stressed, they can offer additional advice such as "Relax and focus on your swing."

[1507] Intermediate User Use Cases

[1508] User:

[1509] Intermediate golfers wear smart glasses while swinging. A camera records the swing, captures facial expression data, a microphone collects audio, and a biometric sensor records heart rate.

[1510] Terminal device:

[1511] The smartphone preprocesses the recorded data and emotion data and sends them to the server.

[1512] Server device:

[1513] The server analyzes the data and obtains the evaluation result that "the timing of raising the head is too early." In addition to feedback based on the evaluation result, feedback reflecting the user's emotional state is generated.

[1514] Terminal device:

[1515] The smartphone receives the feedback and sends it to the smart glasses.

[1516] Smart Glasses:

[1517] The smart glasses visually indicate the appropriate time to lift your head and provide audio instructions such as, "Be aware of your rhythm to stabilize your swing speed." If the user is concentrating too much, the glasses provide additional advice such as, "Keep your concentration and keep swinging."

[1518] Through these processes, users can analyze their swing in real time and receive personalized feedback. The introduction of an emotion engine allows for more appropriate advice based on the user's psychological state, enabling efficient and effective improvement of swing technique.

[1519] The processing flow will be explained below.

[1520] Step 1:

[1521] The user wears the smart glasses and performs a golf swing. The smart glasses' camera records the user's swing in real time, the facial recognition camera records the user's facial expressions, biometric sensors measure the user's heart rate and galvanic skin response, and a microphone collects audio.

[1522] Step 2:

[1523] The video data of the swing recorded by the smart glasses, as well as emotional data such as facial expressions, voice, and heart rate, are transferred to the device via Wi-Fi or Bluetooth.

[1524] Step 3:

[1525] The device stores the video data and emotion data received from the smart glasses. The stored data is preprocessed, such as trimming unnecessary parts and converting the data format. The video data is processed first, and then the emotion data is preprocessed.

[1526] Step 4:

[1527] The device adds metadata (time information, environmental conditions, etc.) to the preprocessed video data, and after preprocessing and adding the metadata, uploads this data to the server.

[1528] Step 5:

[1529] The server receives the pre-processed video data and queues it for processing. Machine learning algorithms are used to analyze the swing grip, ball placement, swing speed, ball trajectory, and body movement.

[1530] Step 6:

[1531] The server receives the pre-processed emotion data and uses emotion recognition algorithms to identify the user's emotional state, for example, assessing stress levels and concentration levels based on the user's facial expressions and heart rate.

[1532] Step 7:

[1533] The server generates the analysis results. From the swing data, it makes evaluations such as "weak grip" or "bad ball placement," and from the emotional data, it makes evaluations such as "high stress level" or "concentrated."

[1534] Step 8:

[1535] The server generates personalized feedback for each user based on the analysis results. The content and presentation of the feedback is adjusted according to the user's emotional state. For example, if the user is feeling stressed, the server will provide additional feedback such as "Relax and concentrate on your swing."

[1536] Step 9:

[1537] The server transmits the generated feedback data to the terminal. The feedback data includes data for displaying the ideal grip position and ball placement as visual feedback, and audio guidance as auditory feedback.

[1538] Step 10:

[1539] The terminal receives the feedback data and prepares it for transmission to the smart glasses, after which the feedback data is transferred to the smart glasses.

[1540] Step 11:

[1541] The smart glasses provide visual feedback (e.g., correct grip position and ball placement) on the display, and auditory feedback (voice messages) through a voice output device, instructing the user to "hold the grip tighter and adjust the ball placement to the right." They also provide additional advice based on the user's emotional state.

[1542] Step 12:

[1543] The user can then adjust their swing based on the feedback and reflect it in their next swing. Users can take advantage of real-time feedback to continually improve their swing.

[1544] Through these specific steps, users can receive real-time analysis of their swing and personalized feedback. The introduction of an emotion engine allows for more appropriate advice based on the user's psychological state, enabling efficient and effective improvement of swing technique.

[1545] Example 2

[1546] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1547] Conventional golf swing analysis systems can analyze a user's swing and provide feedback, but they cannot provide personalized feedback that takes the user's emotional state into account. As a result, the advice given ignores the user's psychological state, making it difficult to perform optimal practice. To solve this problem, a new system is required to measure the user's emotional state in real time and provide feedback that reflects this in the analysis.

[1548] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1549] In this invention, the server includes smart glasses, which are wearable devices worn by a user, a means for recording and measuring a golf swing and a user's emotional state in real time using a camera and a sensor mounted on the smart glasses, a terminal device for receiving the recorded video data and emotional data and trimming and formatting the data, a means for analyzing the preprocessed video data and emotional data and generating an evaluation and feedback based on the swing and the user's emotional state, and a means for providing the generated feedback visually and audibly to the smart glasses, thereby enabling personalized feedback that takes the user's emotional state into consideration.

[1550] "Smart glasses" are wearable devices worn by users that have built-in cameras and sensors and collect video data of golf swings and biometric data of the user.

[1551] The "camera" is a photographic device installed in the smart glasses that records the user's golf swing and facial expressions in real time.

[1552] A "sensor" is a measuring device installed in smart glasses that acquires biometric data such as the user's heart rate and skin galvanic response.

[1553] "Video data" refers to video data of a golf swing recorded by the smart glasses camera.

[1554] "Emotional data" refers to data that indicates the user's emotional state, acquired through the smart glasses' facial recognition camera, microphone, and biometric sensors.

[1555] A "terminal device" is a device that receives video data and emotion data sent from smart glasses and performs data preprocessing (trimming and format conversion).

[1556] A "server" is a device that analyzes the pre-processed video data and emotional data and generates evaluations and feedback based on swing and emotional state.

[1557] "Preprocessing" refers to the process of trimming and formatting data performed by the terminal device.

[1558] "Analysis" means the processing of data by the Server for the purposes of evaluating the data and generating feedback.

[1559] "Feedback" refers to visual and audible advice provided to the user based on the evaluation results generated by the server.

[1560] "Visual feedback" refers to visual information displayed on the smart glasses screen that shows corrections to your golf swing and the ideal swing form.

[1561] "Auditory feedback" refers to advice in the form of audio messages played through the audio output device of the smart glasses.

[1562] This invention uses smart glasses, a wearable device worn by the user, to collect video data of the golf swing, analyze it with an analysis system, and provide visual and auditory feedback.Furthermore, by combining it with an emotion engine, it is possible to provide more personalized feedback based on the user's emotional state.

[1563] Overall system configuration

[1564] 1. Smart Glasses:

[1565] Equipped with a camera: Records the user's swing in real time.

[1566] Facial expression recognition camera: Captures the user's facial expression data required for the emotion engine.

[1567] Microphone: Collects audio data.

[1568] Biometric sensors: measure biometric data such as heart rate and galvanic skin response.

[1569] Display: Provides visual feedback.

[1570] Audio output device: Provides auditory feedback.

[1571] 2. Terminal device (e.g., smartphone):

[1572] Receiving video data: Receive video data from the smart glasses.

[1573] Receiving emotional data: Receiving facial expressions, voice, and biometric data.

[1574] Data preprocessing: Trimming and format conversion of video data and emotion data.

[1575] Communication with the server: Send the preprocessed data to the server.

[1576] 3. Server Device:

[1577] Swing Data Analysis: Analyze grip, ball placement, swing speed, ball trajectory, and body movement.

[1578] Emotional data analysis: Recognize the user's emotional state and reflect it in the feedback.

[1579] Feedback generation: Generates user-specific visual and auditory feedback.

[1580] Sending feedback: Sending the generated feedback to the end device.

[1581] Program processing

[1582] Smart Glasses Processing

[1583] The system uses a camera to collect video data of the user's golf swing. At the same time, a facial recognition camera records the user's facial expressions, a microphone collects audio data, and biometric sensors capture biometric data such as heart rate and skin galvanic response.

[1584] The collected data is transferred to a terminal device.

[1585] Terminal processing

[1586] The video data and emotion data received from the smart glasses are saved and unnecessary parts are trimmed.

[1587] The video data is converted into an appropriate format, and metadata (time information, environmental conditions, etc.) is added to the emotion data.

[1588] The preprocessed data is uploaded to a server device.

[1589] Server device processing

[1590] The system analyzes pre-processed video data sent from the terminal device to identify and evaluate the swing grip, ball placement, swing speed, ball trajectory, and body movement.

[1591] Emotional data is analyzed to recognize the user's emotional state, for example, to determine whether the user is stressed or focused.

[1592] Generate feedback based on the analysis results. Dynamically adjust the content and presentation of the feedback depending on the user's emotional state. For example, if the user is feeling stressed, provide advice on how to relax.

[1593] The feedback content is transmitted to the terminal device.

[1594] Terminal device feedback processing

[1595] The feedback received from the server is prepared for transmission to the smart glasses and transferred to the smart glasses.

[1596] Feedback processing for smart glasses

[1597] Visual feedback (e.g., correct grip position and ball placement) is provided on the display.

[1598] Auditory feedback (voice messages) is played on the audio output device, instructing the user to, for example, "Hold a tighter grip and correct the ball placement to the right."

[1599] Specific examples

[1600] Beginner user use cases

[1601] 1. A user wears smart glasses and performs a golf swing. The smart glasses' camera records the swing and facial expressions, the microphone collects audio data, and the biometric sensor records the heart rate.

[1602] 2. The terminal device receives the recorded data and emotion data, preprocesses the data (trimming unnecessary parts, format conversion, adding metadata), and sends it to the server.

[1603] 3. The server analyzes the video data and generates feedback that reflects the user's emotional state, such as "Your grip is weak" or "The ball is placed too left." For example, if the user is feeling stressed, the server provides additional feedback such as "Relax and concentrate on your swing."

[1604] 4. The terminal device receives the feedback and sends it to the smart glasses.

[1605] 5. The smart glasses visually display the correct grip position and provide audio instructions such as "Hold a firmer grip and correct the ball placement to the right." If the user feels stressed, the glasses also provide additional advice such as "Relax and focus on your swing."

[1606] Intermediate User Use Cases

[1607] 1. The user wears the smart glasses while swinging. The camera records the swing, captures facial expression data, collects audio with a microphone, and records heart rate with a biometric sensor.

[1608] 2. The terminal device preprocesses the recorded data and emotion data and sends them to the server.

[1609] 3. The server analyzes the data and obtains the evaluation result that "the timing of raising the head is too early." In addition to feedback based on the evaluation result, feedback reflecting the user's emotional state is generated.

[1610] 4. The terminal device receives the feedback and sends it to the smart glasses.

[1611] 5. The smart glasses visually indicate the appropriate time to raise your head and provide audio instructions such as, "Be aware of your rhythm to stabilize your swing speed." If the user is concentrating too much, the smart glasses provide additional advice such as, "Keep your concentration and keep swinging."

[1612] Prompt Sentence Examples

[1613] Example prompts to be input to the generative AI model:

[1614] Your task is to write a program to operate smart glasses for a golf swing analysis system. The system will collect data by combining cameras, facial recognition, voice recognition, and biometric sensors to analyze the user's swing and emotional state. It will then provide appropriate feedback in real time. Through the process from data collection to feedback provision, we aim to improve the user's swing technique and care for their emotions.

[1615] This prompt statement enables the generative AI model to understand the overall flow of the system and generate the necessary program code.

[1616] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1617] Step 1:

[1618] The user wears smart glasses and performs a golf swing. The smart glasses collect video data of the user's golf swing using a built-in camera. At the same time, the facial recognition camera captures the user's facial expression data, and the microphone collects audio data. The smart glasses also use biometric sensors to capture biometric data such as heart rate and galvanic skin response. This data is stored in the smart glasses' internal memory.

[1619] input:

[1620] User's golf swing

[1621] User's facial expression

[1622] User Voice

[1623] User's biometric information

[1624] output:

[1625] Golf swing video data

[1626] User facial expression data

[1627] User voice data

[1628] User biometric data

[1629] Step 2:

[1630] All data collected by the smart glasses is transferred to the terminal device in real time using wireless communication such as Bluetooth or Wi-Fi.

[1631] input:

[1632] Golf swing video data

[1633] User facial expression data

[1634] User voice data

[1635] User biometric data

[1636] output:

[1637] Transferred video data

[1638] Transferred facial expression data

[1639] Transferred audio data

[1640] Transmitted biometric data

[1641] Step 3:

[1642] The terminal device stores the data received from the smart glasses and trims any unnecessary parts. For example, unnecessary parts before and after a swing are cut out. The video data is also converted into an appropriate format for analysis. Metadata such as time information and environmental conditions are added to the emotion data. After this preprocessing, the data is stored in the terminal device's internal memory.

[1643] input:

[1644] Transferred video data

[1645] Transferred facial expression data

[1646] Transferred audio data

[1647] Transmitted biometric data

[1648] output:

[1649] Video data after trimming

[1650] Format converted video data

[1651] Emotion data with added metadata

[1652] Step 4:

[1653] The terminal device uploads the pre-processed data to the server, which transfers the data via the Internet. Once uploaded, the data is stored on the server for analysis.

[1654] input:

[1655] Video data after trimming

[1656] Format converted video data

[1657] Emotion data with added metadata

[1658] output:

[1659] Preprocessed data stored on the server

[1660] Step 5:

[1661] The server analyzes the pre-processed video data and emotional data. It uses AI algorithms to identify and evaluate swing grip, ball placement, swing speed, ball trajectory, and body movement. It also analyzes the emotional data to recognize the user's emotional state (e.g., stress, concentration).

[1662] input:

[1663] Preprocessed video data

[1664] Emotion data with added metadata

[1665] output:

[1666] Swing data analysis results

[1667] Emotion data analysis results

[1668] Step 6:

[1669] The server generates personalized visual and auditory feedback for the user based on the analysis results. The content and presentation of the feedback are dynamically adjusted according to the user's emotional state. For example, if the user is feeling stressed, advice to help them relax is generated. The generated feedback is stored on the server.

[1670] input:

[1671] Swing data analysis results

[1672] Emotion data analysis results

[1673] output:

[1674] Individual user feedback

[1675] Step 7:

[1676] The server transmits the generated feedback to the terminal device, with data transfer occurring over the Internet.

[1677] input:

[1678] Individual user feedback

[1679] output:

[1680] Feedback sent to the device

[1681] Step 8:

[1682] The terminal device receives the feedback and prepares it for transmission to the smart glasses. The feedback is transferred to the smart glasses via Bluetooth or Wi-Fi.

[1683] input:

[1684] Feedback sent to the device

[1685] output:

[1686] Feedback transmitted to smart glasses

[1687] Step 9:

[1688] The smart glasses display visual feedback (e.g., correct grip position and ball placement) on the display, and also play auditory feedback (e.g., "Hold your grip tighter and correct the ball placement to the right") through an audio output device, allowing the user to receive feedback in real time.

[1689] input:

[1690] Feedback transmitted to smart glasses

[1691] output:

[1692] Visual and auditory feedback provided to the user

[1693] (Application example 2)

[1694] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1695] Currently, many factory robot and worker motion analysis systems have difficulty detecting motion errors and efficiency declines in real time and providing appropriate feedback. Furthermore, few systems consider the psychological state and stress level of workers to improve their motion. Therefore, there is a need for more effective and adaptive feedback systems to improve production efficiency and reduce errors.

[1696] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1697] In this invention, the server includes a means for evaluating and generating feedback on operations, placement, and efficiency, a means for analyzing emotional data and recognizing the stress and concentration of workers, and a means for providing the generated feedback visually and audibly. This makes it possible to analyze the operations of factory robots and workers in real time and provide appropriate feedback. It is also possible to provide more personalized feedback by taking into account the psychological state of the workers.

[1698] A "user" is a person or operator who uses a motion analysis system for golf swings or robotic work.

[1699] A "wearable device" is a device that can be worn by a user and is capable of collecting and displaying data.

[1700] "Smart glasses" are wearable devices equipped with cameras, sensors, displays, etc., that function as an interface for visual and auditory data.

[1701] A "camera" is a device that takes and records images and videos.

[1702] A "golf swing" refers to a golf hitting motion, and is an act for collecting data on a user's swing motion.

[1703] "Real time" refers to processing events as they occur, without any time lag.

[1704] "Recording means" refers to a method or device for recording video data using a camera mounted on smart glasses.

[1705] "Video data" refers to data that displays a series of images recorded by a camera in a time-series sequence.

[1706] "Emotion data" is data that indicates the emotional state of the user based on biological reactions such as the user's heart rate and facial expressions.

[1707] "Preprocessing" refers to the process of processing or converting collected data to make it easier to analyze.

[1708] A "terminal device" is an electronic device capable of data communication and processing, including smartphones and tablets.

[1709] A "server device" is a computer system for storing, managing, and analyzing data.

[1710] "Evaluation" is the act of judging the quality of the results of data analysis based on specific criteria.

[1711] "Feedback" refers to instructions or advice provided to the user based on the evaluation results.

[1712] "Visual feedback" is information provided to a user using a visual medium such as a display.

[1713] "Auditory feedback" refers to spoken messages or instructions provided to a user through an audio output device.

[1714] "Efficiency of action" is a measure of how effectively an action is performed to achieve a goal.

[1715] "Layout" refers to the state of how things or people are arranged.

[1716] An "error rate" is the rate at which errors occur during a particular task or activity.

[1717] "Physical responses" are biometric data that indicate the user's physical changes and condition.

[1718] System Configuration

[1719] Hardware

[1720] 1. Smart Glasses

[1721] camera

[1722] Facial Recognition Camera

[1723] microphone

[1724] Biometric Sensor

[1725] display

[1726] Audio output device

[1727] 2. Terminal Device

[1728] Smartphone or tablet

[1729] 3. Server Device

[1730] Computer system for data analysis

[1731] software

[1732] 1. Video analysis software

[1733] (e.g. OpenCV)

[1734] 2. Facial Recognition Software

[1735] 3. Audio Analysis Software

[1736] 4. Biological Data Analysis Software

[1737] 5. Emotion Engine

[1738] 6. Feedback Generator

[1739] Overall system processing

[1740] 1. Data Collection and Preprocessing

[1741] Smart Glasses:

[1742] The camera installed in the smart glasses captures the user's work and movements in real time.

[1743] A facial expression recognition camera captures the user's facial expression data, and a microphone collects voice data.

[1744] Biosensors are used to measure biometric data such as heart rate and skin galvanic response.

[1745] Terminal device:

[1746] Receive video data and emotion data transmitted from the smart glasses.

[1747] Trim out any unwanted parts and convert to the appropriate format.

[1748] The preprocessed data is transferred to a server device.

[1749] 2. Data analysis and feedback generation

[1750] Server device:

[1751] Using video analysis software, we analyze the efficiency, placement, and error rate of movements.

[1752] Using facial expression recognition software and biometric data analysis software, the system analyzes the user's emotional state, thereby identifying the user's stress and concentration level.

[1753] The emotional engine generates feedback based on the results of motion analysis, dynamically adjusting the visual and auditory feedback according to the user's emotional state.

[1754] Generate feedback:

[1755] Based on the analysis results, feedback is generated that provides visual and auditory indications of work improvements and points to note.

[1756] The server transmits the feedback content to the terminal device.

[1757] 3. Providing Feedback

[1758] Terminal device:

[1759] The received feedback is forwarded to the smart glasses.

[1760] Smart Glasses:

[1761] Visual feedback is provided on the display, such as ideal operating positions and areas requiring attention.

[1762] The auditory feedback is played on an audio output device to instruct the user as a voice message.

[1763] Specific examples

[1764] Use cases for beginner users:

[1765] A user wears smart glasses and works on a factory production line. The smart glasses' camera captures their movements and facial expressions, the microphone collects audio data, and the biometric sensor records their heart rate. The terminal device preprocesses this data and sends it to a server. Based on the analysis results, the server provides visual and auditory feedback on areas for improvement and important points to note.

[1766] Example prompt sentence:

[1767] "Analyze the robot's movements during work and generate feedback to maximize efficiency. Also, recognize the worker's stress and concentration levels and provide advice based on that."

[1768] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1769] Step 1:

[1770] Data collection using smart glasses

[1771] Input: User movements, facial expressions, voice data, biometric data (heart rate, skin galvanic response)

[1772] How it works: A camera on the smart glasses captures the user's movements in real time, a facial recognition camera records the user's facial expressions, a microphone collects audio data, and biometric sensors measure heart rate and galvanic skin response.

[1773] Output: Acquisition of video data, facial expression data, audio data, and biometric data

[1774] Step 2:

[1775] Data reception and preprocessing by the terminal device

[1776] Input: Video data, facial expression data, audio data, and biometric data sent from smart glasses

[1777] Specific operation: The terminal device (smartphone or tablet) receives the data sent from the smart glasses, trims unnecessary parts, converts the video data into an appropriate format, and adds metadata (time information, environmental conditions, etc.) to the emotion data.

[1778] Output: Preprocessed video data and emotion data

[1779] Step 3:

[1780] Data transmission from terminal device to server

[1781] Input: Preprocessed video data and emotion data

[1782] Specific operations: The terminal device uploads the preprocessed data to the server device.

[1783] Output: Preprocessed data sent to the server

[1784] Step 4:

[1785] Data analysis by server

[1786] Input: Preprocessed data sent from the terminal

[1787] Specific operations: The server analyzes the video data and evaluates the efficiency, placement, and error rate of the movements. It also uses facial expression recognition software and biometric data analysis software to analyze the user's emotional data and recognize their stress and concentration levels.

[1788] Output: Movement evaluation results and emotional state analysis results

[1789] Step 5:

[1790] Feedback Generation

[1791] Input: Movement evaluation results and emotional state analysis results

[1792] Specific behavior: The server utilizes an emotion engine to generate feedback based on the behavior analysis results and the user's emotional state. The visual and auditory feedback content is dynamically adjusted.

[1793] Output: Visual and auditory feedback

[1794] Step 6:

[1795] Sending feedback to terminal devices

[1796] Input: Server-generated feedback

[1797] Specific operations: The server device transmits the generated feedback to the terminal device.

[1798] Output: Feedback sent to the terminal

[1799] Step 7:

[1800] Sending feedback from the device to the smart glasses

[1801] Input: Feedback sent to the terminal

[1802] Specific operation: The terminal device transmits the feedback content to the smart glasses.

[1803] Output: Feedback sent to the smart glasses

[1804] Step 8:

[1805] Feedback provided by smart glasses

[1806] Input: Feedback sent to the smart glasses

[1807] Specific operation: Smart glasses display visual feedback on the display and play auditory feedback through the audio output device. For example, instructions for correcting movement placement or advice on preventing mistakes are provided via both visual and audio feedback.

[1808] Output: The visual and auditory feedback the user receives.

[1809] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1810] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1811] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1812] [Fourth embodiment]

[1813] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1814] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1815] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1816] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1817] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1818] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1819] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1820] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1821] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1822] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1823] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1824] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1825] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1826] The golf play assistance service of the present invention uses smart glasses worn by the user to collect video data of the golf swing in real time, analyzes it with an analysis system, and provides visual and auditory feedback. The system consists of smart glasses, a terminal device (such as a smartphone), and a server device.

[1827] Overall system configuration

[1828] 1. Smart Glasses:

[1829] Equipped with a camera: Records your swing from your perspective.

[1830] Display: Provides visual feedback.

[1831] Audio output device: Provides auditory feedback.

[1832] 2. Terminal device (e.g., smartphone):

[1833] Receiving data: Receives recording data from smart glasses.

[1834] Data preprocessing: trimming and formatting data.

[1835] Communication with the server: Send the preprocessed data to the server.

[1836] 3. Server Device:

[1837] Data Analysis: Analyze each element of your swing to generate an evaluation and feedback.

[1838] Sending feedback: Sending the generated feedback back to the end device.

[1839] Program processing

[1840] 1. Smart Glasses Processing:

[1841] A camera is used to collect video data of a user's golf swing.

[1842] The collected video data is transferred to the terminal device.

[1843] 2. Terminal Processing:

[1844] Receives video data transferred from smart glasses.

[1845] Trim unnecessary parts and convert the data format into one that is easy to analyze.

[1846] The preprocessed data is sent to a server device.

[1847] 3. Processing of server equipment:

[1848] The preprocessed video data transmitted from the terminal device is received.

[1849] Machine learning algorithms are used to analyze swing grip, ball placement, swing speed, ball trajectory, and body movement.

[1850] Based on the analysis results, feedback is generated for each user.

[1851] The feedback content is transmitted to the terminal device.

[1852] 4. Terminal device feedback processing:

[1853] Process the feedback received from the server.

[1854] Sending the processed feedback to the smart glasses.

[1855] 5. Smart Glasses Feedback Processing:

[1856] Visual feedback (ideal grip position and ball placement) is displayed on the screen.

[1857] Auditory feedback (voice messages) is played on an audio output device.

[1858] Specific examples

[1859] Beginner user use cases

[1860] User:

[1861] A beginner user wears smart glasses and performs a golf swing. The camera in the smart glasses records the swing and transmits the data to a smartphone in real time.

[1862] Terminal device:

[1863] The smartphone receives the recorded data, performs pre-processing, and then sends the data to the server. Pre-processing includes trimming unnecessary parts and format conversion.

[1864] Server device:

[1865] The server analyzes the pre-processed data and generates evaluation results such as "weak grip" or "ball placement too left." Based on the analysis results, specific feedback is generated and sent to the smartphone.

[1866] Terminal device:

[1867] The smartphone receives the feedback and sends it to the smart glasses.

[1868] Smart Glasses:

[1869] The smart glasses visually display the correct grip position on the HUD and provide audio advice such as "Hold a firm grip and adjust the ball placement to the right."

[1870] Intermediate User Use Cases

[1871] User:

[1872] An intermediate golfer wears the smart glasses while swinging. The camera records the swing and sends the data to a smartphone.

[1873] Terminal device:

[1874] The smartphone pre-processes the recorded data and sends it to the server.

[1875] Server device:

[1876] The server analyzes the preprocessed data and obtains evaluation results such as "the timing of lifting the head is too early" and "the swing speed is unstable." Feedback based on the evaluation results is generated and sent to the smartphone.

[1877] Terminal device:

[1878] The smartphone receives the feedback and sends it to the smart glasses.

[1879] Smart Glasses:

[1880] The smart glasses visually display the appropriate timing to raise your head on the HUD and provide audio guidance such as "be aware of rhythm to stabilize your swing speed."

[1881] Through these processes, the user can understand in real time what corrections to make to their swing and effectively improve their technique.

[1882] The processing flow will be explained below.

[1883] Step 1:

[1884] The user wears the smart glasses and performs a golf swing, with the camera in the smart glasses recording the user's swing in real time.

[1885] Step 2:

[1886] The swing video data recorded by the smart glasses is transferred to the device via Wi-Fi or Bluetooth.

[1887] Step 3:

[1888] The device saves the video data received from the smart glasses. Once saved, it performs pre-processing such as trimming unnecessary parts and converting the data format.

[1889] Step 4:

[1890] The device adds metadata (time information, environmental conditions, etc.) to the preprocessed video data. After preprocessing and adding metadata, the data is uploaded to the server.

[1891] Step 5:

[1892] The server receives the pre-processed video data and queues it for processing. Machine learning algorithms are used to analyze each element of the swing (grip, ball placement, swing speed, ball trajectory, and body movement).

[1893] Step 6:

[1894] The server generates an analysis result, which may include evaluations such as "weak grip" or "bad ball placement," as well as specific suggestions such as "unstable swing speed" or "head lifting timing too early."

[1895] Step 7:

[1896] Based on the analysis results, the server generates individualized feedback for each user, including visual feedback such as data to display the ideal grip position and ball placement, and audio guidance as auditory feedback.

[1897] Step 8:

[1898] The server transmits the generated feedback data to the terminal.

[1899] Step 9:

[1900] The terminal prepares the received feedback data for transmission to the smart glasses and transfers it to the smart glasses.

[1901] Step 10:

[1902] The smart glasses provide visual feedback (e.g., correct grip position and ball placement) on the display.

[1903] Step 11:

[1904] The smart glasses play auditory feedback (voice messages) through an audio output device, instructing the user, for example, to "tighten the grip and adjust the ball placement to the right."

[1905] Step 12:

[1906] The user can then adjust their swing based on the feedback and reflect it in their next swing. Users can take advantage of real-time feedback to continually improve their swing.

[1907] Through these specific steps, users can analyze their swing in real time and receive effective feedback, allowing them to improve their swing technique efficiently, regardless of location.

[1908] Example 1

[1909] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1910] There is a need for a system that can provide real-time feedback to improve golf swing technique. However, conventional systems lack coordination between the data collection, preprocessing, analysis, and feedback stages, making it difficult to provide effective training. Another issue is that it is difficult for users to specifically identify which elements of their swing need improvement.

[1911] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1912] In this invention, the server includes smart glasses, which are wearable devices worn by a user, a means for recording a golf swing in real time using a camera mounted on the smart glasses, a terminal device for receiving the recorded video data, trimming unnecessary portions, and converting the data format, a server device equipped with a machine learning algorithm for analyzing the preprocessed video data, analyzing each element of the swing, evaluating the swing, and generating feedback, and a means for providing the generated feedback visually and audibly to the smart glasses, thereby enabling the user to receive specific and effective feedback in real time.

[1913] "Smart glasses" are wearable devices worn by users that have visual and auditory interfaces.

[1914] The "camera" is a device installed in the smart glasses that records the golf swing from the user's perspective in real time.

[1915] "Recorded data" refers to video data of golf swings collected by the smart glasses camera.

[1916] The "terminal device" is a device that receives video data sent from the smart glasses, trims unnecessary parts, and converts the data into a format that is easy to analyze.

[1917] "Preprocessing" refers to the process of trimming unnecessary parts of the recorded data and converting the data format into one that is easier to analyze.

[1918] The "server device" is a computer system that analyzes the pre-processed video data, analyzes each element of the swing, and generates an evaluation and feedback.

[1919] A "machine learning algorithm" is a program used by the server device, and is a technology for analyzing factors such as swing grip, ball placement, swing speed, ball trajectory, and body movement.

[1920] "Feedback" refers to advice and evaluation information for each user that is generated based on the analysis results and is provided in visual and auditory formats.

[1921] "Visual feedback" refers to information such as the ideal grip position and ball placement that is displayed on the smart glasses display.

[1922] "Auditory feedback" refers to an audio message played through the audio output device of the smart glasses.

[1923] MODE FOR CARRYING OUT THE INVENTION

[1924] This golf play assistance service uses smart glasses worn by the user to collect video data of the golf swing in real time, analyzes it with an analysis system, and provides visual and auditory feedback. The system consists of smart glasses, a terminal device, and a server device.

[1925] Smart Glasses

[1926] Hardware: The smart glasses are equipped with a camera to record the user's swing from their perspective, and also have a display and audio output to provide visual and audio feedback.

[1927] Specific features:

[1928] A camera is used to collect video data of the user's golf swing and transfer it to a terminal device in real time.

[1929] The collected video data is transferred to a terminal device using wireless communication means such as Wi-Fi.

[1930] The analyzed feedback is displayed visually on a display and audibly reproduced through an audio output device.

[1931] terminal device

[1932] Hardware: Terminal devices are general-purpose devices such as smartphones and tablets.

[1933] Software: An application is installed to preprocess video data and communicate with the server.

[1934] Specific features:

[1935] Receives video data sent from smart glasses.

[1936] The received video data is trimmed and converted into a format suitable for analysis (e.g., MP4 format).

[1937] The pre-processed video data is encrypted and sent to a server via the Internet.

[1938] Server device

[1939] Hardware: A high-performance computer system with the processing power to analyze large amounts of data.

[1940] Software: Includes analysis programs that implement machine learning algorithms (e.g., TensorFlow, PyTorch).

[1941] Specific features:

[1942] The preprocessed video data transmitted from the terminal device is received.

[1943] Machine learning algorithms are used to analyze swing grip, ball placement, swing speed, ball trajectory, and body movement.

[1944] Generate evaluation and feedback based on the analysis results.

[1945] The generated feedback is transmitted to the terminal device.

[1946] Specific examples

[1947] Beginner user use cases

[1948] User: A beginner user wears the smart glasses and performs a golf swing. The smart glasses' camera records the swing and transmits the data to a smartphone in real time.

[1949] Terminal device: The smartphone receives the recorded data, performs pre-processing, and then sends the data to the server. Pre-processing includes trimming unnecessary parts and format conversion.

[1950] Server device: The server analyzes the pre-processed data and generates evaluation results such as "the grip is weak" or "the ball is placed too left." Based on the analysis results, specific feedback is generated and sent to the smartphone.

[1951] Terminal device: The smartphone receives the feedback and sends it to the smart glasses.

[1952] Smart glasses: The smart glasses visually show the correct grip position on the display and provide audio advice such as "Hold a firmer grip and adjust the ball placement to the right."

[1953] Prompt Sentence Examples

[1954] "Please explain in natural language the process of analyzing a video of a beginner golfer's swing and providing feedback on areas for improvement."

[1955] In this way, the user can understand in real time what corrections to make to their swing and effectively improve their technique.

[1956] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1957] Step 1: Collecting video data with smart glasses

[1958] User: The user wears the smart glasses and performs a golf swing.

[1959] Input: The user's golf swing motion.

[1960] How it works: The smart glasses' camera records the entire swing from the user's perspective.

[1961] Output: Video data stored inside the smart glasses.

[1962] Step 2: Transferring video data from smart glasses to your device

[1963] Smart glasses: Smart glasses transfer recorded video data to the terminal device.

[1964] Input: Video data stored on smart glasses.

[1965] Specific operation: Smart glasses transmit video data to terminal devices in real time via wireless communication such as Wi-Fi.

[1966] Output: Video data transferred to the terminal device.

[1967] Step 3: Receiving and preprocessing video data on the terminal device

[1968] Terminal: The terminal device receives the video data transmitted from the smart glasses.

[1969] Input: Video data sent from smart glasses.

[1970] Specific operation: Trim unnecessary parts of the video data and convert it into a format that is easy to analyze (e.g., MP4 format).

[1971] Output: Preprocessed video data.

[1972] Step 4: Sending data from the terminal to the server

[1973] Terminal: The terminal device transmits the preprocessed video data to the server device.

[1974] Input: Preprocessed video data.

[1975] Specific operation: The terminal device encrypts data and sends it to the server via the Internet.

[1976] Output: Preprocessed video data transferred to the server.

[1977] Step 5: Video data analysis by the server

[1978] Server: The server device analyzes the received preprocessed video data.

[1979] Input: Preprocessed video data.

[1980] Specific behavior: Machine learning algorithms are used to analyze swing grip, ball placement, swing speed, ball trajectory, and body movement.

[1981] Output: Analysis results and feedback information.

[1982] Step 6: Sending feedback from the server to the terminal device

[1983] Server: The server device sends the generated feedback to the terminal device.

[1984] Input: Analysis results and feedback information.

[1985] Specific operations: The feedback information is formatted and sent to the terminal device via the Internet.

[1986] Output: Feedback information transmitted to the terminal device.

[1987] Step 7: End device processes feedback and sends it to the smart glasses

[1988] Terminal: The terminal device processes the feedback information received from the server and sends it to the smart glasses.

[1989] Input: Feedback information received from the server.

[1990] What it does: Feedback information is optimized as visual and auditory feedback and wirelessly transmitted to the smart glasses.

[1991] Output: Feedback information transmitted to the smart glasses.

[1992] Step 8: Providing feedback with smart glasses

[1993] Smart Glasses: Smart glasses provide the received feedback to the user.

[1994] Input: Feedback information sent from the terminal device.

[1995] What it does: Visual feedback is shown on the display, and auditory feedback is played through the audio output device.

[1996] Output: The feedback presented to the user.

[1997] (Application example 1)

[1998] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1999] Accuracy and efficiency are required for robot operations within factories. In particular, if a transport robot delivers packages incorrectly or operates improperly, it can affect overall production efficiency. The purpose of this invention is to solve these issues by analyzing robot operations within factories in real time and providing rapid feedback.

[2000] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[2001] In this invention, the server includes a means for generating an evaluation and feedback for the operation, a means for providing the generated feedback visually and audibly to the smart glasses, and a means for improving the operation of the robot in the factory, thereby making it possible to analyze the operation of the robot in the factory in real time and provide appropriate feedback.

[2002] "User" refers to the person who wears and uses this system.

[2003] "Wearable device" refers to a device that can be worn by a user, in this case smart glasses.

[2004] "Smart glasses" refers to eyeglass-type devices equipped with cameras, displays, etc.

[2005] "Camera" refers to a device that captures images or videos.

[2006] "Movement" refers to the series of movements performed by a robot in a factory.

[2007] "Real-time" refers to data being processed immediately, without delay.

[2008] "Recording" refers to the recording of an action using a camera.

[2009] "Means" refers to a method or apparatus for performing a particular function or process.

[2010] "Video data" refers to video information of recorded actions.

[2011] "Receiving" refers to receiving data.

[2012] "Preprocessing" refers to the process of changing data into a form suitable for analysis.

[2013] "Terminal device" refers to a device that receives and pre-processes data, such as a smartphone.

[2014] "Server device" refers to a device that analyzes data and generates feedback.

[2015] "Evaluation" refers to the analysis results of recorded and analyzed behavior.

[2016] "Feedback" refers to suggestions for improvement or instructions provided based on the evaluation results.

[2017] "Visual" refers to providing information in a form that can be seen by the user, such as through a display.

[2018] "Auditory" refers to providing information to the user in an audible form, such as through voice.

[2019] "Accuracy" refers to the degree to which an action is performed as planned.

[2020] "Efficiency" refers to the degree to which resources are used optimally to obtain maximum results.

[2021] "Pickup position" refers to the correct position where the robot will pick up the luggage.

[2022] "Smoothness of movement" refers to the degree to which a robot can perform movements continuously and without interruption.

[2023] This invention relates to a system that monitors and analyzes the operation of robots in factories in real time and provides feedback. This system consists of smart glasses, which are wearable devices worn by users, a camera mounted on the smart glasses, a terminal device that receives and preprocesses the data, and a server device that analyzes the data.

[2024] System configuration

[2025] 1. Smart Glasses

[2026] The smart glasses, a wearable device worn by the user, are equipped with a camera that can record the robot's movements in real time, and also have a display and audio output device that provide visual and auditory feedback.

[2027] 2. Terminal Device

[2028] The recorded video data is sent to a terminal device (e.g., a smartphone). This terminal device performs preprocessing, trimming unnecessary parts and converting the data format to one that is easier to analyze. The preprocessed data is then sent to a server.

[2029] 3. Server Device

[2030] The server analyzes the received preprocessed video data and evaluates the factory robot's operations. Specifically, it has the ability to analyze accuracy, efficiency, and the smoothness of the pickup position and operation of the goods. This is done using machine learning algorithms such as OpenCV and TensorFlow. Based on the evaluation results, it generates feedback to be provided to the user. This feedback is then sent to the terminal device.

[2031] 4. Providing Feedback

[2032] The terminal device transmits the feedback received from the server to the smart glasses, which provide visual feedback (e.g., displaying the correct action position on the HUD) and auditory feedback (e.g., a voice command saying "Proceed to the next action").

[2033] Specific examples

[2034] For example, suppose a delivery robot operating in a factory frequently mis-delivers packages. The smart glasses worn by the user record the robot's movements in real time, and after preprocessing by a terminal device, send the data to a server. The server analyzes the robot's movement data and obtains the evaluation result that "the package is not being picked up in the correct position." Based on this evaluation result, the smart glasses visually display "correct the position to pick up the package correctly" and provide a voice instruction to "proceed to the next action."

[2035] Prompt Sentence Examples

[2036] Prompt: "Generate code for a robotic behavior analysis system in a factory. The server will be implemented in Python and will use the OpenCV and TensorFlow libraries to analyze the robotic behavior. The smart glasses will use Google Glass to provide real-time feedback. The end device will be an Android smartphone."

[2037] In this way, the present invention can improve robotic operation within a factory, increasing overall production efficiency.

[2038] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[2039] Step 1:

[2040] The smart glasses worn by the user use a built-in camera to record the factory robot's operations in real time. In this step, the smart glasses capture the operation video and transfer it to a terminal device as video data. The input is the factory robot's operations, and the output is the recorded video data.

[2041] Step 2:

[2042] The terminal device receives the video data sent from the smart glasses. The terminal device then preprocesses the video data. Specifically, it trims unnecessary parts and converts them into a data format suitable for analysis. The input is the recorded video data, and the output is the preprocessed video data.

[2043] Step 3:

[2044] The terminal device transmits the preprocessed video data to the server device. The server device has a means for analyzing the received video data and evaluates the accuracy, efficiency, package pickup position, and smoothness of the robot's movements from the video data. The input is the preprocessed video data, and the output is the evaluation results.

[2045] Step 4:

[2046] The server device uses machine learning algorithms (e.g., OpenCV and TensorFlow) to evaluate the robot's movements. Specifically, the motion analysis algorithm analyzes the video data and identifies the accuracy of each movement and areas for improvement. The input is the preprocessed video data, and the output is the evaluation results.

[2047] Step 5:

[2048] The server generates feedback based on the evaluation results. Using a generative AI model, it creates specific feedback content to provide to the user. In this case, both visual and auditory feedback is generated. The input is the evaluation results, and the output is the feedback content.

[2049] Step 6:

[2050] The server device sends the generated feedback content to the terminal device. The terminal device sends the received feedback to the smart glasses and presents it to the user in an appropriate manner. At this stage, the input is the feedback content and the output is the feedback transfer to the smart glasses.

[2051] Step 7:

[2052] The smart glasses provide feedback to the user using a display device and an audio output device. Visual feedback shows the correct operating position on the display, and audio feedback indicates the next action. The input is the feedback content sent from the terminal device, and the output is the visual and audio feedback received by the user.

[2053] In this way, a system is created that analyzes the operation of robots in a factory in real time and provides rapid feedback.

[2054] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[2055] The present invention uses smart glasses, a wearable device worn by the user, to collect video data of the golf swing, which is then analyzed by an analysis system to provide visual and auditory feedback. Furthermore, by combining this with an emotion engine that recognizes the user's emotional state, more personalized advice can be provided.

[2056] Overall system configuration

[2057] 1. Smart Glasses:

[2058] Equipped with a camera: Records the user's swing in real time.

[2059] Facial expression recognition camera: Captures the user's facial expression data required for the emotion engine.

[2060] Microphone: Collects audio data.

[2061] Biometric sensors: Measure heart rate, skin galvanic response, etc.

[2062] Display: Provides visual feedback.

[2063] Audio output device: Provides auditory feedback.

[2064] 2. Terminal device (e.g., smartphone):

[2065] Receiving video data: Receive video data from the smart glasses.

[2066] Receiving emotional data: Receiving facial expressions, voice, and biometric data.

[2067] Data preprocessing: Trimming and format conversion of video data and emotion data.

[2068] Communication with the server: Send the preprocessed data to the server.

[2069] 3. Server Device:

[2070] Swing Data Analysis: Analyze grip, ball placement, swing speed, ball trajectory, and body movement.

[2071] Emotional data analysis: Recognize the user's emotional state and reflect it in the feedback.

[2072] Feedback generation: Generates user-specific visual and auditory feedback.

[2073] Sending feedback: Sending the generated feedback to the end device.

[2074] Program processing

[2075] Smart Glasses Processing

[2076] The system uses a camera to collect video data of the user's golf swing. At the same time, a facial recognition camera records the user's facial expressions, a microphone collects audio data, and biometric sensors capture biometric data such as heart rate and skin galvanic response.

[2077] The collected data is transferred to a terminal device.

[2078] Terminal processing

[2079] The video data and emotion data received from the smart glasses are saved and unnecessary parts are trimmed.

[2080] The video data is converted into an appropriate format, and metadata (time information, environmental conditions, etc.) is added to the emotion data.

[2081] The preprocessed data is uploaded to a server device.

[2082] Server device processing

[2083] The system analyzes pre-processed video data sent from the terminal device to identify and evaluate the swing grip, ball placement, swing speed, ball trajectory, and body movement.

[2084] Emotional data is analyzed to recognize the user's emotional state, for example, to determine whether the user is stressed or focused.

[2085] Generate feedback based on the analysis results. Dynamically adjust the content and presentation of the feedback depending on the user's emotional state. For example, if the user is feeling stressed, provide advice on how to relax.

[2086] The feedback content is transmitted to the terminal device.

[2087] Terminal device feedback processing

[2088] The feedback received from the server is prepared for transmission to the smart glasses and transferred to the smart glasses.

[2089] Feedback processing for smart glasses

[2090] Visual feedback (e.g., correct grip position and ball placement) is provided on the display.

[2091] Auditory feedback (voice messages) is played on the audio output device, instructing the user to, for example, "Hold a tighter grip and correct the ball placement to the right."

[2092] Specific examples

[2093] Beginner user use cases

[2094] User:

[2095] A beginner wears smart glasses and performs a golf swing. The smart glasses' camera records the swing and facial expressions, the microphone collects audio data, and the biometric sensor records the heart rate.

[2096] Terminal device:

[2097] The smartphone receives the recording data and emotion data, and after preprocessing, sends the data to the server.

[2098] Server device:

[2099] The server analyzes the video data and generates feedback that reflects the user's emotional state, such as "Your grip is weak" or "The ball is placed too left." For example, if the user is feeling stressed, the server provides additional feedback such as "Relax and concentrate on your swing."

[2100] Terminal device:

[2101] The smartphone receives the feedback and sends it to the smart glasses.

[2102] Smart Glasses:

[2103] The smart glasses visually display the correct grip position and provide audio instructions such as "Hold a firmer grip and adjust the ball placement to the right," and if the user feels stressed, they can offer additional advice such as "Relax and focus on your swing."

[2104] Intermediate User Use Cases

[2105] User:

[2106] Intermediate golfers wear smart glasses while swinging. A camera records the swing, captures facial expression data, a microphone collects audio, and a biometric sensor records heart rate.

[2107] Terminal device:

[2108] The smartphone preprocesses the recorded data and emotion data and sends them to the server.

[2109] Server device:

[2110] The server analyzes the data and obtains the evaluation result that "the timing of raising the head is too early." In addition to feedback based on the evaluation result, feedback reflecting the user's emotional state is generated.

[2111] Terminal device:

[2112] The smartphone receives the feedback and sends it to the smart glasses.

[2113] Smart Glasses:

[2114] The smart glasses visually indicate the appropriate time to lift your head and provide audio instructions such as, "Be aware of your rhythm to stabilize your swing speed." If the user is concentrating too much, the glasses provide additional advice such as, "Keep your concentration and keep swinging."

[2115] Through these processes, users can analyze their swing in real time and receive personalized feedback. The introduction of an emotion engine allows for more appropriate advice based on the user's psychological state, enabling efficient and effective improvement of swing technique.

[2116] The processing flow will be explained below.

[2117] Step 1:

[2118] The user wears the smart glasses and performs a golf swing. The smart glasses' camera records the user's swing in real time, the facial recognition camera records the user's facial expressions, biometric sensors measure the user's heart rate and galvanic skin response, and a microphone collects audio.

[2119] Step 2:

[2120] The video data of the swing recorded by the smart glasses, as well as emotional data such as facial expressions, voice, and heart rate, are transferred to the device via Wi-Fi or Bluetooth.

[2121] Step 3:

[2122] The device stores the video data and emotion data received from the smart glasses. The stored data is preprocessed, such as trimming unnecessary parts and converting the data format. The video data is processed first, and then the emotion data is preprocessed.

[2123] Step 4:

[2124] The device adds metadata (time information, environmental conditions, etc.) to the preprocessed video data, and after preprocessing and adding the metadata, uploads this data to the server.

[2125] Step 5:

[2126] The server receives the pre-processed video data and queues it for processing. Machine learning algorithms are used to analyze the swing grip, ball placement, swing speed, ball trajectory, and body movement.

[2127] Step 6:

[2128] The server receives the pre-processed emotion data and uses emotion recognition algorithms to identify the user's emotional state, for example, assessing stress levels and concentration levels based on the user's facial expressions and heart rate.

[2129] Step 7:

[2130] The server generates the analysis results. From the swing data, it makes evaluations such as "weak grip" or "bad ball placement," and from the emotional data, it makes evaluations such as "high stress level" or "concentrated."

[2131] Step 8:

[2132] The server generates personalized feedback for each user based on the analysis results. The content and presentation of the feedback is adjusted according to the user's emotional state. For example, if the user is feeling stressed, the server will provide additional feedback such as "Relax and concentrate on your swing."

[2133] Step 9:

[2134] The server transmits the generated feedback data to the terminal. The feedback data includes data for displaying the ideal grip position and ball placement as visual feedback, and audio guidance as auditory feedback.

[2135] Step 10:

[2136] The terminal receives the feedback data and prepares it for transmission to the smart glasses, after which the feedback data is transferred to the smart glasses.

[2137] Step 11:

[2138] The smart glasses provide visual feedback (e.g., correct grip position and ball placement) on the display, and auditory feedback (voice messages) through a voice output device, instructing the user to "hold the grip tighter and adjust the ball placement to the right." They also provide additional advice based on the user's emotional state.

[2139] Step 12:

[2140] The user can then adjust their swing based on the feedback and reflect it in their next swing. Users can take advantage of real-time feedback to continually improve their swing.

[2141] Through these specific steps, users can receive real-time analysis of their swing and personalized feedback. The introduction of an emotion engine allows for more appropriate advice based on the user's psychological state, enabling efficient and effective improvement of swing technique.

[2142] Example 2

[2143] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[2144] Conventional golf swing analysis systems can analyze a user's swing and provide feedback, but they cannot provide personalized feedback that takes the user's emotional state into account. As a result, the advice given ignores the user's psychological state, making it difficult to perform optimal practice. To solve this problem, a new system is required to measure the user's emotional state in real time and provide feedback that reflects this in the analysis.

[2145] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[2146] In this invention, the server includes smart glasses, which are wearable devices worn by a user, a means for recording and measuring a golf swing and a user's emotional state in real time using a camera and a sensor mounted on the smart glasses, a terminal device for receiving the recorded video data and emotional data and trimming and formatting the data, a means for analyzing the preprocessed video data and emotional data and generating an evaluation and feedback based on the swing and the user's emotional state, and a means for providing the generated feedback visually and audibly to the smart glasses, thereby enabling personalized feedback that takes the user's emotional state into consideration.

[2147] "Smart glasses" are wearable devices worn by users that have built-in cameras and sensors and collect video data of golf swings and biometric data of the user.

[2148] The "camera" is a photographic device installed in the smart glasses that records the user's golf swing and facial expressions in real time.

[2149] A "sensor" is a measuring device installed in smart glasses that acquires biometric data such as the user's heart rate and skin galvanic response.

[2150] "Video data" refers to video data of a golf swing recorded by the smart glasses camera.

[2151] "Emotional data" refers to data that indicates the user's emotional state, acquired through the smart glasses' facial recognition camera, microphone, and biometric sensors.

[2152] A "terminal device" is a device that receives video data and emotion data sent from smart glasses and performs data preprocessing (trimming and format conversion).

[2153] A "server" is a device that analyzes the pre-processed video data and emotional data and generates evaluations and feedback based on swing and emotional state.

[2154] "Preprocessing" refers to the process of trimming and formatting data performed by the terminal device.

[2155] "Analysis" means the processing of data by the Server for the purposes of evaluating the data and generating feedback.

[2156] "Feedback" refers to visual and audible advice provided to the user based on the evaluation results generated by the server.

[2157] "Visual feedback" refers to visual information displayed on the smart glasses screen that shows corrections to your golf swing and the ideal swing form.

[2158] "Auditory feedback" refers to advice in the form of audio messages played through the audio output device of the smart glasses.

[2159] This invention uses smart glasses, a wearable device worn by the user, to collect video data of the golf swing, analyze it with an analysis system, and provide visual and auditory feedback.Furthermore, by combining it with an emotion engine, it is possible to provide more personalized feedback based on the user's emotional state.

[2160] Overall system configuration

[2161] 1. Smart Glasses:

[2162] Equipped with a camera: Records the user's swing in real time.

[2163] Facial expression recognition camera: Captures the user's facial expression data required for the emotion engine.

[2164] Microphone: Collects audio data.

[2165] Biometric sensors: measure biometric data such as heart rate and galvanic skin response.

[2166] Display: Provides visual feedback.

[2167] Audio output device: Provides auditory feedback.

[2168] 2. Terminal device (e.g., smartphone):

[2169] Receiving video data: Receive video data from the smart glasses.

[2170] Receiving emotional data: Receiving facial expressions, voice, and biometric data.

[2171] Data preprocessing: Trimming and format conversion of video data and emotion data.

[2172] Communication with the server: Send the preprocessed data to the server.

[2173] 3. Server Device:

[2174] Swing Data Analysis: Analyze grip, ball placement, swing speed, ball trajectory, and body movement.

[2175] Emotional data analysis: Recognize the user's emotional state and reflect it in the feedback.

[2176] Feedback generation: Generates user-specific visual and auditory feedback.

[2177] Sending feedback: Sending the generated feedback to the end device.

[2178] Program processing

[2179] Smart Glasses Processing

[2180] The system uses a camera to collect video data of the user's golf swing. At the same time, a facial recognition camera records the user's facial expressions, a microphone collects audio data, and biometric sensors capture biometric data such as heart rate and skin galvanic response.

[2181] The collected data is transferred to a terminal device.

[2182] Terminal processing

[2183] The video data and emotion data received from the smart glasses are saved and unnecessary parts are trimmed.

[2184] The video data is converted into an appropriate format, and metadata (time information, environmental conditions, etc.) is added to the emotion data.

[2185] The preprocessed data is uploaded to a server device.

[2186] Server device processing

[2187] The system analyzes pre-processed video data sent from the terminal device to identify and evaluate the swing grip, ball placement, swing speed, ball trajectory, and body movement.

[2188] Emotional data is analyzed to recognize the user's emotional state, for example, to determine whether the user is stressed or focused.

[2189] Generate feedback based on the analysis results. Dynamically adjust the content and presentation of the feedback depending on the user's emotional state. For example, if the user is feeling stressed, provide advice on how to relax.

[2190] The feedback content is transmitted to the terminal device.

[2191] Terminal device feedback processing

[2192] The feedback received from the server is prepared for transmission to the smart glasses and transferred to the smart glasses.

[2193] Feedback processing for smart glasses

[2194] Visual feedback (e.g., correct grip position and ball placement) is provided on the display.

[2195] Auditory feedback (voice messages) is played on the audio output device, instructing the user to, for example, "Hold a tighter grip and correct the ball placement to the right."

[2196] Specific examples

[2197] Beginner user use cases

[2198] 1. A user wears smart glasses and performs a golf swing. The smart glasses' camera records the swing and facial expressions, the microphone collects audio data, and the biometric sensor records the heart rate.

[2199] 2. The terminal device receives the recorded data and emotion data, preprocesses the data (trimming unnecessary parts, format conversion, adding metadata), and sends it to the server.

[2200] 3. The server analyzes the video data and generates feedback that reflects the user's emotional state, such as "Your grip is weak" or "The ball is placed too left." For example, if the user is feeling stressed, the server provides additional feedback such as "Relax and concentrate on your swing."

[2201] 4. The terminal device receives the feedback and sends it to the smart glasses.

[2202] 5. The smart glasses visually display the correct grip position and provide audio instructions such as "Hold a firmer grip and correct the ball placement to the right." If the user feels stressed, the glasses also provide additional advice such as "Relax and focus on your swing."

[2203] Intermediate User Use Cases

[2204] 1. The user wears the smart glasses while swinging. The camera records the swing, captures facial expression data, collects audio with a microphone, and records heart rate with a biometric sensor.

[2205] 2. The terminal device preprocesses the recorded data and emotion data and sends them to the server.

[2206] 3. The server analyzes the data and obtains the evaluation result that "the timing of raising the head is too early." In addition to feedback based on the evaluation result, feedback reflecting the user's emotional state is generated.

[2207] 4. The terminal device receives the feedback and sends it to the smart glasses.

[2208] 5. The smart glasses visually indicate the appropriate time to raise your head and provide audio instructions such as, "Be aware of your rhythm to stabilize your swing speed." If the user is concentrating too much, the smart glasses provide additional advice such as, "Keep your concentration and keep swinging."

[2209] Prompt Sentence Examples

[2210] Example prompts to be input to the generative AI model:

[2211] Your task is to write a program to operate smart glasses for a golf swing analysis system. The system will collect data by combining cameras, facial recognition, voice recognition, and biometric sensors to analyze the user's swing and emotional state. It will then provide appropriate feedback in real time. Through the process from data collection to feedback provision, we aim to improve the user's swing technique and care for their emotions.

[2212] This prompt statement enables the generative AI model to understand the overall flow of the system and generate the necessary program code.

[2213] The flow of the identification process in the second embodiment will be described with reference to FIG.

[2214] Step 1:

[2215] The user wears smart glasses and performs a golf swing. The smart glasses collect video data of the user's golf swing using a built-in camera. At the same time, the facial recognition camera captures the user's facial expression data, and the microphone collects audio data. The smart glasses also use biometric sensors to capture biometric data such as heart rate and galvanic skin response. This data is stored in the smart glasses' internal memory.

[2216] input:

[2217] User's golf swing

[2218] User's facial expression

[2219] User Voice

[2220] User's biometric information

[2221] output:

[2222] Golf swing video data

[2223] User facial expression data

[2224] User voice data

[2225] User biometric data

[2226] Step 2:

[2227] All data collected by the smart glasses is transferred to the terminal device in real time using wireless communication such as Bluetooth or Wi-Fi.

[2228] input:

[2229] Golf swing video data

[2230] User facial expression data

[2231] User voice data

[2232] User biometric data

[2233] output:

[2234] Transferred video data

[2235] Transferred facial expression data

[2236] Transferred audio data

[2237] Transmitted biometric data

[2238] Step 3:

[2239] The terminal device stores the data received from the smart glasses and trims any unnecessary parts. For example, unnecessary parts before and after a swing are cut out. The video data is also converted into an appropriate format for analysis. Metadata such as time information and environmental conditions are added to the emotion data. After this preprocessing, the data is stored in the terminal device's internal memory.

[2240] input:

[2241] Transferred video data

[2242] Transferred facial expression data

[2243] Transferred audio data

[2244] Transmitted biometric data

[2245] output:

[2246] Video data after trimming

[2247] Format converted video data

[2248] Emotion data with added metadata

[2249] Step 4:

[2250] The terminal device uploads the pre-processed data to the server, which transfers the data via the Internet. Once uploaded, the data is stored on the server for analysis.

[2251] input:

[2252] Video data after trimming

[2253] Format converted video data

[2254] Emotion data with added metadata

[2255] output:

[2256] Preprocessed data stored on the server

[2257] Step 5:

[2258] The server analyzes the pre-processed video data and emotional data. It uses AI algorithms to identify and evaluate swing grip, ball placement, swing speed, ball trajectory, and body movement. It also analyzes the emotional data to recognize the user's emotional state (e.g., stress, concentration).

[2259] input:

[2260] Preprocessed video data

[2261] Emotion data with added metadata

[2262] output:

[2263] Swing data analysis results

[2264] Emotion data analysis results

[2265] Step 6:

[2266] The server generates personalized visual and auditory feedback for the user based on the analysis results. The content and presentation of the feedback are dynamically adjusted according to the user's emotional state. For example, if the user is feeling stressed, advice to help them relax is generated. The generated feedback is stored on the server.

[2267] input:

[2268] Swing data analysis results

[2269] Emotion data analysis results

[2270] output:

[2271] Individual user feedback

[2272] Step 7:

[2273] The server transmits the generated feedback to the terminal device, with data transfer occurring over the Internet.

[2274] input:

[2275] Individual user feedback

[2276] output:

[2277] Feedback sent to the device

[2278] Step 8:

[2279] The terminal device receives the feedback and prepares it for transmission to the smart glasses. The feedback is transferred to the smart glasses via Bluetooth or Wi-Fi.

[2280] input:

[2281] Feedback sent to the device

[2282] output:

[2283] Feedback transmitted to smart glasses

[2284] Step 9:

[2285] The smart glasses display visual feedback (e.g., correct grip position and ball placement) on the display, and also play auditory feedback (e.g., "Hold your grip tighter and correct the ball placement to the right") through an audio output device, allowing the user to receive feedback in real time.

[2286] input:

[2287] Feedback transmitted to smart glasses

[2288] output:

[2289] Visual and auditory feedback provided to the user

[2290] (Application example 2)

[2291] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[2292] Currently, many factory robot and worker motion analysis systems have difficulty detecting motion errors and efficiency declines in real time and providing appropriate feedback. Furthermore, few systems consider the psychological state and stress level of workers to improve their motion. Therefore, there is a need for more effective and adaptive feedback systems to improve production efficiency and reduce errors.

[2293] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[2294] In this invention, the server includes a means for evaluating and generating feedback on operations, placement, and efficiency, a means for analyzing emotional data and recognizing the stress and concentration of workers, and a means for providing the generated feedback visually and audibly. This makes it possible to analyze the operations of factory robots and workers in real time and provide appropriate feedback. It is also possible to provide more personalized feedback by taking into account the psychological state of the workers.

[2295] A "user" is a person or operator who uses a motion analysis system for golf swings or robotic work.

[2296] A "wearable device" is a device that can be worn by a user and is capable of collecting and displaying data.

[2297] "Smart glasses" are wearable devices equipped with cameras, sensors, displays, etc., that function as an interface for visual and auditory data.

[2298] A "camera" is a device that takes and records images and videos.

[2299] A "golf swing" refers to a golf hitting motion, and is an act for collecting data on a user's swing motion.

[2300] "Real time" refers to processing events as they occur, without any time lag.

[2301] "Recording means" refers to a method or device for recording video data using a camera mounted on smart glasses.

[2302] "Video data" refers to data that displays a series of images recorded by a camera in a time-series sequence.

[2303] "Emotion data" is data that indicates the emotional state of the user based on biological reactions such as the user's heart rate and facial expressions.

[2304] "Preprocessing" refers to the process of processing or converting collected data to make it easier to analyze.

[2305] A "terminal device" is an electronic device capable of data communication and processing, including smartphones and tablets.

[2306] A "server device" is a computer system for storing, managing, and analyzing data.

[2307] "Evaluation" is the act of judging the quality of the results of data analysis based on specific criteria.

[2308] "Feedback" refers to instructions or advice provided to the user based on the evaluation results.

[2309] "Visual feedback" is information provided to a user using a visual medium such as a display.

[2310] "Auditory feedback" refers to spoken messages or instructions provided to a user through an audio output device.

[2311] "Efficiency of action" is a measure of how effectively an action is performed to achieve a goal.

[2312] "Layout" refers to the state of how things or people are arranged.

[2313] An "error rate" is the rate at which errors occur during a particular task or activity.

[2314] "Physical responses" are biometric data that indicate the user's physical changes and condition.

[2315] System Configuration

[2316] Hardware

[2317] 1. Smart Glasses

[2318] camera

[2319] Facial Recognition Camera

[2320] microphone

[2321] Biometric Sensor

[2322] display

[2323] Audio output device

[2324] 2. Terminal Device

[2325] Smartphone or tablet

[2326] 3. Server Device

[2327] Computer system for data analysis

[2328] software

[2329] 1. Video analysis software

[2330] (e.g. OpenCV)

[2331] 2. Facial Recognition Software

[2332] 3. Audio Analysis Software

[2333] 4. Biological Data Analysis Software

[2334] 5. Emotion Engine

[2335] 6. Feedback Generator

[2336] Overall system processing

[2337] 1. Data Collection and Preprocessing

[2338] Smart Glasses:

[2339] The camera installed in the smart glasses captures the user's work and movements in real time.

[2340] A facial expression recognition camera captures the user's facial expression data, and a microphone collects voice data.

[2341] Biosensors are used to measure biometric data such as heart rate and skin galvanic response.

[2342] Terminal device:

[2343] Receive video data and emotion data transmitted from the smart glasses.

[2344] Trim out any unwanted parts and convert to the appropriate format.

[2345] The preprocessed data is transferred to a server device.

[2346] 2. Data analysis and feedback generation

[2347] Server device:

[2348] Using video analysis software, we analyze the efficiency, placement, and error rate of movements.

[2349] Using facial expression recognition software and biometric data analysis software, the system analyzes the user's emotional state, thereby identifying the user's stress and concentration level.

[2350] The emotional engine generates feedback based on the results of motion analysis, dynamically adjusting the visual and auditory feedback according to the user's emotional state.

[2351] Generate feedback:

[2352] Based on the analysis results, feedback is generated that provides visual and auditory indications of work improvements and points to note.

[2353] The server transmits the feedback content to the terminal device.

[2354] 3. Providing Feedback

[2355] Terminal device:

[2356] The received feedback is forwarded to the smart glasses.

[2357] Smart Glasses:

[2358] Visual feedback is provided on the display, such as ideal operating positions and areas requiring attention.

[2359] The auditory feedback is played on an audio output device to instruct the user as a voice message.

[2360] Specific examples

[2361] Use cases for beginner users:

[2362] A user wears smart glasses and works on a factory production line. The smart glasses' camera captures their movements and facial expressions, the microphone collects audio data, and the biometric sensor records their heart rate. The terminal device preprocesses this data and sends it to a server. Based on the analysis results, the server provides visual and auditory feedback on areas for improvement and important points to note.

[2363] Example prompt sentence:

[2364] "Analyze the robot's movements during work and generate feedback to maximize efficiency. Also, recognize the worker's stress and concentration levels and provide advice based on that."

[2365] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[2366] Step 1:

[2367] Data collection using smart glasses

[2368] Input: User movements, facial expressions, voice data, biometric data (heart rate, skin galvanic response)

[2369] How it works: A camera on the smart glasses captures the user's movements in real time, a facial recognition camera records the user's facial expressions, a microphone collects audio data, and biometric sensors measure heart rate and galvanic skin response.

[2370] Output: Acquisition of video data, facial expression data, audio data, and biometric data

[2371] Step 2:

[2372] Data reception and preprocessing by the terminal device

[2373] Input: Video data, facial expression data, audio data, and biometric data sent from smart glasses

[2374] Specific operation: The terminal device (smartphone or tablet) receives the data sent from the smart glasses, trims unnecessary parts, converts the video data into an appropriate format, and adds metadata (time information, environmental conditions, etc.) to the emotion data.

[2375] Output: Preprocessed video data and emotion data

[2376] Step 3:

[2377] Data transmission from terminal device to server

[2378] Input: Preprocessed video data and emotion data

[2379] Specific operations: The terminal device uploads the preprocessed data to the server device.

[2380] Output: Preprocessed data sent to the server

[2381] Step 4:

[2382] Data analysis by server

[2383] Input: Preprocessed data sent from the terminal

[2384] Specific operations: The server analyzes the video data and evaluates the efficiency, placement, and error rate of the movements. It also uses facial expression recognition software and biometric data analysis software to analyze the user's emotional data and recognize their stress and concentration levels.

[2385] Output: Movement evaluation results and emotional state analysis results

[2386] Step 5:

[2387] Feedback Generation

[2388] Input: Movement evaluation results and emotional state analysis results

[2389] Specific behavior: The server utilizes an emotion engine to generate feedback based on the behavior analysis results and the user's emotional state. The visual and auditory feedback content is dynamically adjusted.

[2390] Output: Visual and auditory feedback

[2391] Step 6:

[2392] Sending feedback to terminal devices

[2393] Input: Server-generated feedback

[2394] Specific operations: The server device transmits the generated feedback to the terminal device.

[2395] Output: Feedback sent to the terminal

[2396] Step 7:

[2397] Sending feedback from the device to the smart glasses

[2398] Input: Feedback sent to the terminal

[2399] Specific operation: The terminal device transmits the feedback content to the smart glasses.

[2400] Output: Feedback sent to the smart glasses

[2401] Step 8:

[2402] Feedback provided by smart glasses

[2403] Input: Feedback sent to the smart glasses

[2404] Specific operation: Smart glasses display visual feedback on the display and play auditory feedback through the audio output device. For example, instructions for correcting movement placement or advice on preventing mistakes are provided via both visual and audio feedback.

[2405] Output: The visual and auditory feedback the user receives.

[2406] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[2407] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[2408] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[2409] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[2410] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[2411] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[2412] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[2413] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[2414] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[2415] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[2416] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[2417] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[2418] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[2419] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[2420] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[2421] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[2422] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[2423] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[2424] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[2425] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[2426] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[2427] The following is further disclosed regarding the above embodiment.

[2428] (Claim 1)

[2429] smart glasses, which are wearable devices worn by users;

[2430] A means for recording a golf swing in real time using a camera mounted on the smart glasses;

[2431] a terminal device that receives the recorded video data and performs preprocessing;

[2432] a server device that analyzes the pre-processed video data and generates swing evaluation and feedback;

[2433] means for providing the generated feedback visually and audibly to the smart glasses;

[2434] A system including:

[2435] (Claim 2)

[2436] 10. The system of claim 1, wherein the evaluation and feedback analyzes swing grip, ball placement, ball trajectory, head lift timing, and body movement.

[2437] (Claim 3)

[2438] 10. The system of claim 1, wherein the visual feedback presented to the smart glasses indicates an ideal grip position, ball placement, etc., and the auditory feedback is provided as a voice message.

[2439] "Example 1"

[2440] (Claim 1)

[2441] smart glasses, which are wearable devices worn by users;

[2442] A means for recording a golf swing in real time using a camera mounted on the smart glasses;

[2443] a terminal device that receives the recorded video data, trims unnecessary portions, and converts the data format;

[2444] a server device that includes a machine learning algorithm for analyzing the pre-processed video data, and that analyzes each element of the swing and generates an evaluation and feedback;

[2445] means for providing the generated feedback visually and audibly to the smart glasses;

[2446] A system including:

[2447] (Claim 2)

[2448] 10. The system of claim 1, wherein the evaluation and feedback analyzes swing grip, ball placement, ball trajectory, head lift timing, and body movement.

[2449] (Claim 3)

[2450] 10. The system of claim 1, wherein the visual feedback presented to the smart glasses indicates an ideal grip position, ball placement, etc., and the auditory feedback is provided as a voice message.

[2451] "Application Example 1"

[2452] (Claim 1)

[2453] smart glasses, which are wearable devices worn by users;

[2454] a means for recording the movement in real time by a camera mounted on the smart glasses;

[2455] a terminal device that receives the recorded video data and performs preprocessing;

[2456] a server device that analyzes the pre-processed video data and generates evaluations and feedback for the actions;

[2457] means for providing the generated feedback visually and audibly to the smart glasses;

[2458] A means for improving factory robotic operation;

[2459] A system including:

[2460] (Claim 2)

[2461] 2. The system of claim 1, wherein the evaluation and feedback analyzes the accuracy, efficiency, pickup position, and smoothness of the robot's movements.

[2462] (Claim 3)

[2463] The system of claim 1, wherein the visual feedback presented to the smart glasses indicates the correct pickup position, corrections to the operation, etc., and the auditory feedback is provided as a voice message.

[2464] "Example 2: Combining Emotion Engines"

[2465] (Claim 1)

[2466] smart glasses, which are wearable devices worn by users;

[2467] A means for recording and measuring the golf swing and the user's emotional state in real time using a camera and a sensor mounted on the smart glasses;

[2468] a terminal device that receives the recorded video data and emotion data and trims and converts the format of the data;

[2469] a server device that analyzes the pre-processed video data and emotion data and generates evaluations and feedback based on swings and emotional states;

[2470] means for providing the generated feedback visually and audibly to the smart glasses;

[2471] A system including:

[2472] (Claim 2)

[2473] 10. The system of claim 1, wherein the evaluation and feedback analyzes swing grip, ball placement, ball trajectory, head lift timing, and body movement, and further reflects the user's emotional state.

[2474] (Claim 3)

[2475] The system of claim 1, wherein the visual feedback presented to the smart glasses displays the ideal grip position, ball placement, etc., and the auditory feedback is provided as a voice message, and the content is dynamically adjusted depending on the user's emotional state.

[2476] "Application e...

Claims

1. smart glasses, which are wearable devices worn by users; A means for recording a golf swing in real time using a camera mounted on the smart glasses; a terminal device that receives the recorded video data and performs preprocessing; a server device that analyzes the pre-processed video data and generates swing evaluation and feedback; means for providing the generated feedback visually and audibly to the smart glasses; A system including:

2. 10. The system of claim 1, wherein the evaluation and feedback analyzes swing grip, ball placement, ball trajectory, head lift timing, and body movement.

3. 10. The system of claim 1, wherein the visual feedback presented to the smart glasses indicates an ideal grip position, ball placement, etc., and the auditory feedback is provided as a voice message.

Images