Image processing system including electronic device

Abstract

The present invention can provide an image processing system that can enhance immersion in a VR environment by more accurately reflecting the user's movements captured by an image sensor and naturally implementing movements in an environment or virtual space where augmented reality can be displayed, thereby enabling realistic interaction.

Description

Technology Field

[0001] The present invention relates to a system that analyzes data received from sensors including image sensors and provides customized support for the performance of ballet movements according to the analyzed user's movements. In particular, the invention relates to a system capable of designing suitable ballet movements by considering the user's physical condition and characteristics, and performing such movements. Background Technology

[0002] Programs that combine rehabilitation therapy with artistic exercise are not only effective for improving physical and mental health but also possess diverse potential applications. In particular, the artistic and expressive characteristics of ballet can be utilized in various fields, including not only rehabilitation but also learning motivation, motor skill enhancement, and the design of customized programs for people with disabilities. Ballet is beneficial for improving balance, flexibility, and muscle strength; by combining this with image processing technology, the scope of application can be expanded beyond rehabilitation therapy to include ballet instruction, exercise equipment recommendations, and the design of movements tailored for the disabled.

[0003] However, existing ballet programs often fail to adequately reflect the individual physical conditions of people with disabilities or beginner learners. For instance, despite the fact that the difficulty and range of movements required vary by type of disability, standardized programs are provided, leading to problems such as an increased risk of injury or reduced therapeutic effectiveness during the performance of movements. Furthermore, general learning programs often lack movement design that reflects the user's current state and goals, which can limit learning effectiveness or cause people with disabilities to feel alienated due to their inability to properly execute movements.

[0004] Existing rehabilitation and learning programs tend to focus solely on physical exercise, which limits their ability to stimulate users' mental motivation or provide a sense of artistic accomplishment. Consequently, users often fail to consistently participate in the program, or the overall effectiveness of the treatment and learning diminishes. In particular, if movement design fails to reflect user characteristics, it can lead to situations where users find it difficult to perform movements or learn independently without continuous intervention from a therapist or coach.

[0005] Therefore, a system is required that can design customized movements for rehabilitation, learning, and motor skill improvement by reflecting the user's physical characteristics, optimize the user experience through real-time feedback, and provide continuously updated personalized plans. Prior art literature

[0007] Published Patent Application No. 2018-0068609, Japanese Published Patent Application No. 2014-69030 The problem to be solved

[0008] The objective of the present invention is to provide an image processing system that supports rehabilitation treatment by designing ballet movements tailored to the user's physical condition and characteristics, thereby solving the aforementioned problems.

[0009] First, the primary objective is to provide a video processing system that enables safe and effective ballet performance by automatically adjusting exercise intensity and range suitable for the user.

[0010] In addition, the purpose is to provide an image processing system capable of improving the user's ballet movements and tracking the progress of rehabilitation treatment through real-time feedback and monitoring. means of solving the problem

[0011] To achieve the above objective, the present invention may implement an image processing system including an electronic device, comprising: an image sensor configured to generate image data by converting incident light through a lens into an electrical signal; an image processing module comprising a binning processing unit that performs binning by grouping adjacent pixels within the image data received from the image sensor, and processes and analyzes the image data by detecting and tracking the features of an object included in the binning processed image; and a display unit that displays the content processed and analyzed by the image processing module as virtual reality.

[0012] In addition, the binning processing module may also convert the resolution of the received image data or the signal strength of each pixel.

[0013] In addition, an image processing system can be implemented by being characterized in that the objects detected and tracked by the object detection and tracking module are the head, upper body, and legs. Effects of the invention

[0014] According to the present invention, the following effects can be achieved.

[0015] First, the present invention can maximize safety and effectiveness in the learning and rehabilitation process by analyzing the user's physical condition and characteristics and designing ballet movements suitable for each individual.

[0016] In addition, the present invention can precisely track the position of the user's major joints through deep learning-based joint extraction and correction technology (OpenPose, HRNet, etc.), and thereby accurately analyze the user's motion data and improve the accuracy and consistency of the posture.

[0017] In addition, the present invention can fuse sensor data to supplement inaccurate or missing information and precisely evaluate the user's condition.

[0018] In addition, the present invention can detect excessive muscle use or balance abnormalities during the motion analysis process and warn of the risk of injury in advance.

[0019] In addition, the present invention can enhance safety by continuously monitoring the user's condition during rehabilitation treatment and recommending an appropriate exercise intensity.

[0020] In addition, the present invention can improve learning effects by providing real-time feedback based on user motion data, and can effectively support the learning and rehabilitation process by tracking the user's improvement process and iteratively optimizing motion design according to the results.

[0021] Furthermore, the present invention can be applied to various fields beyond ballet movement analysis and rehabilitation therapy, such as fitness, sports training, and flexibility and balance training, and can provide flexible customized solutions according to the different physical characteristics and learning goals of each user.

[0022] Furthermore, the present invention stimulates learning motivation and enhances a sense of accomplishment by visually providing the user's improvement status and learning goals. In particular, it can improve learning sustainability by enabling the systematic management of the learning process based on data.

[0023] Furthermore, according to the present invention, by implementing a 360-degree stereoscopic display, the user can observe their ballet movements in three dimensions and immediately correct incorrect postures. In particular, since the user can check their movements from various angles, it is effective in enhancing the accuracy and artistry of the movements.

[0024] In addition, the present invention provides real-time feedback to the display, allowing the user to experience a high level of immersion during the learning and rehabilitation process. Brief explanation of the drawing

[0025] FIG. 1 is a block diagram showing an image processing system that supports rehabilitation treatment through ballet according to an embodiment of the present invention. FIG. 2 is a schematic diagram illustrating the process of binning an input image in a binning processing unit according to an embodiment of the present invention. FIG. 3 is a conceptual diagram of precisely tracking major joints of the body using a modified OpenPose according to an embodiment of the present invention. Figure 4 is a drawing illustrating a user performing ballet movements with a marker attached to extract joint positions. FIG. 5 is a flowchart illustrating the process of precisely correcting the joint positions extracted from a deep learning analysis model by attaching the markers shown in FIG. 4 and capturing the joint positions of a user performing ballet movements with a camera, analyzing the relative positional relationships and patterns between the markers. FIG. 6 is a drawing showing that, according to an embodiment of the present invention, a user's ballet movements are displayed on a display as a 360-degree stereoscopic display. FIG. 7 is a flowchart illustrating an image processing process that supports rehabilitation treatment based on ballet movements according to an embodiment of the present invention. Specific details for implementing the invention

[0026] In order to implement an image processing system that utilizes a multi-camera system and wearable sensors to analyze a user's ballet movements in 360 degrees and supports effective rehabilitation treatment and ballet learning through customized movement design, real-time feedback, and multi-angle visualization, preferred embodiments of the present invention will be described in detail below with reference to the attached drawings. However, embodiments of the present invention may be modified in various different forms, and the scope of the present invention should not be interpreted as being limited by the embodiments described below.

[0027] FIG. 1 is a block diagram illustrating an image processing system that supports rehabilitation treatment through ballet according to an embodiment of the present invention.

[0028] Referring to FIG. 1, the system of the present invention includes a body condition analysis module, an image processing module, a sensor unit, a customized ballet movement design module, a storage unit, a learning module, a display, a communication module, and a management server.

[0029] First, the present invention must accurately determine the user's physical condition in order to evaluate the user's current condition and calculate the range of motion possible. To this end, the physical condition analysis module illustrated in FIG. 1 evaluates the user's current condition and calculates the range of motion possible by analyzing data sensed by a sensor unit capable of measuring the physical condition (e.g., range of motion, muscle strength, balance ability) while the user is wearing it, or by analyzing images captured by a camera.

[0030] A method for evaluating a user's current condition using only wearable sensor data involves measuring the user's body movements and condition through various wearable sensors (41), such as an IMU, pressure sensor, and electromyography (EMG) sensor, and analyzing the user's joint angles, muscle activation levels, and movement patterns through this data to evaluate the user's current condition.

[0031] There is an analysis method that combines camera and sensor data, which enables more precise analysis of the body condition than analysis methods using only wearable sensor data, by presenting ballet movements on a display and using a camera to capture the movements while the user follows them. Data captured by the camera analyzes the positions of major joints through deep learning-based keypoint extraction models (OpenPose, HRNet, etc.), and by fusing this with data collected from wearable sensors, it is possible to supplement missing details or correct errors in the sensing data by comparing it with the sensor data.

[0032] In order to evaluate the user's current state using only wearable sensor data, it is necessary to include a sensor unit for analyzing the physical state to measure the user's joint range of motion, muscle strength, balance ability, etc. Since the sensor unit is intended to determine the user's initial physical state, it does not necessarily have to be a wearable sensor. However, if it is a wearable sensor, it is easy to wear and there are fewer restrictions on movement, so it is preferable for the sensor unit of the present invention to be composed of a wearable sensor.

[0033] The sensor unit utilizes an Inertial Measurement Unit (IMU) to track joint movements, an Electromyography (EMG) sensor to collect electrical muscle signals, and a pressure sensor to evaluate balance. These sensors provide foundational data for precisely understanding the user's physical condition.

[0034] Data sensed by the sensor unit is transmitted to the storage unit and the body condition analysis module, where it is integratedly analyzed to evaluate the user's physical characteristics. The data collected by each sensor is compared and combined to calculate an accurate physical condition, and noise removal and data correction are performed during this process. Through this, the range and limits of ballet movements that the user can perform are established, and potential risks that may occur during the performance of specific movements are predicted.

[0035] The sensor unit may be configured to include an electromyography (EMG) sensor, a pressure sensor, or an inertial measurement unit (IMU) to measure body conditions (e.g., joint range of motion, muscle strength, balance ability).

[0036] Electromyography (EMG) sensors measure the electrical activity of muscles to analyze the interaction between nerves and muscles. When a user performs specific ballet movements, muscle activation levels and fatigue can be assessed by monitoring the patterns of electrical signals generated by the muscles. For example, if insufficient activation of lower body muscles is detected, the system adjusts the user's training intensity based on this finding or suggests exercises to strengthen specific muscle groups.

[0037] Furthermore, EMG sensors can detect abnormal muscle signals to provide advance warnings of injury risks. If muscle fatigue exceeds a specific threshold, it notifies the user that they are under excessive load and may recommend measures to reduce it. This allows users to safely continue rehabilitation treatment, while enabling therapists to gain a more detailed understanding of the user's condition based on muscle change data.

[0038] The pressure sensor measures the pressure distribution between the user's soles and the ground to analyze their ability to maintain balance. This sensor monitors in real-time how the user distributes their body weight during movement and provides immediate feedback if balance is lost. Since maintaining balance is crucial during ballet movements, the pressure sensor analyzes the user's center of gravity shift patterns to establish an ideal weight distribution and corrects movements based on this.

[0039] Furthermore, pressure sensors are particularly useful for users at risk of falls and are suitable for rehabilitation training aimed at improving balance. For example, if a user's weight is shifted to one side during a movement requiring balance on one leg, the system can assist in correcting posture through visual warnings and clearly provide therapists and the user with the progress of their balance improvement through long-term data recording.

[0040] An IMU (Inertial Measurement Unit) sensor is a device used to measure user movement data and contains an accelerometer and a gyroscope. The accelerometer measures the user's linear acceleration to detect changes in speed and direction of movement, while the gyroscope detects rotational angular velocity to measure changes in angle when the user rotates or tilts.

[0041] The IMU sensor calculates the user's movement patterns by comprehensively analyzing data collected from the accelerometer and gyroscope. This data supports real-time tracking of changes in speed, direction, and angle of movement in three axes (horizontal, vertical, and rotational), allowing for accurate identification of the user's posture or movement path.

[0042] IIMU sensors, fabricated as a single chip using MEMS technology, are lightweight and compact, making them easy to integrate with wearable devices and utilized in various fields including rehabilitation therapy and motion analysis. In particular, IMUs are effective for precisely monitoring and analyzing a user's body movements to evaluate the consistency of motion or compare it with target movements.

[0043] However, while IMU sensors utilize accelerometers and gyroscopes to measure user movement data, data accumulation errors (drift) may occur during prolonged use. Drift refers to the phenomenon where position or angle data measured by the IMU sensor gradually deviates from the actual values. This issue can degrade accuracy in the analysis of precise movements, such as ballet steps. Therefore, to minimize drift, it is advisable to combine IMU sensors with cameras or other sensors to apply correction algorithms, although this is not the only approach.

[0044] The drift correction algorithm operates by periodically resetting IMU data or integrating external reference data to reduce drift, and in the present invention, drift can be minimized by correcting in combination with other sensors, correcting using a filtering algorithm, or periodically resetting the IMU.

[0045] 1) Fusion of IMU and sensor data

[0046] To minimize drift when combined with other sensors, IMU data is combined with external sensors (e.g., cameras, electromyography (EMG) sensors, GPS, or pressure sensors, etc.) to correct for drift.

[0047] First, drift can be corrected by fusing IMU and camera data.

[0048] While IMU sensors track user movement data, errors can accumulate due to drift during prolonged use. To compensate for this, the IMU data is corrected by fusing absolute joint position data collected from the camera. For example, when a user raises their arm during a ballet movement, the camera accurately measures the positions of the elbow and shoulder joints and complements the movement based on the relative position changes in the IMU data.

[0049] In this process, the camera provides an absolute reference point, and the IMU tracks minute changes in movement, mutually complementing each other's limitations. Through this, users can receive more accurate motion analysis and correction in real time, and minimize accumulated errors caused by drift.

[0050] Next, drift can be corrected by fusing IMU and electromyography (EMG) sensor data.

[0051] While IMUs track joint movement, they cannot verify whether the appropriate muscles are being activated during motion. To address this, EMG sensors are fused to monitor the state of muscle activation. For example, when a user lifts their leg, the IMU measures the angle and speed, while the EMG evaluates the activation level of the thigh muscles.

[0052] By combining IMU and EMG sensor data, it is possible to simultaneously verify whether the user is performing movements correctly and whether muscles are being properly activated. If muscle activation is insufficient or excessive load is applied, the system provides appropriate feedback to reduce the risk of injury and improve movement accuracy.

[0053] Furthermore, drift can be corrected by fusing IMU, camera, and pressure sensor data.

[0054] While IMU sensors measure the angle and acceleration of joint movements, they may not accurately reflect the balance of the foot during movement. To compensate for this, fusing pressure sensor data allows for real-time monitoring of the user's balance and weight distribution. For example, pressure sensors mounted on the soles of the feet measure weight shift and changes in balance, while motion data captured by a camera is analyzed to evaluate how well the user matches the designed movements.

[0055] Pressure sensors provide data when weight is shifted to one foot or balance is off, while cameras visually complement joint movements. By combining the two data, it is possible to detect subtle balance changes or joint angle anomalies that IMU sensors might miss, thereby providing more precise motion correction and feedback.

[0056] 2) Correction using filtering algorithms

[0057] The present invention removes noise and drift using a Kalman filter or an Extended Kalman Filter (EKF). This algorithm compares current sensor data with the previous state, calculates the most reliable value, and performs continuous correction.

[0058] The Kalman Filter and the Extended Kalman Filter (EKF) are filtering algorithms used to precisely correct sensor data. These algorithms improve data reliability by eliminating noise and drift generated by IMU sensors. In particular, they are useful for providing stable and accurate data in precision tasks such as motion analysis.

[0059] The filtering algorithm predicts the most reliable value by comparing current sensor data with previous state data. It calculates the optimal result by combining data measured by the sensor with data predicted by the system model, continuously correcting for noise and errors in the process. For example, angular velocity data measured by an IMU sensor undergoes a filtering process to reduce the influence of external vibrations and more accurately reflect actual movement.

[0060] This algorithm is essential, particularly in motion analysis systems such as rehabilitation therapy. The Kalman filter is suitable for linear systems, while the extended Kalman filter can process non-linear data, making it suitable for analyzing complex movements. This allows users to receive accurate motion data in real time and enhances the reliability and accuracy of the system by minimizing accumulated errors caused by drift.

[0061] 3) Periodic IMU reset

[0062] The IMU sensor can reduce drift error by detecting a stationary state or periodically resetting it using reference point data. In a stationary state, the sensor initializes the measurement value by establishing a new, motionless reference point, thereby minimizing the error.

[0063] Reference point data is used to reset IMU data by utilizing external sensors (e.g., cameras, GPS, etc.) or predefined stop positions. For example, when a user stops at a specific position after performing a ballet movement, the system resets the initial state of the IMU based on that position. This process helps maintain sensor data stably and enables accurate motion analysis and feedback over a long period of time.

[0064] The correction algorithm described earlier plays a crucial role in maintaining the accuracy and consistency of movements in ballet motion analysis. For example, when a user performs a leg-lifting and twisting motion, angles and movements are measured based on IMU sensor data; however, angle data can become distorted if drift occurs. The correction algorithm adjusts for this distortion in real time, enabling accurate motion analysis. Furthermore, by enhancing data reliability through sensor fusion and filtering techniques, it allows for the precise evaluation of the difference between the movements performed by the user and the designed goals. Additionally, such correction algorithms significantly improve system reliability and user experience, while providing stable data even during long-term rehabilitation training.

[0065] Through this data refinement process, the system of the present invention provides highly reliable results and can maximize the effectiveness of rehabilitation treatment for the user. In addition, the IMU is a physical sensor that is lightweight and portable, making it suitable for use in various rehabilitation environments.

[0067] Below, we will explain a method for analyzing a user's state by using a camera to capture the user's movements as they follow ballet movements displayed on a screen, and fusing this with sensing data.

[0068] The system according to the present invention further comprises an image processing module for processing and analyzing images of a user's movements captured by a camera to analyze the user's physical condition.

[0069] The image processing module analyzes the image of a user following ballet movements displayed on a screen to analyze the user's joint positions, movement speeds, and posture, and calculates joint angles, movement paths, and balance status to evaluate the accuracy of the posture, thereby supplementing the data collected by the sensor unit, and can also improve the precision of the user's physical condition measured by the sensor unit through visual evaluation of the user's movements.

[0070] In addition, beyond supplementing data from other sensors, the camera alone may be used to verify the user's state through image analysis of the user following the actions presented to the user in the present invention, but is not limited thereto.

[0071] The image processing module can be used not only for measuring the user's physical condition but also for image analysis necessary for the user's motion analysis described later; this will be explained in the section on user motion analysis.

[0072] The image processing module analyzes the user's condition using an artificial intelligence learning model based on images captured by the camera. The results of the physical condition analysis are provided as visual information via a display. This allows users to intuitively understand their physical limitations and potential for improvement, and the system can also provide immediate warnings if abnormal signs are detected during operation.

[0074] Images captured by a camera to analyze the user's state may contain unwanted signals (noise) due to lighting conditions, environmental factors, and sensor noise. Since this can degrade the accuracy of joint position extraction or motion analysis, it is necessary to enhance data reliability by removing unnecessary signals while preserving the details of user movements (joint clarity, motion contours, etc.).

[0075] To this end, the image processing module according to the present invention includes a preprocessing unit and improves the quality of the image by removing noise and adjusting contrast of the captured image through AI-based preprocessing.

[0076] Images captured by a camera may contain various types of noise, such as random noise, where random values ​​are added to image pixels due to lighting conditions or electronic interference; compression artifacts, which correspond to block-type distortion or loss of detail occurring during the image compression process; and environmental noise, where image quality is degraded due to external factors such as dust, shadows, and reflections.

[0077] Since noise distorts or causes the loss of detailed image information necessary for high-precision tasks such as joint extraction and motion analysis, the process of removing it is very important.

[0078] AI-based noise removal models learn how to remove noise by learning through comparison between clean images and images mixed with noise. In this case, DnCNN (Deep Convolutional Neural Network) and Noise2Noise can be used as deep learning-based noise removal models.

[0079] DnCNN is an analysis module that utilizes a CNN architecture specialized in identifying noise patterns and restoring original images, while Noise2Noise is a model capable of training solely on noisy data without clean images. It can remove noise by understanding its distribution.

[0080] For example, the method for generating a noise-removed image using DnCNN is as follows.

[0081] The preprocessing unit of the image processing module decomposes a noisy image into pixels, identifies noise patterns within the pixel-level image through a CNN layer (extracts features), and restores the original image while preserving its detailed information by removing noise according to the learned patterns.

[0082] At this time, the learned pattern is stored in the storage unit, and subsequent images or images taken in similar environments can be analyzed based on the previously learned pattern, and the time required for image analysis can be reduced by adjusting only minute differences.

[0083] When noise is removed from the image, the DnCNN module outputs a clean, noise-removed image while preserving details such as the original image's boundaries and textures.

[0084] In addition, the preprocessing unit performs image standardization and resizing in addition to the function of removing noise from images, enabling the analysis model through AI learning to better recognize joint features.

[0085] Image standardization is the process of normalizing images to enable a model to process input data consistently. In this process, the range of image pixel values ​​is standardized, or pixel values ​​are normalized based on the mean and standard deviation to make the data distribution uniform. This reduces the impact of factors such as lighting variations and color differences, helping the model learn joint features stably.

[0086] Resizing is the process of converting the resolution of an input image into a fixed size that a deep learning model can process. It reduces or enlarges the original image while maintaining the aspect ratio and minimizing distortion. This process improves the computational efficiency of the model and prevents performance degradation that may occur when processing images of different sizes.

[0087] The preprocessing unit according to the present invention may further include a binning processing unit. By providing the binning processing unit, the present invention can capture more light, thereby improving light sensitivity and being advantageous in low-light environments. It can also reduce image noise by combining multiple pixel values, and since the resolution is lowered, the frame rate of the camera can be increased, making it advantageous for high-speed imaging processing. Additionally, the signal is increased, thereby improving the signal-to-noise ratio (SNR).

[0088] The process of binning the input image in the binning processing unit is as follows.

[0089] First, the binning processing unit groups adjacent pixels as shown in FIG. 2. For example, it groups pixel blocks of a fixed size, such as 2x2, 3x3, or 4x4, into one group.

[0090] Figure 2a shows a simple form of binning for black and white image binning. It illustrates the process of 16 (4x4) pixels being grouped into a single pixel. In this process, the resolution of the original image is reduced to 1 / 16, but the signal strength of each pixel is increased.

[0091] In Fig. 2b, complex binning considering the Bayer pattern is performed for color image binning. Independent binning is performed for each color channel of red (R), green (G), and blue (B), and only pixels of the same color are combined to preserve color information. At this time, the reason there are more green pixels than other colors is that the human eye is more sensitive to green.

[0092] The process of grouping adjacent pixels as illustrated in FIGS. 2a and 2b may be performed in a physical image sensor or implemented in software during the digital processing stage.

[0093] Next, after grouping pixels, a new value is generated by combining the pixel values ​​within each group through summation or averaging, and the combination method is as follows:

[0094] Summing: Increases brightness by adding all pixel values ​​within a group; this method is advantageous in low-light environments and is effective in increasing light sensitivity.

[0095] Averaging: Generates a new value by calculating the average of pixel values ​​within a group; this method is effective in reducing noise and improving the signal-to-noise ratio (SNR).

[0096] The combined value is used as a representative value for the grouped pixels, which allows for the provision of a higher-quality signal at the expense of reduced resolution and enables optimal state analysis even in high-light and low-light environments.

[0097] In the present invention, an example was given of analyzing an image captured by a camera in a binning processing unit included in a preprocessing unit, but the binning processing unit may be included in a camera for capturing images or in an image processing module that analyzes captured images, and is not limited thereto.

[0098] Next, the joint position extraction unit extracts joint positions using a deep learning-based keypoint extraction model based on an image from which noise has been removed in the preprocessing unit or an image from which illumination and SNR have been improved by performing binning.

[0099] Deep learning-based keypoint extraction models are technologies that automatically recognize the locations of major body joints in images or videos and convert them into 2D or 3D coordinates. By analyzing the contours of the body through deep learning networks, they accurately detect the locations of major joints such as shoulders, elbows, and knees. Representative models include OpenPose, HRNet, DeepLabCut, and CNN, each of which enables real-time motion analysis and posture tracking.

[0100] In this invention, a modified OpenPose model is used to optimize real-time processing and improve accuracy by modifying the OpenPose model.

[0101] The modified OpenPose can precisely track major joints of the body and, as shown in Fig. 3, generate a skeleton structure to visualize the user's full-body movements, making it advantageous for multi-joint tracking. It can also extract joint data in real time, making it suitable for analyzing fast and continuous movements such as ballet. It is useful for analyzing detailed movements of the arms, legs, and spine during ballet movements and for evaluating balance or posture accuracy. Additionally, it supports multiple users, allowing each joint to be tracked individually when multiple users perform ballet movements simultaneously.

[0102] The process of visualizing the user's full-body movements using the modified OpenPose model used in the present invention is as follows.

[0103] (a) Input Image: The input image is a 2D image in RGB format containing one or more people captured by a camera, and can have various resolutions and sizes.

[0104] In this step, important features are extracted from the input image. A hybrid CNN is used to process the image across multiple layers to generate feature maps that capture features at various levels. Additionally, features are extracted at multiple scales to detect people of various sizes.

[0105] The hybrid CNN according to the present invention is a structure capable of performing initial feature extraction and basic pattern recognition using a lightweight CNN that provides fast processing speed, and performing precise feature extraction by applying a high-performance CNN to areas with complex poses or overlaps.

[0106] Lightweight CNNs are network structures designed with a small number of parameters and computational resources, enabling execution in limited environments such as mobile devices or embedded systems, and can provide fast inference speeds and low memory usage. Examples of lightweight CNNs include MobileNet, which uses Depthwise Separable Convolution; ShuffleNet, which applies channel shuffling techniques; and SqueezeNet, which utilizes a Fire Module structure. Among these, they can be selectively adopted and utilized depending on the environment in which the lightweight CNN is accepted.

[0107] High-performance CNNs are deep network structures designed for high accuracy and performance. Due to the large number of parameters and computational loads, they require powerful hardware resources. Examples of high-performance CNNs include VGGNet, which utilizes multiple convolutional layer stacks; ResNet, which uses residual connections; and Inception, which extracts multi-scale features. Among these, the specific CNN can be selectively adopted and utilized depending on the environment in which it is adopted.

[0108] Furthermore, the hybrid CNN features the ability for lightweight and high-performance CNNs to process images in parallel simultaneously, effectively combining features extracted from them to dynamically adjust or selectively activate the contributions of the lightweight and high-performance CNNs based on image complexity.

[0109] As a result, the present invention improves overall processing speed through basic feature extraction using a lightweight CNN, ensures accuracy in complex situations with a high-performance CNN, and enables efficient use of computing resources through selective processing based on the situation, thereby simultaneously achieving improved processing speed and accuracy.

[0110] (b) Generation of Part Confidence Maps: Estimate the location of body parts based on the feature maps extracted in step (a). Generate confidence maps for each of the 18 body parts (e.g., nose, neck, shoulder, elbow, wrist, etc.).

[0111] Each confidence map takes the form of a 2D heatmap representing the probability that a corresponding body part is located at a specific location in the image, and brighter areas indicate a higher probability that a specific joint is located at that position. Subsequently, a multi-stage CNN is used to progressively improve the accuracy of the confidence map.

[0112] In the Part Confidence Maps generation step according to the present invention, an adaptive multi-scale processing method that dynamically adjusts the scale according to the complexity of the image can be used to improve recognition accuracy and processing efficiency for people of various sizes, and the process is as follows.

[0113] First, to apply an adaptive multi-scale processing method, the complexity of the image must be evaluated by considering the size of the figures in the input image, the degree of overlap between figures, the complexity of the poses, and the complexity of the background.

[0114] Next, multi-scale processing is performed on a single input image by changing it to various sizes. For example, for complex images, a scale smaller than the original (image reduction) is used to grasp the overall structure, while a larger scale (closer to the original size) is used for complex parts to capture detailed features. On the other hand, for simple images, an intermediate scale is primarily used for processing, thereby reducing unnecessary scaling and increasing computational efficiency. To accurately estimate the poses of people of varying sizes within an image, an adaptive kernel method is utilized to dynamically adjust the size of the convolution kernel based on image characteristics to control the scale of the input image; or a multi-scale feature fusion method is employed to generate feature maps of different sizes and effectively combine them.

[0115] The adaptive multi-scale processing method according to the present invention combines a density-based attention mechanism that can more accurately extract important features by allocating more computational resources to complex regions of an image, and an adaptive weighting method that evaluates the importance of features extracted at each scale and dynamically assigns weights according to importance, so that they operate complementarily to improve recognition accuracy and processing efficiency for people of various sizes.

[0116] In this process, a density-based attention mechanism operates first to identify complex regions of the image and allocate computational resources, after which an adaptive weighting method evaluates the importance of features extracted from each region and assigns weights, thereby enabling more accurate and efficient estimation of person poses of various sizes and complexities.

[0117] (c) Estimation of Part Affinity Fields (PAFs): PAFs, which are arrow-shaped vector fields shown on the right side of the image in step (c) illustrated in Fig. 3, are 2D vector fields representing the association between body parts. From the confidence map generated in step (b), 38 PAFs are generated for pairs of connected body parts, i.e., two joint points (e.g., shoulder-elbow, elbow-wrist). Each PAF represents the direction and strength between two connected body parts.

[0118] In this stage, the connection strength between body parts can be dynamically weighted according to the situation to improve accuracy in situations with a lot of overlap and reduce false detections.

[0119] To adjust the weights, the reliability or degree of overlap of the joint points must be considered.

[0120] When adjusting weights based on joint point reliability, the weights are adjusted according to the reliability of joint point detection in the confidence map, and higher weights are assigned to joint point connections with high reliability.

[0121] When adjusting weights based on the degree of overlap, the average value of each PAF vector is used to adjust the weights in areas where the body parts of multiple people overlap, thereby enabling accurate connection of body parts even in situations where there is overlap.

[0122] Through this dynamic weight assignment, the PAF can effectively estimate the poses of multiple people even in complex situations, and in step (c), the accuracy of the PAF is improved through a multi-stage CNN, similar to step (b).

[0123] (d) Bipartite Matching: The PAF is used to determine how joint points between each body part should be connected to one another. For each connection, a connection score is calculated by integrating along the PAF, and only connections exceeding a threshold are retained, while the rest are removed.

[0124] (e) Parsing Results: This is the step of finally estimating the pose and generating results through pose assembly. Each body part is connected by lines to represent the overall pose, and as shown in Fig. 3, the postures of the two people are accurately distinguished and displayed.

[0125] - Pose Assembly: Construct the entire pose using K-partite graph problems. Divide this into multiple bipartite graph problems for efficient solving, and use a greedy algorithm to find the optimal connections.

[0126] - Final Output: Generates a final pose estimation result for each person, including detected keypoints and the connections between them. The result can be visualized over the original image or output in JSON format.

[0127] Through this stepwise processing, OpenPose according to the present invention can accurately estimate the pose of each individual even in images containing multiple people while maintaining a constant processing time regardless of the number of people, and provides an efficient system capable of real-time processing.

[0129] Next, HRNet (High-Resolution Network), a deep learning-based keypoint extraction model, can precisely extract joint positions even in high-resolution images, excelling in detailed motion analysis and stably tracking joint positions even under lighting changes or complex backgrounds. With higher precision in joint position tracking than OpenPose, it is particularly effective for high-precision tasks such as analyzing small angle changes or minute differences in movement like ballet movements, as well as balance analysis.

[0130] Furthermore, DeepLabCut enables custom training tailored to specific movements or joints, allowing for specialized motion analysis in ballet and providing high accuracy with only a small amount of training data. While it is useful for individually learning specific ballet movements (e.g., leg lifts, twists, etc.) and analyzing repetitive motion patterns, its real-time processing performance lags behind that of OpenPose or HRNet.

[0131] Furthermore, while CNNs are effective for extracting features from images and can be utilized for simple motion analysis, they are insufficient for precise motion analysis—such as in ballet—because sophisticated joint tracking like that of OpenPose or HRNet is difficult with CNNs alone and requires additional structures or algorithms; they must be used in combination with other advanced algorithms.

[0133] As explained earlier, due to the individual characteristics of each deep learning-based keypoint extraction model, it is necessary to select and use an appropriate model depending on the situation.

[0134] For example, OpenPose according to the present invention is used when real-time analysis is important, HRNet is used when precise joint tracking is required, and DeepLabCut is used when customized learning of specific movements is required.

[0135] In addition, for ballet movement analysis, it is advisable to use a combination of OpenPose and HRNet, or to utilize DeepLabCut complementarily when the analysis of specific movements is required, but this is not limited to these methods.

[0136] In ballet motion analysis, analyzing ballet movements by combining the modified OpenPose and HRNet enables the simultaneous achievement of accuracy and real-time capability by rapidly extracting joint data with the modified OpenPose and finely correcting it with HRNet. The combination method and workflow between them are as follows.

[0137] [Step 1] Extract initial joint data using the modified OpenPose.

[0138] The image or video input from the camera is passed to the modified OpenPose, and the modified OpenPose extracts the positions of major joints (shoulders, elbows, knees, etc.) in the form of heatmaps and skeletons.

[0139] The extracted joint data is stored as 2D coordinates, and the separated and sorted data is passed to the next step, and the data is passed in the following form.

[0140] Output example: {Joint ID, x coordinate, y coordinate, probability}

[0141] [Step 2] Correct the joint data using HRNet.

[0142] The 2D joint coordinates generated from the modified OpenPose and the original image are input into HRNet, and HRNet corrects the detailed positions of the modified OpenPose data through a high-resolution network.

[0143] For example, it precisely adjusts minute errors in coordinates estimated by the modified OpenPose at joints such as elbows and knees, and outputs corrected joint data as new 2D or 3D coordinates.

[0144] [Step 3] Fusion and output the data.

[0145] The final result is derived by comparing the joint data from the modified OpenPose and HRNet. In this process, the data rapidly provided by the modified OpenPose is used for real-time analysis, while the HRNet data is utilized to improve accuracy based on corrected coordinates.

[0146] The final output visualizes joint data and motion patterns in a skeleton structure, providing real-time feedback to the user.

[0148] The joint position extraction unit according to the present invention may also correct the position of the joint by attaching a marker to provide high precision by supplementing the results of a deep learning-based joint tracking algorithm (modified OpenPose, HRNet, etc.).

[0149] When a user attaches a marker as shown in FIG. 4 and performs a ballet movement, the joint position extraction unit can analyze the relative positional relationship and pattern between the markers captured by the camera to precisely correct the joint position extracted from the deep learning analysis model, and the process is as shown in FIG. 5.

[0150] (S51) Marker placement and filming: Markers are attached to the tights and skin at regular intervals between joint areas (shoulder, elbow, knee, etc.) and between joints (thigh, calf, etc.), and the user's movements are filmed with a high-resolution camera to collect position data of the markers. Each marker is designed to use a unique color, pattern, reflective material, etc., to increase recognition accuracy, and the markers are placed between joints. In cases where the markers are obscured by the ballet costume, an attached sensor (43) may be used instead of the markers, as shown in FIG. 4.

[0151] (S52) Analysis of relative positions between markers: The center coordinates of the markers are extracted, and the distance and angle between the markers are calculated. The joint position is calculated based on the relative positions between the markers, and the joint position is predicted according to a specific rule.

[0152] (S53) Combination of marker data and joint data: Combine joint data and marker position data extracted from a deep learning model such as modified OpenPose or HRNet. At this time, if the joint data provides approximate coordinates, finely adjust the coordinates using the distance and angle between markers.

[0153] (S54) Marker pattern-based joint position correction

[0154] - Setting the reference point: Set the most stable position among the markers on the tights (e.g., hip or shoulder marker) as the reference point.

[0155] - Joint Position Calculation: Corrects the joint position using data (distance, angle) around the marker corresponding to each joint. Example: The elbow position is calculated based on the relative position between the upper arm (shoulder-elbow) marker and the forearm (elbow-wrist) marker.

[0156] (S55) Derivation of correction result: Final joint coordinates are calculated by fusing marker-based calculation results with deep learning joint data.

[0157] By using markers to precisely correct the joint positions extracted from a deep learning analysis model, the angles of the legs and arms can be calculated precisely, and the user's balance and posture can be evaluated during complex ballet movements (e.g., lifting the leg, twisting).

[0159] In the following, we will describe how the user's condition is ultimately determined by combining physical conditions (e.g., joint range of motion, muscle strength, balance ability) measured by the sensor unit (electromyography (EMG) sensor, pressure sensor, or inertial measurement unit (IMU)) with the results of an analysis by the image processing module of the user's movements following ballet movements displayed on the display, and explain the target items of the comprehensive analysis results based on this.

[0160] First, the process of determining the final user state by combining the physical condition measured by the sensor unit with the results of the analysis by the image processing module of the user's movements following the ballet movements displayed on the screen is as follows.

[0161] Physical data (acceleration, electromyography, pressure) collected from the sensor unit is combined with joint position data extracted from the image processing module. This process is carried out through two main steps.

[0162] - Time Synchronization: Ensures consistency in motion analysis by aligning sensor and image processing data along the same time axis. For example, joint acceleration data measured by an IMU sensor and joint position change data extracted by a camera are aligned along the time axis.

[0163] - Sensor Fusion Algorithm: Combines sensor and image data to evaluate user status more precisely. For example, it compares image data regarding joint positions with IMU data to correct sensor errors or supplement insufficient information.

[0164] Sensor fusion algorithms combine data collected from various types of sensors to provide precise and reliable user state information that is difficult to obtain from a single sensor. This is achieved by mutually complementing data between sensors or correcting errors, and the process by which sensor fusion algorithms evaluate the user's state by combining joint position data and sensor data is as follows (the IMU will be used as an example below).

[0165] 1) Data Supplementation

[0166] Video data can accurately track the position and movement of joints through external observation using a camera, but it can be affected by environmental factors such as lighting, camera angle, and occlusion (parts obscured by other objects).

[0167] The IMU tracks the user's joint movements in real time and can provide motion information regardless of lighting conditions, but data drift (cumulative error) may occur.

[0168] Sensor fusion algorithms combine and complement the strengths of two sets of data. For example: if the camera misses some joint data due to occlusion, it uses IMU data to compensate, and if drift occurs in the IMU data, it corrects the drift based on the camera data.

[0169] 2) Sensor data comparison and calibration

[0170] Sensor fusion algorithms correct errors by analyzing the correlation between two data sets.

[0171] - Time Synchronization: Since IMU data and video data can be collected at different rates, the time axes of the data are aligned. For example, 100Hz data from the IMU and 30FPS data from the camera are synchronized to compare data at the same time interval.

[0172] - Data Consistency Verification: Joint position data extracted from the camera is compared with joint acceleration and angular velocity data calculated by the IMU to check for discrepancies. For example, if the camera data indicates the arm is slightly raised, but the IMU data determines that the arm has moved downward, the IMU data is considered to be experiencing drift and is corrected accordingly.

[0173] 3) Generation of combined data

[0174] Sensor fusion algorithms generate combined data based on the reliability of different data.

[0175] - Weighted Fusion: Data is integrated by setting the reliability of each sensor as a weight. For example, if camera data has high reliability, that data is reflected more, while IMU data is used secondarily.

[0176] - Application of filtering algorithms: Uses algorithms such as the Kalman filter to remove noise and minimize uncertainty that may arise during the process of combining the two data sets.

[0177] 4) Output and Status Evaluation

[0178] Combined data is used to evaluate user status in the following ways

[0179] - Accurate calculation of joint angles and positions: When video data and IMU data are combined, joint positions and angles can be calculated more accurately. For example, it is possible to evaluate how well the leg angles in ballet movements match the target movement.

[0180] - Precise analysis of movement: Utilizes acceleration and angular velocity data from the IMU to analyze the speed and pattern of motion, and evaluates detailed movements by comparing this with changes in joint positions in video data.

[0182] Next, the results of fusing sensor data and image data are used to derive comprehensive analysis results by evaluating the user's state from various angles, and these results are divided into the following key items.

[0183] - Disability Assessment: Quantitatively evaluates the user's disability status based on joint angles, range of motion, muscle activity, etc. For example, if movement in a specific joint is restricted or muscle signals are abnormally weak, the possibility of a musculoskeletal disorder can be diagnosed.

[0184] - Muscle Development Assessment: Analyzes muscle activity data to evaluate the user's overall muscle development status and the development status of specific muscle groups.

[0185] - Muscle Strength Assessment: Evaluates muscle strength status by measuring muscle activity during specific movements using electromyography (EMG) data. Checks for the presence of weakened muscle groups.

[0186] - Symmetry Assessment: Evaluate the left-right symmetry of the body during movement performance to identify problems of overloading one side of the muscles or joints.

[0187] - Injury Risk Assessment: It detects excessive stress on joints or muscles during movement performance to assess the risk of injury. For example, if excessive load is applied to a specific joint or asymmetrical movements occur during repetitive actions, it can warn of a potential injury and recommend preventive measures. It can also be used to evaluate the severity of existing injuries and monitor rehabilitation progress.

[0188] - Joint Range of Motion: Measures the maximum range of motion of the joints using video and sensor data. This is used to precisely evaluate the user's flexibility and movement performance ability.

[0189] - Flexibility Measurement: Evaluates body flexibility by analyzing joint angle data. Verifyes whether joint movement during specific actions has reached the target level.

[0190] - Static Posture Assessment: Evaluates the user's fixed posture to analyze balance and muscle tone. It can identify physical load caused by incorrect posture.

[0191] - Dynamic Movement Pattern Analysis: Analyzes overall movement patterns to evaluate the continuity and stability of the movement. This is useful for analyzing sophisticated movements, such as those in ballet.

[0192] - Balance Ability: Analyzes the user's balance status using IMU and pressure sensor data. Measures balance ability by evaluating weight distribution and center of gravity shift during movement.

[0193] - Load distribution evaluation: Analyzes weight distribution while walking or standing by measuring the load distribution on the soles of the feet using pressure sensors.

[0194] - Comprehensive Status Analysis and Result Derivation: The user's overall condition is evaluated by comprehensively considering factors such as the degree of disability, muscle development, and injury risk. For example, for a ballet learner, movement accuracy, flexibility, muscle strength, and balance ability are assessed based on their current state; based on this, areas requiring improvement and optimal training plans can be presented. Customized reports tailored to various user groups, such as athletes, rehabilitation patients, or the general public for health management, can be provided.

[0195] As such, the fusion results of sensor data and image data are not limited to single-item analysis but analyze the user's physical condition from multiple angles and provide an integrated evaluation; this can be applied to various fields such as disability diagnosis, improvement of exercise performance, health management, and injury prevention, and can be usefully utilized in the following application areas.

[0196] - Assessment of Disability and Rehabilitation Support: The severity of musculoskeletal disorders can be diagnosed through the analysis of joint angles and movements. For example, if the range of motion of a specific joint falls short of the normal range or the speed of movement is slow, issues such as lack of joint flexibility or muscle weakness can be identified. Based on this, physical therapy programs can be designed, or personalized rehabilitation training can be proposed. Through this process, the severity of the disability can be assessed, and rehabilitation or treatment plans can be recommended if necessary.

[0197] - Muscle Strengthening Training for Normal Individuals: Even if a user possesses a normal physical condition, the level of muscle development can be analyzed through data. For example, by analyzing the strength and balance of leg muscles during movement, it is possible to assess cases where the strength of specific muscle groups (e.g., quadriceps, gastrocnemius) is insufficient or skewed to one side. This assessment can be utilized to provide muscle development status reports or suggest strengthening exercises for specific body parts to athletes, dancers, or the general public seeking fitness management.

[0198] - Exercise Equipment Recommendation: The output and status evaluation results can be utilized to recommend optimal exercise equipment based on the user's physical condition and learning goals. By synthesizing the user's learning or rehabilitation level determined by the learning module with data analyzed by the image processing module, the user's joint range of motion, muscle strength level, and balance ability are evaluated in detail, and suitable exercise equipment is suggested to improve or supplement these capabilities. For example, a user with a limited range of motion may be recommended stretching bands or resistance bands to improve flexibility, while a user with poor balance may use a balance board or stabilization training tool.

[0199] The exercise equipment recommendation process goes beyond simply compensating for the user's weaknesses; it focuses on increasing the efficiency and achievement of ballet movements through the use of equipment linked to learning goals. For example, users who need to strengthen can train specific muscle groups using leg press machines or dumbbells, and can use equipment such as foam rollers or Pilates rings to correct postural problems that repeatedly occur during the performance of specific ballet movements.

[0200] Furthermore, if a user struggles to maintain balance while performing the ballet "Arabesque," a balance board may be recommended as a training tool. Utilizing this equipment can strengthen weight transfer and centering abilities, thereby improving the stability and support required for the Arabesque movement. Similarly, if a lack of muscle strength is observed during high leg lifts, strength training using resistance bands helps facilitate the execution of the target movement more easily.

[0201] - Exercise Recommendations for Learning Ballet: Based on the analysis of ballet movements, it is possible to recommend exercises designed to compensate for the user's weaknesses. For example, if flexibility is lacking, exercises to improve it, such as stretching and yoga, are suggested; if balance is poor, balance training such as single-leg standing and balance board exercises is recommended; if muscle strength is insufficient, strength training like squats and lunges to strengthen leg muscles is recommended; and if endurance is lacking, aerobic exercises may be suggested to improve the ability to perform sustained movements required in ballet.

[0202] - Movement Correction and Skill Improvement: In activities requiring intricate movements, such as ballet or gymnastics, it evaluates the user's current movement patterns to provide feedback that enables the learning of more efficient and accurate movements. For example, it can provide specific advice to correct issues such as insufficient toe-pointing angle or asymmetrical posture.

[0203] - Injury Prevention: Injury risk can be assessed using sensing data. For example, it detects cases where excessive pressure is applied to specific joints during repetitive movements and suggests posture advice or alternative exercises to minimize this pressure.

[0204] - Sports Performance Enhancement: Targeting athletes or dancers, we provide customized training plans to maximize performance by analyzing the speed, power, and balance of specific movements. For example, to improve jump height in ballet, we analyze muscle strength and propulsion during the jump and recommend plyometric training based on the findings.

[0205] - Fitness and Wellness: Can assess the physical fitness and exercise capabilities of the general public to provide advice on managing health in daily life. For example, it recommends yoga, Pilates, or home training exercises based on joint range of motion and balance ability.

[0206] - Cognitive-Motor Training: Brain-body coordination is evaluated through sophisticated movements such as ballet, and based on this, training is designed to simultaneously improve cognitive and motor skills. For example, the user's reaction speed and sense of rhythm may be analyzed while performing movements to a specific rhythm, and training to improve them may be provided.

[0207] User status assessment performed by combining sensing and video data, as described above, can be utilized in various fields such as rehabilitation, exercise recommendation, injury prevention, sports improvement, and wellness management, going beyond simple motion analysis.

[0208] [Customized Ballet Movement Design Module]

[0209] The image processing system according to the present invention is configured to further include a customized ballet movement design module, wherein the ballet movement design module generates ballet movements suitable for the user based on data analyzed by the body condition analysis module, presents the generated ballet movements on a display, and adjusts the difficulty and intensity of the movements to suit the user according to the user's learning or rehabilitation situation while following the presented movements, thereby designing ballet movements so that the user can continuously develop without overexerting themselves beyond their limits.

[0210] The customized ballet movement design module can generate new customized learning movements by selecting from movements stored in the movement library based on the user's body data and learning goals, or by combining multiple movement elements.

[0211] For example, if a high-difficulty Grand Jeté movement in ballet is set as a learning goal, the detailed elements required to achieve it can be analyzed to select and combine suitable movements or other exercises.

[0212] To perfectly execute the Grand Jeté, explosive leg strength and flexibility, an excellent sense of balance, and harmonious movement between the upper and lower body are essential. To achieve this, a customized training program is designed to strengthen the necessary elements after analyzing the user's current strength, flexibility, and balance. For example, basic strength exercises such as squats and lunges are included to reinforce leg strength, while hamstring and hip stretching exercises are added to improve flexibility. Additionally, exercises involving balancing on one leg or utilizing a balance board can be designed to train the sense of balance.

[0213] In addition, the detailed movements of the grand jeté, which is the learning objective, are trained in stages. For example, to improve the stability of the landing movement, simple jump and landing drills are repeated, and the target movement is gradually perfected by connecting this to exercises of extending the leg in the air.

[0214] These combinations of movements are optimized based on the user's current state and can provide results optimized for learning goals. Through this, users can effectively learn advanced ballet movements and develop the necessary physical elements step by step.

[0215] The customized ballet movement design module is updated and optimized in real-time to effectively support the user's learning and rehabilitation processes by adjusting difficulty levels based on movement data or suggesting alternative movements as needed. For instance, if a user struggles with a specific movement, the module helps them adapt by lowering the difficulty or modifying parts of the movement. Conversely, if a user achieves a specific goal quickly, it recommends higher-difficulty movements to promote continuous progress.

[0216] The customized ballet movement design module generates ballet movements tailored to the user's physical condition, rehabilitation goals, or learning situation. This process utilizes data collected through sensors and image processing modules, including joint range of motion, muscle strength, and balance. For example, when a user performs a leg lift, the module analyzes thigh muscle strength and joint angles to design a suitable movement within the available range. This approach helps users perform movements safely and effectively.

[0217] The custom ballet movement design module selects the most suitable movement from the movement library or creates new custom movements by combining multiple movement elements. For example, it is possible to design a movement that combines leg lifting with arm movements to train upper and lower body balance simultaneously. This approach not only meets the individual needs of users but also generates movement combinations optimized for specific rehabilitation goals (e.g., improving flexibility, strengthening muscles).

[0218] Movements generated by the customized ballet movement design module adjust difficulty and intensity according to the user's current condition. For beginners, it presents basic balance movements or low-angle leg lifts, gradually increasing the difficulty. For example, if the design initially requires a leg lift at a 30% angle, it supports gradual growth by expanding the target angle to 50% as muscle strength increases. Through these adjustments, the customized ballet movement design module is engineered to enable users to continuously improve without overexerting themselves beyond their limits.

[0219] Subsequently, the design gradually expands the target range based on learning achievement or rehabilitation progress to reach up to 100%. For example, leg lifting and twisting movements are set based on the maximum angle the user can currently perform, and arm movements are adjusted in a similar manner to maximize the effects of gradual learning and rehabilitation.

[0220] The image processing system according to the present invention stores individual body data and tracks changes in condition occurring during rehabilitation treatment based on this data. For example, if the flexibility of a user's specific joint improves or muscle strength increases during the rehabilitation process, the system automatically expands the range of motion that can be performed. Conversely, if the user feels fatigued or experiences difficulty with a specific movement, the system responds to the user's condition in real time by lowering the difficulty level or recommending alternative movements.

[0221] The customized ballet movement design module tracks the user's rehabilitation progress or learning development, designing more challenging movements or setting new goals based on their improved condition. For example, it initially provides movements focused on flexibility and balance, but as rehabilitation progresses, movements aimed at strengthening muscles and improving technique are added. This approach goes beyond simply enhancing movement performance; it personalizes the user's ballet learning experience to provide a continuous sense of accomplishment.

[0222] In addition, the customized ballet movement design module according to the present invention uses artificial intelligence to predict the future state based on the user's current data and further refines the movement design based on this. For example, if joint flexibility improves by 5% every week in current rehabilitation data, the AI ​​predicts the range of motion the user can perform in a few weeks and designs the next steps based on this, and the prediction results are provided to the user and the therapist to help set goals and establish training plans.

[0223] In addition, AI can learn from a large amount of user data to simultaneously reflect common patterns and individual characteristics.

[0224] - Common Patterns: Learns from various user data to derive the most effective movements for specific rehabilitation goals (e.g., flexibility improvement) or learning objectives. For instance, if data analysis reveals that a specific stretching movement improves flexibility for the majority of users, it is set as the recommended movement.

[0225] - Personalized Optimization: AI analyzes specific user data to suggest customized movements tailored to the individual's physical condition, learning speed, and preferences.

[0226] In the long term, motion libraries can leverage artificial intelligence (AI) to become increasingly sophisticated based on users' rehabilitation progress and training data. AI plays a crucial role in data analysis, updating training models, and designing user-customized motions.

[0227] AI can add new ballet movements designed by experts to the library and evaluate the effectiveness of existing movements to improve them if necessary. For example, it analyzes various user data to measure how effective specific movements are on average for achieving rehabilitation or learning goals, and then updates the library by improving movements with low success rates or adding alternative movements.

[0228] AI collects and analyzes feedback (e.g., difficulty, effectiveness) provided by users after performing actions to evaluate the suitability of the actions and improves the model with new training data. For instance, if a user evaluates a particular action as "too difficult," the AI ​​may automatically adjust the difficulty or suggest an alternative action.

[0229] In addition, if it is determined that a user has a disability or requires rehabilitation based on personal information stored in the storage unit or physical characteristics identified by the physical condition analysis module, the customized ballet movement design module may substitute movements that the user can perform by considering the disability or missing parts according to the user's current condition (whether a disability or rehabilitation is required). For example, in the case of a user with a limited range of motion in a specific joint, ballet movements suitable for the user may be designed by designing alternative movements that minimize or exclude the movement of that joint.

[0231] The customized ballet movement design module is designed based on movements stored in the movement library. The ballet movement videos stored in the movement library may include postures, basic movements, and sequences. Among the ballet movement videos, postures may vary depending on the position of the feet, arms, and body; postures based on arm position may include en bas, anavent, en haut, à la scôte, and alonge. Among the ballet movement videos, basic movements may include plié, re-révé, tandoux, rond de jambe, pirouette, and jatte de chat, and sequences may be movements that combine these postures and basic movements.

[0232] The motion library is a core component of the ballet motion design module and is a database storing various ballet movements designed by experts. Structured with detailed metadata such as the difficulty, intensity, target muscle groups, and balance requirements of each movement, this library is utilized to select or design movements tailored to the user's physical condition, rehabilitation goals, or learning situation. For example, it is broadly composed of movements ranging from simple balance exercises for beginners to complex movements requiring advanced technical skills, enabling it to meet diverse user needs.

[0233] Each movement stored in the motion library is designed based on specific metadata. Difficulty levels are classified into beginner, intermediate, and advanced to suggest movements suited to the user's current skill or rehabilitation level, while intensity specifies the required muscle strength for execution, allowing users to select movements appropriate for their fitness and muscle condition. Additionally, each movement defines a target muscle group, designed to strengthen specific muscle groups or focus on rehabilitation therapy. Balance requirements are also included, providing an opportunity to identify and improve the technical elements required to maintain physical balance during movement execution.

[0234] In addition, ballet videos stored in the motion library according to the present invention contain information on age, gender, and age suitable for each performance, as well as information on the type of disability and disability grade that makes it difficult for the user to follow the video. Through this, the customized ballet movement design module can design ballet movements suitable for the user by suggesting alternative movements based on the need for disability or rehabilitation, thereby enabling the user to learn safely and effectively. Furthermore, by making corrections based on the degree of disability or rehabilitation when the learning module judges the user's achievement, it provides a fair evaluation and learning environment that considers the user's physical limitations, allowing even users with disabilities to feel a sense of accomplishment during the learning process.

[0235] The movement library supports users' long-term rehabilitation and ballet learning, becoming more sophisticated over time. New movements are added by experts, and existing movements are improved based on user data. Furthermore, it provides AI-based features that learn from data from multiple users to automatically suggest the most effective movement combinations or rehabilitation strategies. Through this, the library transcends a simple database to establish itself as a powerful tool for personalized ballet learning and rehabilitation therapy.

[0236] AI-based motion libraries become more sophisticated and expand as the number of users and data increases over time. By integrally learning from a large volume of user data, more generalized and effective motion design principles can be derived. At the same time, fully personalized training plans can be designed using individual user data, allowing the system to evolve into an autonomous and adaptive system that satisfies all users.

[0238] The learning module according to the present invention performs various functions to enable the user to learn ballet movements suitable for them and effectively achieve rehabilitation goals. This module is closely linked with a body condition analysis module, a ballet movement design module, and an image processing module to provide a user-customized learning experience.

[0239] First, the learning module provides user-customized movement guidance through the display. The display visually presents the results of the user's physical condition analysis and the customized movements generated by the ballet movement design module.

[0240] A motion guide is displayed on the display screen, and the posture, sequence of movements, angles, etc., of the action to be performed by the user are shown as animations or step-by-step explanations. For example, a motion trajectory is displayed on the screen along with a message such as "Lift your leg at a 45-degree angle."

[0241] The learning module collects and analyzes motion data performed by the user and evaluates current learning achievement and rehabilitation progress. To this end, the learning module collects data such as joint position, movement accuracy, and muscle activity in real time from image processing modules and sensors, and quantitatively analyzes how close the user is to set rehabilitation or learning goals based on the collected data. For example, it evaluates whether the angle and balance maintenance time during a leg-lifting movement have reached the standard values.

[0242] Depending on the analyzed degree of rehabilitation and learning achievement, the learning module can request the creation of new movements or request alternative movements from the ballet movement design module.

[0243] If the user successfully performs a current movement or records a certain level of achievement, the learning module may request a new movement from the ballet movement design module when the difficulty and intensity are gradually increased. For example: if the user successfully performs a basic leg lift, it requests an advanced movement of lifting the leg while raising one hand to maintain balance.

[0244] In addition, if a user is unable to repeatedly perform a specific action, the overall learning process can be designed to lower the difficulty or request an alternative action to ensure that the user's limitations are not exceeded.

[0245] The learning module monitors the user's actions in real time via a camera, and the image processing module analyzes the captured data to track the user's posture and movements. While the user performs an action, the display visually indicates accuracy by comparing the current movement with the target movement in real time, and provides feedback to help maintain the correct posture and movement flow. For example, if the user loses balance or the angle is incorrect during a leg-lifting exercise, immediate visual or auditory feedback guides them to the correct posture. This real-time feedback helps the user follow the learning curve more quickly. For instance, if the user's leg position fails to reach the target angle, the screen may display a warning such as "Lift it a little higher" along with directions for correction.

[0246] The learning module can clearly display the sequence of movements on the display to enable learning by breaking down complex ballet movements into steps according to the user's learning status and needs. For example, if the first step is to perform the movement of 'extending toes and maintaining balance,' the second step can provide a step-by-step learning guide such as 'lifting the leg.'

[0247] In addition, by linking with user data, it can suggest movements of appropriate difficulty based on the user's physical condition and learning achievement data, adjust the difficulty by reflecting real-time feedback, recommend movements requiring additional training by analyzing deficiencies in specific movements from user data, and prevent overload by analyzing the user's muscle fatigue to adjust the number of repetitions or recommend rest time.

[0248] The learning module may generate a personalized profile based on the user's learning history, movement performance ability, and rehabilitation progress data, and provide user-customized learning or rehabilitation plans, such as suggesting the next movements to perform, learning goals, or necessary rehabilitation processes based on the generated profile. Additionally, the learning module may analyze data trends over time to visualize changes in the user's achievement and set long-term rehabilitation and learning goals.

[0249] The learning module according to the present invention provides appropriate feedback and results to a user by considering the disabled or missing parts when the user has a disability or requires rehabilitation treatment, replacing the movements with those the user can perform, or by evaluating based on non-disabled parts. For example, in the case of a user with a limited range of motion in a specific joint, alternative movements that minimize or exclude the movement of the relevant joint are designed and presented on a display to enable the user to learn safely and effectively.

[0250] Furthermore, if the learning module includes movements that are difficult for the user to perform, it automatically converts them into alternative movements or evaluates performance based on the results of the available body parts. For example, if a user with limited movement in one arm attempts to perform a ballet movement, the evaluation is based on the movement of the other arm, and the overall performance is calculated by reflecting the performance of the non-disabled body parts as a weight. This ensures that users do not feel alienated during the learning process and can receive feedback tailored to their abilities.

[0251] Furthermore, the display clearly shows the movements the user can currently perform and alternative movements, and the learning module reflects achievement based on the user's non-disabled body parts. For example, if a user is unable to use one leg during a leg-lifting exercise, the achievement is calculated based on the angle and balance duration of the movement performed with the non-disabled leg; this is considered equivalent to the achievement of the non-disabled leg, and feedback is provided to the user. This approach provides a fair evaluation and learning environment that takes into account the user's physical limitations, and is designed to allow users with disabilities to experience a sense of accomplishment during the learning process.

[0253] The learning module can provide various forms of feedback to enable real-time evaluation and correction of the accuracy and improvement of movements as the user learns ballet movements or performs rehabilitation training.

[0254] Visual feedback displays action comparison results on the display in the form of text, graphs, or animations, helping users intuitively understand the difference between their actions and the target actions.

[0255] Auditory feedback provides specific instructions to help the user improve their movements through voice guidance, and tactile feedback utilizes vibration signals from a wearable device to convey to the user errors in movement or directions for correction.

[0256] The feedback in the learning modules also supports personalized settings based on the user's preferences and learning goals. Users can select their preferred method among visual, auditory, and tactile feedback, and adjust the difficulty level to meet the needs of a wide range of learners, from beginners to advanced students. These features help users feel a sense of accomplishment during the learning process and enable them to continue rehabilitation training or ballet studies.

[0257] In addition, feedback provided by the user after performing a ballet movement presented by the learning module (e.g., difficulty, effectiveness) can be applied to adjust the difficulty of the ballet movement presented to the user or to select an alternative movement, thereby being used to design user-customized ballet movements.

[0258] The feedback in the learning modules tracks the user's learning progress and rehabilitation progress, enabling the setting of long-term learning and rehabilitation goals based on this data. User progress is displayed in the form of graphs or visual reports, reinforcing learning motivation by clearly presenting the difference between current achievement and goals.

[0259] The learning module records and analyzes all movements performed by the user and feedback data to support the setting of long-term learning and rehabilitation goals. During this process, the user's rehabilitation progress and learning progress are evaluated, and the recorded data is utilized to systematically track the user's condition. The stored data is accumulated as foundational information that enables the quantitative analysis of changes in the user's condition, such as joint range of motion, changes in muscle strength, and movement accuracy.

[0260] The analyzed data is provided to users and therapists in the form of visual reports containing graphs, tables, or brief explanations. These reports clearly present the gap between current achievement and goals, helping users easily understand their progress and maintain their motivation to learn. At the same time, therapists can refer to these reports to gain a more precise understanding of the user's condition and effectively establish treatment goals and learning plans.

[0261] Furthermore, the learning module sets new rehabilitation goals or adjusts existing ones based on analysis results. For instance, if a specific movement consistently causes difficulty, it lowers the difficulty level or suggests alternative movements, enabling the user to make continuous progress without overexerting themselves. In this way, the learning module supports collaboration between the user and the therapist and plays a role in optimizing the rehabilitation and learning process.

[0262] The learning module intuitively guides the process of movement execution by displaying user-customized movements and the user's movements on a screen, enabling users to effectively perform rehabilitation and ballet learning. Through this, users can clearly understand the posture and flow of the movements to be performed and receive necessary instructions during execution.

[0263] The process by which a user proceeds with learning or rehabilitation through ballet is as follows.

[0264] In the learning module, the user follows the customized movements presented through the display, and the camera captures the user's ballet movements in real time and transmits them to the video processing module.

[0265] The image processing module rapidly extracts joint data from the video received from the camera using OpenPose, evaluates approximate motion based on this, transmits the extracted data to HRNet to finely correct joint positions, and precisely calculates the angles and balance of the legs and arms. It can also precisely correct joint positions extracted by a deep learning analysis model by analyzing the relative positional relationships and patterns between markers captured by the camera.

[0266] The result of fusing sensor data and image data in the image processing module evaluates the user state from various angles to derive a comprehensive analysis result, which is then transmitted to the learning module.

[0267] Based on the results derived from the image processing module, the learning module visualizes the user's learning or rehabilitation progress to display the difference between the movements performed by the user and the target movements, learning achievement or rehabilitation progress status, avatars, etc., through the display.

[0268] In addition, the learning module updates the motion library using artificial intelligence (AI) based on the user's rehabilitation progress, learning data, feedback provided to the user by the learning module, or user feedback.

[0269] The display function of the system according to the present invention combines a display with AR (augmented reality) / VR (virtual reality) technology to support users in learning ballet movements more intuitively and effectively, thereby enabling the more effective achievement of rehabilitation and learning goals through an immersive learning environment and real-time feedback.

[0270] AR technology provides more intuitive movement guidance by overlaying digital content onto a real environment. It can provide guidelines by visualizing the ballet movements the user needs to perform as an augmented 3D model by overlaying a 3D skeleton or trajectory of the target movement onto the user's actual screen.

[0271] For example, when a user lifts their leg, the target position appears as a 3D trajectory on the display, and the user's current leg position is overlaid and displayed on top of it.

[0272] In addition, a virtual ballet teacher (guide character) appears as an AR character to demonstrate the movements the user must follow and guide them on the correct posture. For example, the AR ballet teacher says, "Raise your arms to this position," and demonstrates the corresponding posture.

[0273] VR (Virtual Reality) technology immerses users in a virtual environment, allowing them to experience ballet learning and rehabilitation processes more vividly. By providing various virtual practice spaces, it enables users to wear a VR headset and train in a virtual ballet studio, practicing a wide range of movements without the constraints of the real environment. For example, users can practice jump movements in a virtual ballet studio, performing the actions without worrying about space size or obstacles.

[0274] In addition, while a user performs an action in a VR environment, the results of analyzing joint positions and posture can be immediately provided as virtual visual feedback. For example, to correct an incorrect posture, the correct posture is visualized in 3D along with a message such as "Extend your elbow further." Since it is obvious to a person skilled in the art that a VR headset, virtual environment simulation software, and interaction devices (controllers, haptic feedback) are required to implement a VR environment, a description thereof will be omitted.

[0275] By combining displays with AR / VR technology, the learning effect can also be maximized as follows.

[0276] By enhancing immersion and combining real and virtual environments, learner concentration can be increased and learning effectiveness maximized. Additionally, by improving the precision of motion correction, it can help users perform precise corrections by allowing them to directly compare their movements with target movements in AR or VR.

[0277] In addition, it can be implemented as follows to provide fun and motivation, stimulate learning motivation through gamified AR / VR features, and enable users to continue rehabilitation training and ballet learning in an enjoyable way.

[0278] To motivate users to continue learning and rehabilitation training, the learning module includes a feature that changes the appearance of the user's avatar according to the level of learning. This avatar intuitively indicates learning achievement by showing a visually improved appearance as the user learns set ballet movements or achieves rehabilitation goals.

[0279] The avatar's appearance changes automatically based on the user's training data. In its initial state, the avatar wears simple practice clothes and has a blank expression, reflecting the state of a beginner. If the user successfully performs set ballet movements or achieves rehabilitation goals, the avatar gradually changes to sophisticated ballet attire and a confident expression. For example, when the success rate of leg lifts exceeds a certain level, the avatar upgrades from practice clothes to performance ballet attire, and a smile is added to its face.

[0280] The avatar's age changes to reflect learning progress. It starts with the appearance of a young child, but gradually evolves into the look of a mature ballerina as the user achieves learning goals. This allows users to visually track their learning progress while providing a sense of accomplishment and enjoyment.

[0281] Furthermore, when the learning module is implemented for rehabilitation training, the avatar's appearance can change to become progressively younger as the learning progresses. Initially, it starts with an older appearance, but as the user achieves rehabilitation goals, it gradually transforms into a younger and more energetic look. This change conveys a symbolic meaning that the user is regaining their physical abilities and developing into a more dynamic and flexible state during the rehabilitation process. Through this, users can visually verify their learning outcomes and experience both fun and a sense of accomplishment in rehabilitation and ballet learning.

[0282] These changes in avatar appearance serve as an important tool for motivating learning. Users can intuitively check their learning results through their avatars, experiencing a sense of accomplishment, and set specific goals such as "achieving a goal to upgrade the avatar to the next level." This helps users move beyond simple repetition of training and enjoy learning like a game.

[0283] As a result, changes in the avatar's appearance provide the fun of learning while visually expressing the user's achievement, making it an effective way to increase the sustainability of the learning and rehabilitation process. This feature strengthens the user's immersion and motivation, helping to achieve a more effective success rate in rehabilitation and ballet learning.

[0284] The image processing system according to the present invention includes a display, and is configured such that a learning module presents ballet movements for learning or rehabilitation to a user, and when the user performs the presented ballet movements, the results are displayed on the same display screen to allow comparison between the user's movements and the presented movements.

[0285] The learning module first presents target ballet movements to the user via a display. As the user follows and performs the presented movements, the image processing module analyzes the user's movements in real time and provides the analyzed data to the same display screen. During this process, the user's movements are compared with the presented ballet movements, and both movements are visually displayed simultaneously.

[0286] Motion comparison is performed in two ways. The first method involves overlaying the motion data performed by the user onto the presented motion. In this case, the user's motion trajectory or joint position appears on the presented motion in real time, helping the user clearly understand the difference between their own motion and the target motion. However, to minimize delays occurring during image processing and data analysis, the presented ballet motion is displayed with a delay of a certain amount (e.g., 0.5 seconds) reflecting the image processing time. This allows the user to compare their own motion with the target motion at the same point in time, facilitating real-time correction.

[0287] The second method utilizes the same display screen as divided areas, showing the presented ballet movement on one side and the movement performed by the user on the other. This approach clarifies the comparison process, helping the user visually and intuitively grasp the difference between their own movement and the target movement. Both the presented movement and the user's movement are displayed in real time, and the present movement is also displayed with a delay equal to the video processing time, allowing the two movements to be compared at the same point in time.

[0288] This display configuration provides users with real-time feedback and efficiently supports the ballet movement learning and rehabilitation process. Users can directly compare and correct their own movements with target movements on the same screen, thereby maximizing the effectiveness of learning and rehabilitation.

[0290] According to an embodiment of the present invention, by combining multiple cameras (350) and sensor data, ballet movements performed by a user can be implemented three-dimensionally in 360 degrees on a display. Through this, the user can observe their movements in real time from various angles and can precisely check their posture and balance status. This technology is applicable not only to still images but also to continuous motion images, and provides a function to view movements from various viewpoints while rotating or zooming in / out.

[0291] To implement a 360-degree stereoscopic display, multiple cameras are arranged in a circular or semi-circular configuration around the user to capture their movements from various angles. All user actions are captured by adjusting the focal length and field of view (FOV) between the cameras. Each camera utilizes a synchronization device to capture images at the same time, ensuring that all data is consistently collected along the same time axis through inter-camera time synchronization. These collected images from multiple angles are then transmitted to an image processing module. Based on the collected data, the image processing module reconstructs the user's movements into a 3D model. It rapidly generates initial data using OpenPose and then performs precision correction using HRNet to provide more accurate motion data.

[0292] The image processing module generates a high-precision 3D model by integrating data input from multiple cameras. In this process, image stitching algorithms are utilized to combine images from individual cameras into a single continuous model; depth information can be enhanced using stereo cameras or LiDAR sensors, while joint position and motion data are combined with sensor data (IMU, EMG, pressure sensors, etc.) to more accurately reproduce user movements. GPU-based real-time data processing technology rapidly performs this model generation and rendering, enabling users to verify their movements in real time.

[0293] The generated 3D data is provided to a display module, and in the continuous motion mode shown in FIG. 6 (a, b, c), the user's movement can be played back while the viewpoint is freely changed or the speed is adjusted, and in the still image mode shown in FIG. 6 (d, e, f), specific movements can be viewed from a fixed viewpoint. Through this, the present invention supports the user in understanding and correcting their movements more intuitively through the rotation and zoom functions of the display.

[0294] In addition, the 360-degree stereoscopic image displayed on the screen visually highlights incorrect posture by comparing the movements performed by the user with the target movements. For example, if the angle of the arms or legs is incorrect, the corresponding area is highlighted in color (e.g., red) so that the user can correct it immediately.

[0295] Users can manipulate the screen via a touchscreen, mouse, keyboard, or gesture recognition. For example, they can rotate movements by dragging the screen or zoom in and out using pinch gestures, and fix specific points in time or change the speed of movements using keyboard input. Additionally, captured data and analysis results are transmitted to a management server via a communication module for storage and analysis. This allows for the long-term evaluation of the user's learning and rehabilitation performance and enables the updating of customized learning plans if necessary.

[0297] The image processing system according to the present invention is configured to further include a storage unit.

[0298] The storage unit stores the user's personal information entered by the system input module or user terminal.

[0299] The personal information stored in the storage unit may include at least one of age, gender, and age. Additionally, if the user is a person with a disability or requires rehabilitation treatment, the storage unit may further include the type of disability, body parts requiring rehabilitation, and disability grade. This is because there may be limitations on the ballet movements that can be performed depending on the user's age, gender, age, and disability status.

[0300] The storage unit stores feedback regarding the difficulty or suitability of ballet movements perceived by the user in relation to the ballet movements presented by the learning module, which are input by the user using an input module or a user terminal. The stored feedback can be reflected in the movement design module by the learning module in the future to better satisfy the user's preferences and needs.

[0301] The storage unit stores individual physical condition analysis results, learning content, academic achievement or rehabilitation progress, and settings configured by the user.

[0302] The storage unit records and manages the results of analyzing the individual user's physical condition. This data includes key physical condition information such as joint range of motion, muscle strength levels, balance ability, and muscle activity. For example, if a user's joint angle limitations or the degree of weakness in specific muscle groups are analyzed, this information is stored and utilized to optimize movement design and learning plans. This record serves as data to track changes in the user's physical condition and rehabilitation progress.

[0303] Learning content regarding ballet movements or training processes performed by the user is also recorded in the storage unit. This data includes information such as the type of movement learned by the user, difficulty level, number of repetitions, and performance accuracy. For example, if a user records 85% accuracy in a specific ballet movement, this information is saved and utilized to design movements of appropriate difficulty and intensity for the next learning stage. This storage function serves to provide a personalized learning experience for each user.

[0304] The user's academic achievement and rehabilitation progress are also recorded in the storage unit. Achievement data includes a quantitative evaluation of how effectively the user performed target movements, while rehabilitation progress data tracks the progress toward achieving the rehabilitation goals set by the user. For example, it records how much muscle strength or flexibility has improved during the rehabilitation movements performed by the user each week, and based on this, a long-term rehabilitation plan can be established.

[0305] The storage unit also records personalized values ​​set by the user for the learning environment or rehabilitation training. This includes setting information such as the user's preferred feedback method (visual, auditory, tactile), training intensity, and the difficulty of target movements. For example, if a user prefers visual feedback and desires low-intensity movements, these settings are stored so that the system can provide an environment optimized for the user. This data is continuously updated and supports users in enjoying a better experience during the learning and rehabilitation process.

[0306] The storage unit according to the present invention is configured to include a motion library. For ballet videos stored in the motion library, age, gender, and age information suitable for each performance are stored together.

[0307] If the user is disabled or requires rehabilitation treatment, the storage unit additionally stores the type of disability, body parts requiring rehabilitation, and disability grade, and in this case, each ballet video included in the motion library may also store information on the type of disability that is difficult for the user to follow and at least one of the disability grade.

[0308] FIG. 7 illustrates an image processing method that supports rehabilitation therapy based on ballet movements according to an embodiment of the present invention, and the process is as follows.

[0309] (S71) Initial physical condition measurement step: The user measures the initial physical condition, such as joint range of motion, muscle activity, and balance ability, through the sensor unit. Real-time data is collected through the camera and sensors, and this data is transmitted to the analysis unit and storage unit.

[0310] (S72) Data Analysis Phase: The collected data is analyzed through noise removal and data correction in the preprocessing unit and joint position extraction unit of the body condition analysis module and image processing module.

[0311] (S73) Customized motion design step: The customized ballet motion design module designs customized ballet movements suitable for the user's condition from ballet movements stored in the motion library using an artificial intelligence learning module based on the analyzed data.

[0312] (S74) Step of presenting ballet movements to the user: The learning module presents movements designed by the customized ballet movement design module to the user through the display.

[0313] (S75) User's ballet movement execution step: The user follows the ballet movements displayed on the display.

[0314] (S76) Monitoring and recording of progress: The results of movements performed by the user are collected in real time through cameras and wearable sensors, and the user's posture data (joint, muscle, balance data) is monitored in real time by the sensor unit. The learning module evaluates the state of movement performance based on this data and displays it on the display to correct it immediately if necessary. The degree of rehabilitation treatment and learning progress based on the results of movements performed by the user are recorded as long-term data in the storage unit.

[0315] (S77) Movement execution status feedback step: The learning module instructs the customized ballet movement design module to design a movement of appropriate difficulty and intensity for the user by reflecting the user's evaluation of the movement execution status.

[0316] (S78) Prediction of Treatment Progress and Plan Update: Data accumulated over the long term is used to predict the course of treatment, and a customized rehabilitation plan optimized for the user is continuously updated. Additionally, based on data accumulated in the storage, the user and therapist can reset or adjust rehabilitation goals. This is dynamically adjusted according to changes in the user's condition and the level of goal achievement.

[0318] The present invention further comprises a control unit (not shown in the drawing) capable of performing the role of ensuring smooth operation of the entire system and providing an optimal learning and rehabilitation experience to the user by comprehensively managing multiple camera images, sensor data, and a feedback system, and by analyzing and adjusting them in real time.

[0319] The control unit is a component that comprehensively manages the operation of the entire system, including the multi-camera system, image processing module, learning module, feedback module, and display module, and controls the real-time processing of user operation data and the provision of feedback.

[0320] The control unit coordinates the data flow between modules during the user's ballet performance, analyzes collected data to generate appropriate feedback, and provides it in a form optimized for the display.

[0321] The control unit transmits images captured by multiple cameras to an image processing module and simultaneously collects sensor data to generate fused information in real time. While the image processing module constructs a 3D model, the control unit performs a comparison between the user's current movement and the target movement, and based on the analysis results, provides immediate correction guidance in conjunction with the learning module and the feedback module. For example, if the user's joint angle differs from the target movement by more than a certain level, the control unit displays visual guidelines on the display or activates auditory and tactile feedback to support the user in correcting the movement in real time.

[0322] In addition, the control unit is linked to the user interface to enable 360-degree rotation and zoom in / out functions of the display, and performs screen adjustment so that the user can view specific actions from various angles. To this end, the control unit detects user input and controls the display module to output an optimal screen configuration by reflecting it.

[0323] Furthermore, the control unit is linked with a management server to continuously monitor the user's learning and rehabilitation progress based on data analyzed by the learning and feedback modules, and to adjust customized training plans when necessary. The stored data is utilized for long-term learning pattern analysis, exercise equipment recommendations, and the establishment of personalized rehabilitation plans, thereby supporting the provision of an optimized learning and rehabilitation environment for the user.

[0325] The image processing system according to the present invention includes a communication module, through which the system is linked with a management server located at a separate location. The communication module enables real-time data transmission and reception, thereby supporting the management server in integrally managing and analyzing user data and learning results. The management server provides an interface that allows users to directly access and input data or change settings, thereby enhancing the usability and flexibility of the system.

[0326] The management server stores data entered by the user and can generate personalized learning plans or set rehabilitation goals based on it. For example, if a user inputs their physical condition or rehabilitation goals into the server, the management server can analyze this information to recommend suitable ballet movements or transmit it to a learning module to generate learning content.

[0327] Furthermore, the management server can evaluate long-term learning and rehabilitation progress by analyzing data collected from users. This allows administrators to compare data from multiple users or continuously monitor the achievement of specific users. Data stored on the server is provided to users and administrators in the form of visual reports, offering clear insights into learning outcomes and rehabilitation progress.

[0328] The integration of the communication module and the management server creates an environment capable of remotely processing user motion data in real time and providing feedback by rapidly reflecting analysis results. For example, while a user is practicing ballet in a remote location, the management server analyzes the data in real time and provides motion correction guidelines to the display or updates feedback if necessary.

[0329] As such, the image processing system according to the present invention is linked with a management server, thereby enhancing the scalability of the system and providing an efficient structure that supports remote learning and rehabilitation. Through this, users can continuously carry out personalized learning and rehabilitation plans regardless of location or time.

[0331] Although the present invention has been described with reference to preferred embodiments, those skilled in the art will understand that various modifications and changes can be made to the invention without departing from the spirit and scope of the invention as described in the following claims. Explanation of the symbols

[0332] 100 : Body Condition Analysis Module 200 : Image processing module 210 : Preprocessing section 220 : Binning Processor 230 : Joint position extraction unit 300 : Sensor section 310: Electromyography sensor 320: Pressure sensor 330 : IMU sensor 340 : Camera 350 : Multi-camera 400 : Customized Ballet Movement Design Module 41 : Wearable sensor 42 : Marker 43 : Attachable sensor 500 : Storage unit 510 : Action Library 600 : Learning Module 700 : Display 800 : Communication module 900 : Management Server

Claims

Claim 1 An image processing system including an electronic device, comprising: an image sensor configured to generate image data by converting incident light through a lens into an electrical signal; an image processing module comprising a binning processing unit that performs binning by grouping adjacent pixels within the image data received from the image sensor, and processes and analyzes the image data by detecting and tracking the features of an object included in the binning processed image; a display unit that displays the content processed and analyzed by the image processing module as virtual reality; and an image processing system characterized by providing feedback based on the analysis result of the image processing unit. Claim 2 An image processing system according to claim 1, wherein the binning processing module converts the resolution of the received image data or the signal strength of each pixel. Claim 3 An image processing system characterized in that, in paragraph 2, the object detected and tracked by the object detection and tracking module is a head, upper body, and legs.

Images